owóorí / the tokenizer tax

What does Yoruba pay?

The same 1,012 sentences, professionally translated into 204 languages, priced by 12 tokenizers. English pays 1.00. Everyone else pays a premium: more tokens for the same meaning, which means a higher API bill, more latency, and less room in every context window. Pick a language and read its receipt.

cheapest of the 204 languages (frontier-tokenizer median) costliest

Yoruba

For every $100 of English-rate usage, this language is billed $269 at the frontier.

Frontier rate = median of GPT-3.5/4, GPT-4o and Qwen2.5, the tokenizers behind the APIs people actually pay for. Same multiplier applies to latency and context shrinkage.

The shape of the tax

Across all 204 languages, 61% pay more than double English's rate at the frontier, and 17% pay more than four times.

Writing Yoruba with its tone and dot-below marks costs +52% under GPT-4o versus the same text stripped bare: a second tax, on the orthography itself.

English-centric BPE frontier BPE multilingual by design