One Text, Five Tokenizers: LLM Token Counts Compared

A 648-character customer support email comes to 139 tokens on the tokenizer behind GPT-4. Translate that same email into Persian, keep the tokenizer, and it comes to 463 tokens: 3.3 times as many for the same message.

So “how many tokens is this?” has no single answer. It depends on which tokenizer reads the text and on what the text is. We measured six short texts under five tokenizers to see how much the answer moves, and where it barely moves at all.

What an LLM tokenizer does

Before a model sees your text, a tokenizer cuts it into pieces from a fixed vocabulary: whole common words, word fragments, punctuation, sometimes single bytes. Every model family ships its own vocabulary, so the same sentence splits differently from one model to the next. If you want to watch the cutting happen piece by piece, what tokenization actually looks like walks through it with colored examples.

OpenAI’s tiktoken cookbook (checked 2026-10) lists the two encodings most people meet: o200k_base for GPT-4o and GPT-4o mini, and cl100k_base for GPT-4, GPT-4 Turbo and GPT-3.5 Turbo. It also says plainly that encodings “vary in how they split words, group spaces, and handle non-English characters.” The test below puts a number on that sentence.

The same email in English, Persian and Chinese

The test text is an ordinary support reply: a duplicate charge, a $48.20 refund, a jacket that shipped. We wrote it in English, then translated it into Persian and Chinese, keeping every detail. The three versions say the same thing, so any difference in tokens comes from the language and the tokenizer, not the content.

In English the three exact tokenizers land within six tokens of each other (135 to 141). That is why most people never notice tokenizers differ: for English prose, the open ones mostly don’t. The Claude estimate is the outlier in every column, and gets its own section below.

Persian costs far more. On cl100k_base the Persian email takes 463 tokens, 3.3 times the English one. GPT-4o’s newer o200k_base brings that down to 215, less than half, though still 1.6 times English. DeepSeek V4’s tokenizer sits in between at 256.

With Chinese, it depends on the tokenizer. DeepSeek, a Chinese lab, counts the Chinese email at 142 tokens against 141 for English, so on its tokenizer the two languages cost the same. GPT-4o needs 162 and GPT-4 needs 236.

Why some languages cost more tokens

A tokenizer’s vocabulary is a fixed list, and every entry spent on one language is an entry not spent on another. As the names suggest, cl100k_base has roughly 100,000 entries and o200k_base roughly 200,000. When a Persian word has no entry of its own, the tokenizer falls back to smaller pieces, down to single bytes. A Persian letter takes two bytes in UTF-8, so a word can shatter into many tokens.

You can see it in the Tokenizer Visualizer. Paste the Persian email and pick the GPT-4 Turbo / GPT-3.5 encoding (that is cl100k_base). Almost every colored chip holds one or two letters, and the thin empty-looking ones are half a character, mostly the Persian digits in ۴۸٫۲۰ and the small hamza mark: the tokenizer split that character’s two bytes across two tokens.

Switch the same view to GPT-4o and the chips get visibly longer, often whole words. That switch alone takes the count from 463 to 215.

Code and JSON: where tokenizers mostly agree

We also ran three texts that developers paste into models all day: a 19-line Python script that totals a CSV by month, an order record as JSON, and an emoji-heavy social post for a café opening.

Code and JSON are boring, in a good way. The exact tokenizers are within 7% of each other on the Python file and 5% on the JSON, which tops out at 249 tokens on DeepSeek. Emoji spread wider: 117 tokens on GPT-4o, 137 on GPT-4. An emoji that looks like one picture can be several code points glued together: the woman cook in the post is a woman, an invisible joiner and a frying pan, and the tokenizer pays for each part.

The Claude row stands out in both tables, at 1.4 to 1.9 times GPT-4o. That gap isn’t a guess: the estimate is calibrated against Anthropic’s own counts. It also fits Anthropic’s token counting docs (checked 2026-10), which say Claude 4.7 and later models use a newer tokenizer that turns the same text into roughly 30 percent more tokens than earlier Claude models did. A small part is the wrapper: the estimate includes about 11 tokens that Anthropic’s count adds when it wraps text as a message (our calibration offset; the same docs say system-added tokens aren’t billed). That matters on a 300-character post and disappears on a long document. More in counting Claude tokens.

What this means for cost and context windows

Providers bill per token and context windows are measured in tokens, so the tokenizer quietly sets both your price and how much fits. Our Persian email cost 3.3 times as many tokens as the English one on a cl100k tokenizer, at the same price per token; on GPT-4o’s tokenizer the gap was 1.6 times. A support tool replying in Persian should expect a gap of that order on every reply. If you serve several languages, size your budget from the most expensive one, not from English.

Be careful with rules of thumb, too. Google’s token guide (checked 2026-10) says a Gemini token is about 4 characters. Our English email fits that, at about 4.3 characters per token on the Gemini estimate. The Chinese email comes to about 1.6. Any characters-per-token shortcut assumes English.

Count your own text

The Tokenizer Visualizer is a free online tokenizer that shows exactly how GPT-4o or GPT-4 splits your text, piece by piece, which is the fastest way to spot a language or format that costs more than you expected. For a count across many models at once, the Token Counter gives exact numbers for GPT-4o and open-weight models like DeepSeek, and labelled estimates for GPT-6, Claude, Gemini and Grok, with an opt-in exact count from the provider when you need one. OpenAI publishes the GPT-4o and GPT-4 encodings but not GPT-6’s, which is why GPT-6 counts are labelled estimates. Both run in your browser: nothing you paste leaves it unless you switch on that provider count.

One limit worth knowing: a real API request adds a chat template, a system prompt and sometimes tool definitions around your text, so the bill counts a little more than the bare text. Measure the full request before you promise anyone a number.