What Does Tokenization Actually Look Like? A Visual Guide

You probably already know the fact: AI models don't read words, they read tokens. It's the kind of thing that gets repeated in every "how AI works" explainer, right before the writer moves on to something else. But knowing the fact and seeing it happen inside a sentence you actually wrote are two different things.

Here is a sentence, split the way GPT-4o actually reads it:

[Token][ization][ is][ how][ a][ language][ model][ reads][ text][.]

Ten tokens for ten words — but not a clean one-to-one match. "Tokenization" alone became two pieces: "Token" and "ization". Every other word stayed whole, each one carrying its leading space as part of the same token. That detail — the space belongs to the word that follows it, not the word before it — is the kind of thing that only clicks once you see it laid out.

Why the split looks uneven

GPT tokenizers use a vocabulary built from the text they were trained on, mostly English-heavy web content. Common English words earned their own single token because they showed up constantly during training. Rare words, made-up words, and technical jargon didn't get that privilege — they get cut into smaller, more generic pieces the model has actually seen before.

Photo by Patrick Perkins on Unsplash

This is exactly why a prompt full of jargon, code, or invented product names can quietly cost more tokens than a plain-English prompt of the same length — even though both look about the same size to you.

The language gap nobody warns you about

Type the same short greeting in English and in Chinese and count the tokens. "Hello" is one token. "你好" — also just two characters, also just a greeting — commonly comes out to two or more tokens, because the tokenizer's vocabulary has far less coverage of Chinese than English. Emoji have the same problem: a single 🎉 can cost several tokens on its own.

• This isn't a flaw specific to one product — it's a documented, structural side effect of how these tokenizers were trained
• It means non-English text can cost meaningfully more per character to send through an API than English text does
• It also means "count the characters" is a bad proxy for "estimate the cost" once your content is multilingual

Two tokenizers, two different answers

GPT-4o and GPT-4 Turbo don't even use the same tokenizer. GPT-4o uses a newer encoding (o200k_base) with a larger vocabulary than GPT-4 Turbo and GPT-3.5's older one (cl100k_base). The practical effect: the exact same sentence can produce a different token count depending on which model you're sending it to — usually a modest difference for English, but often a much bigger one for non-Latin scripts, where the newer, larger vocabulary covers more ground.

Photo by Markus Kammermann on Unsplash

That's a real, practical reason to check token counts per model rather than assuming one number applies everywhere — which is exactly the gap between a token counter (gives you the number) and a tokenizer visualizer (shows you why the number looks the way it does).

What to actually do with this

You don't need to obsess over token boundaries to write good prompts. But if you're building something that sends a lot of text through an API — multilingual support, code generation, long-document summarization — it's worth spending two minutes watching how your actual content tokenizes before you assume a cost estimate based on word or character count.