What you're looking at
Every color block is one token — the actual unit a GPT model reads and bills in, not a word or a character. Paste any text and watch it split the way the model would actually see it. Short common words are usually their own token; longer or rarer words get cut into pieces ("tokenization" might become token + ization). A leading space is usually attached to the next word's token, not its own — that's why word-boundary tokens look slightly offset from where you'd expect them to break.
Why this matters
Seeing the split changes how you write for a model. A prompt that looks short in characters can be surprisingly expensive in tokens if it's full of rare words, code, or non-English text — and a prompt that looks long can tokenize efficiently if it's plain, common English. This is the same thing our Token Counter measures in aggregate; this tool shows you why the number is what it is, one piece at a time.
Why some scripts split into so many pieces
Switch to text with emoji or non-Latin scripts (中文, العربية, हिन्दी) and you'll usually see far more tokens per character than for English. GPT tokenizers are trained mostly on English-heavy web text, so their vocabulary has efficient single-token chunks for common English words but has to fall back to smaller byte-level pieces for scripts it saw less of during training. This isn't a bug in the tool — it's a real, well-documented cost difference between languages when using these models.
A known limitation, shown honestly
A few token pieces — usually ones from a split emoji or non-Latin character — can display as a broken or replacement character even though the full text reads correctly once every token near it is included. This happens because each chip is decoded on its own for speed, and a handful of tokens only form valid, readable text once joined with their neighbors. It doesn't affect the token count, which is always exact.
FAQ
Which tokenizer does this use?
The same one the models use: OpenAI's published BPE tokenizer, via the o200k_base encoding for GPT-4o and GPT-4o mini, or cl100k_base for GPT-4 Turbo and GPT-3.5.
Does my text get uploaded? No. Tokenizing happens entirely in your browser. Nothing is sent to a server or to any model.
Why does a very long paste stop showing colors partway through? Rendering thousands of individual colored chips gets slow, so the visual view caps at 1,500 tokens. The token count above it is always exact for your full text, regardless of length.
Is this the same tokenizer Claude or Gemini use? No — Anthropic and Google don't publish a browser tokenizer, so there's no way to visualize their exact token boundaries client-side. This tool only covers the OpenAI encodings, which it can show exactly.