A tokenizer is the program that turns text into tokens before a language model reads it, and turns the tokens the model answers with back into text. It uses a fixed vocabulary of pieces learned from a huge amount of text. Each model family has its own, so the same sentence does not count the same in all of them.
Unfortunately, there is no bread.
8 tokensA question from the app. Pick an answer.
The same text in English and in Spanish, on the same model: how many tokens?
It counts pieces of text, not ideas. Each language gets cut differently, so the same idea can take more tokens in one than in another.
The tokenizer was built from more text in some languages than others. So the count changes when you translate, and the cost with it.
With many models it is the other way around: English has more whole words in the vocabulary. It depends on the tokenizer, not on word length.
Roughly speaking, many tokenizers start from single letters (or bytes) and go through a mountain of text looking for the pair that shows up together most often. If "t" and "h" appear side by side all the time, "th" becomes a new piece. Then "th" and "e" form "the", and so on, thousands of times, until the chosen vocabulary size is reached. This idea is known as BPE (byte pair encoding), and there are variants.
The result: whatever was frequent in that text ends up as a single piece, and rare things are built from several. Each piece has a number, and that is what the language model actually receives: a list of numbers, not letters. That is why many models struggle to count how many "r"s a word has: they never saw the letters one by one.
Each model family (GPT, Claude, Gemini) trains its tokenizer on its own text, picks its vocabulary size (from tens of thousands to hundreds of thousands of pieces) and its rules for spaces, capitals and numbers. A tokenizer may treat "Hello", "hello" and " hello" as three different pieces. Numbers are similar: some tokenizers cut them into fixed groups of digits and others however it falls, so "1,250,000" may end up split in odd ways.
The practical consequence: if you switch models, the token count of the same text changes, and with it the cost and how much fits in the context window. A "100,000 token" limit does not mean the same thing in two different models.
Accented letters and characters like "ñ" or "ç" appear less often than plain letters in the text many tokenizers are trained on, and sometimes become a separate piece: a word like "café" can be split right at the "é". Uncommon place names and surnames, like "Worcestershire" or "Llanfair", tend to break into several pieces. Emojis can also take more than one token each.
Stripping accents from non English text to "save" tokens is a bad idea: the difference is usually small and the text gets worse, sometimes with a different meaning (in Spanish, "papá" is dad and "papa" is potato). What does help is knowing that text full of proper names or emojis will weigh more than it looks.
Use the tokenizer of the model you will actually use: many providers offer a counter on their site or in their API, and the API response usually reports how many tokens each request used. Counting with another model's tokenizer gives you a similar number, not the real one.
The "about 4 characters per token" rule was made for English, and many other languages get fewer characters per token. A simple method: take 20 or 30 real texts from your app, count them and average. If you get, say, 3.2 characters per token, use that number to estimate, not a generic one.