A token is the unit of text a language model like ChatGPT works with. It can be a whole word, part of a word, a space or a symbol. The model does not read letters or words: it reads tokens and answers one token at a time. That is why AI limits and prices are counted in tokens.
Token$cito$ learns$ by$ play$ing$.$
7 tokensA question from the app. Pick an answer.
A long, uncommon word: how does it usually reach the model?
Frequent words often get their own token; rare ones are almost never whole in the vocabulary, so they are built from several pieces.
One letter per token is the last resort. The tokenizer uses chunks of several letters whenever it has them, rare words included.
The vocabulary keeps frequent words whole and builds rare ones from pieces. Those cuts do not have to match syllables.
The word has other uses. A bank token is the code that changes every few seconds to confirm it is you. In crypto, a token is a digital asset, like a coin. In programming, an access token is a long password one app uses to get into another, like an API key. This page is about the AI token: the piece of text.
Before the model reads a sentence, a program called a tokenizer splits it into pieces. Common words usually stay whole; rare or long ones break into several pieces. The space is often attached to the next word.
The split above is illustrative. Each model family has its own tokenizer, so the same sentence can be 7 tokens in one and 9 in another. If you need the exact number, use the token counter from the provider you will use.
Many AI APIs charge per token: what you send (your question, the instructions and the chat history) and what the model answers. A long answer costs more than a short one, even if they say the same thing.
Tokens also measure how much text the model can take into account at once. That limit is called the context window: if the conversation goes past it, the oldest part is left out.
Many tokenizers were built with more English text than other languages. That is why, with several models, the same idea written in Spanish or Portuguese takes more tokens than in English: a word like "aprendiendo" breaks into more pieces. More tokens means more cost and less room in the context window.
You do not need to write to AI in English to save money. But if you build an app with thousands of requests, measure tokens in your users' real language, not with English examples.
"A token is a word." Sometimes, but often it is a piece of one. "A token is a letter." No: a token can hold several letters. "It is a cryptocurrency." Not in text AI: same word, different meaning.