Temperature is a setting that changes how much randomness an AI model uses when picking the next token, the piece of text that comes next. Lower it and the likely options get favored more, so answers vary less. Raise it and the less likely ones get more chances. It changes variety, but it does not guarantee accuracy.
The cat climbed onto the ?
A question from the app. Pick an answer.
When completing "The cat climbed onto the...", what does the model predict?
It does not store one word per sentence: it computes probabilities and, depending on the settings, may pick roof, wall or sofa.
It predicts next token probabilities. Settings determine selection; here we use example words.
It does not search: it works out what usually comes next from what it learned.
Ask a chat for "a name for my bakery", then hit "regenerate": you may get another one. It did not forget the first answer or change its mind. When the model picks by drawing, it can take another path, and temperature decides how loaded the dice are.
A language model writes one token at a time. To complete "The cat climbed onto the..." it works out probabilities: roof 45%, wall 30%, sofa 20%, theorem 1% and the rest spread over others (example numbers). Then it has to pick one, and that is where temperature comes in.
At a low temperature, the odds lean toward the favorite: roof comes out much more often. At a high temperature, the odds even out: wall and sofa show up more, and even theorem gets a shot. Since every chosen word changes what comes next, a small difference early on can end in two different answers.
Start with the value the provider recommends for that model. If you want answers that vary less, like pulling data from an invoice or sorting messages, try lowering it. If you want variety, like ten different titles for a post, try raising it. And if you need JSON in a fixed format, use structured outputs: temperature does not guarantee the format.
In a chat app you usually cannot change it: the company already picked a value. Where you do set it is in the API, as one more parameter of the request. The range and the default depend on the provider, and some reasoning models do not accept it or fix it themselves: check the docs for the model you use.
{
"model": "example-model",
"temperature": 0.2,
"messages": [
{ "role": "user", "content": "Sort this message: billing, support or sales" }
]
}"At temperature 0 it always gives the same answer." It comes close, but many providers do not guarantee it: if you need the exact same result every time, save it. "A low temperature stops it from making things up." No: if the most likely answer is a false fact, it will repeat it with full confidence; that is a hallucination, and good sources are what fight it. "Higher means more creative." Up to a point: too high and the text loses the thread.