A probability is calibrated when it matches what happens across many comparable cases: across many comparable predictions with probability 0.8, that option matches the actual outcome about 80% of the time. It is measured in groups, so it promises nothing about a single answer. It applies to any model that classifies.
A question from the app. Pick an answer.
Jev is an AI model that does not write: it picks an option or gives a score, with probabilities.
Probabilities are well calibrated in your tests. 200 tickets got 0.7: how many are right?
Calibrated means that across many cases given 0.7, about 70% are right. It holds for the probability, not the confidence, and it does not say which ones miss.
Being the most likely does not make it certain. With a well calibrated 0.7, about 3 in 10 of those cases miss: some 60 out of 200.
With three teams, a tie would be 0.33 each; 0.7 is one option well ahead. Calibrated, it is right about 70% of the time: some 140 out of 200.
If a well calibrated model assigns 0.8 to an option across many comparable predictions, that option matches the actual outcome about 80% of the time. That does not mean exactly 80 out of a batch of 100, or that this particular case is right: roughly one in five fails, and you do not know which.
Nor does it guarantee the model is calibrated on your data. You check that by testing with your own real cases.
In a test, a deciding model read "Tomorrow is cloudy." and gave "It will rain." 0.29. It leans no, but it is not a firm no like 0.03. If you round it to yes or no, you lose that difference.
What to do with numbers in the middle (act, confirm or hand the case to a person) is up to you, based on what a mistake costs.
Some models give a confidence score on top of the probabilities. In Jev, the confidence of a multiple choice question is computed from the most likely option and the number of options: it is 0 when they are evenly spread and 1 when all the probability is on one option, which does not guarantee being right. A confidence of 0.8 does not mean "80% right": it tells you how concentrated the probabilities are. Other models compute it differently, so the same number does not mean the same thing everywhere.
An example with three options: (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) give the same confidence, 0.4. And Jev's yes or no questions have no separate confidence: 0.03 is a firm "no", not doubt. Doubt sits near 0.5.
With those numbers you can decide what your system does. A pattern TypeSafe suggests uses three ranges: high, the system acts on its own; medium, it acts carefully, confirming or reviewing; low, it does not act and hands the case to a person or asks for clarification.
Where you draw each line depends on what a mistake costs. In the same banking app, showing a balance can need less than approving a transfer. In TypeSafe's Confidence page example, transfers require confirmation from the user even with confidence above 0.9. Start conservative, test with your data and adjust.
Any machine learning classifier that returns probabilities can be looked at this way: a spam filter, a fraud detector, one that guesses the language of a text. Jev puts calibration at the center: TypeSafe says it trains it for that, with a method it calls RLCD (reinforcement learning for calibrated decisions).