Quantization means storing an AI model's weights with fewer bits, for example 8 or 4 instead of 16. The model then takes less memory and can run on an ordinary computer, at the cost of some precision. It has the same parameters as before: each one just takes less space.
A question from the app. Pick an answer.
You have 8 GB of RAM free. The weights take 16 GB at 16 bits. Which one do you get?
The weights alone go past your 8 GB: it does not fit in memory.
The weights take about 8 GB: all you have, with no room to run the model or hold the conversation.
Each weight with a quarter of the bits: about 4 GB. That leaves room to run it and for the chat, at the cost of a little precision.
An AI model is, at bottom, a huge list of numbers: the weights the neural network learned during training. An LLM has billions of them. The usual format stores each one in 16 bits.
Quantizing means rounding those numbers so they fit in fewer bits, like going from 3.14159 to 3.1. Each weight gets a bit less exact, but the model loses no parameters. It is usually done after training.
A rough calculation: parameters × bits ÷ 8 gives you bytes. A 10 billion parameter model needs about 20 GB at 16 bits, 10 GB at 8 bits and 5 GB at 4 bits. That is the math for the weights: the file carries a bit more, and running the model takes extra memory, especially in long chats.
That is the difference between a local model that will not start and one that fits in your laptop's memory. The same goes for on-device AI: models that run on a phone are usually quantized.
When you download a model in Ollama, LM Studio or Hugging Face, the same model shows up in several versions. In names like Q8, Q5 or Q4, the number roughly tells you the bits per weight; the letters after it (like K_M) are variants of the method. "8-bit" or "4-bit" say the same thing, and FP16 or BF16 is the 16-bit version, not quantized.
A rule of thumb: among the versions of the same model, pick the one with the most bits that fits in your memory with room left for the chat. Then test whether the quality is enough for you.
It depends on the model and the task. At 8 bits the loss is usually small, and 4 bits is usually enough. With fewer bits, errors go up. Test it in your language, with your own tasks.
"Quantized is the same as a smaller model." No: it has the same parameters, with fewer bits each. Distillation is something else: it trains another model with what a bigger one teaches it. That one usually comes out smaller, and it can be quantized too.
"Fewer bits is always the worst choice." With the same memory, a big model at 4 bits often does better than a small one at 16.
"It is only for home use." The cloud uses it too: serving with fewer bits takes less hardware, and that shows up in the cost of AI.