What is quantization in AI models? Q4, Q8 and FP8 explained
Updated
Quantization is compressing a language model by storing each weight with fewer bits, for example 8 or 4 instead of the usual 16. A 4-bit copy needs roughly a quarter of the memory, so bigger models fit on smaller graphics cards and often run faster, at the cost of some accuracy. Labels such as Q4_K_M, Q8_0, FP8, AWQ and EXL2 name different quantization formats.
Try it now, free, no account neededStart a story with IRISBuilt to study human life. She keeps studying you.How it works
A model's weights are numbers such as 0.01734, usually released in a 16-bit format (BF16 or FP16). Quantization rounds them onto a coarser grid that fits in fewer bits. To keep the rounding error small, weights are handled in small groups that share a scale, so each group gets a grid fitted to its own range of values.
For a 12-billion-parameter model, the weights alone come to roughly:
- 16-bit: about 24 GB
- 8-bit: about 12 to 13 GB
- Q4_K_M, around 4.5 bits on average: about 7 GB
Context memory comes on top and grows with chat length. What 12B means covers the size side of that sum.
Reading the labels
- BF16 or FP16: the original precision, not quantized.
- FP8, INT8 and Q8_0: 8-bit. Very close to the original for chat.
- Q6_K and Q5_K_M: GGUF files, the format used by llama.cpp and apps built on it such as KoboldCpp, LM Studio and Ollama, sitting between 8 and 4 bits.
- Q4_K_M: the 4-bit GGUF that many home users treat as the sensible default.
- Q3, Q2 and IQ2 files: very small, with clearly visible damage.
- AWQ and GPTQ: 4-bit formats aimed at GPU servers such as vLLM.
- EXL2 and EXL3: ExLlama formats sized in bits per weight, for example 4.65bpw.
Separately, some backends quantize the KV cache, the memory that holds your context, so longer chats fit on the same card. It is a different setting from the model's own quantization.
What you lose in roleplay
At 8 bits most people cannot tell the difference. Around 4 to 5 bits the loss is small in casual chat, though it can show in long stories as slightly vaguer details or more repetition. Below about 4 bits, smaller models suffer most: they forget instructions, break formatting and pick odd words.
Larger models tolerate rounding better. A common rule of thumb is that a bigger model at 4 bits usually beats a smaller one at 8 bits when both need the same memory, which is why people squeeze the largest model their card can hold into a 4-bit file.
Common mistakes
- Blaming the quant for problems that come from the card, the prompt template or the samplers.
- Assuming "Q4" means the same thing in every format. A Q4_K_M GGUF, an AWQ file and a 4.0bpw EXL2 are built differently.
- Picking the smallest file to fit a huge context, then wondering why the character keeps losing the plot.
- Downloading from unknown uploaders. Use well-known sources and check the model card.
- Comparing quants of different models instead of different quants of one model.
Do you need to care?
Only if you run a model yourself. Then you choose the largest model and quant that fits your card with room left for context. When you use gemma-4-12b or gemma-4-31b through UnsavedAI, the model runs on our servers: there is no file to download and no quant to pick, and your device only runs the chat app. SillyTavern without a GPU shows the setup, and Models lists what is online now. To feel how a steady model holds a precise voice, try IRIS.
Questions people ask
Does quantization make a model dumber?
A little, depending on the bits. 8-bit is close to the original, 4-bit loses a small amount, and 2 or 3 bits loses a lot.
What is the best quant for roleplay at home?
The largest model and quant that fits your card while leaving room for context. Q4_K_M and Q5_K_M are common starting points.
Is quantization the same as distillation?
No. Distillation trains a new, smaller model to imitate a larger one. Quantization keeps the same model and rounds its weights.
