AI Compute RadarQuantization
Quantization stores a model's weights at lower precision — for example 4-bit instead of 16-bit — cutting memory to roughly a quarter at a small quality cost.
What is quantization? It is storing a model's weights with fewer bits per number — typically 4 instead of 16 — so the same model needs about a quarter of the memory. Nothing is retrained: the numbers are rounded to a coarser scale, and the answers change slightly in exchange for a file that fits real hardware. A measured example from this site: the Q4_K_M file of Gemma 4 12B is 6.6 GB; at 16 bits per weight the same 12 billion parameters would take about 24 GB.
A model is billions of numbers. Stored at full training precision, each number takes two bytes; quantization rounds them to a coarser scale so each takes only a fraction of that. Think of rounding every price in a supermarket to whole euros: the totals stay almost right, and the notebook you need becomes much smaller.
This is the technology that makes local AI possible at all. A 30-billion-parameter model needs on the order of 60 GB at full precision — beyond any consumer graphics card. The common 4-bit quantizations bring it to roughly a third of that, into the range real machines have.
The cryptic names — Q4_K_M, Q8_0, IQ4_XS — are recipes: how many bits, and which packing method. Higher numbers keep more precision and more gigabytes. For everyday chat use the popular 4-bit recipes hold up well; very aggressive 2- and 3-bit versions save more memory but the rough edges start to show.
Where you see it on the radar
Our fit tables measure the Q4_K_M file where one exists — the quantization most local-AI guides assume — and every table names the exact recipe it measured.
Check a quantized model against your hardware →Related terms