Quantization 101: How Smaller Models Keep Getting Smarter

Cutting weights from 16 bits to 4 bits removes 75% of the information and costs you a few percent of quality. That ratio should be suspicious. It is only possible because weights are not uniformly important, and the entire field of post-training quantization is a set of techniques for finding out which ones matter and spending your bits there.

The formula everything reduces to

Quantization maps a range of floats onto a small set of integers. Dequantization maps back. For asymmetric integer quantization:

# quantize
scale      = (w_max - w_min) / (2**bits - 1)
zero_point = round(-w_min / scale)
q          = clamp(round(w / scale) + zero_point, 0, 2**bits - 1)

# dequantize (this runs in the inference kernel)
w_hat = scale * (q - zero_point)

The error is |w - w_hat|, bounded by half a step, and the step size is scale. Which means the whole game is minimizing scale, and scale is determined by the range of the values you group under one scale. Reduce the range, reduce the error. That is the only lever in the formula.

Why one scale per tensor destroys the model

Naive round-to-nearest with a single scale for an entire weight matrix produces a model that generates garbage, and the reason is outliers. In a trained transformer, a small number of channels carry weights and activations with magnitudes one to two orders of magnitude above the rest of the tensor. These are not noise. They correspond to features the network relies on heavily, and they emerge consistently during training in specific hidden dimensions.

With one scale per tensor, those outliers set w_max. Suppose the outlier is 20 and the typical weight is 0.05. At 4 bits you have 15 steps to cover a range of roughly 40, which puts the scale near 2.7. Every typical weight in that tensor now rounds to the same bucket. You have preserved a handful of large values with perfect fidelity and quantized 99% of the tensor to a constant. The model is destroyed not because 4 bits is too few, but because you allocated all of them to the wrong place.

Per-group scales fix most of it

The cheap fix is to stop sharing one scale across millions of weights. Split each row into contiguous groups of 64 or 128 and store a scale and zero-point per group. Now an outlier only inflates the scale for its own group of 128, and the other groups get tight ranges.

The overhead is easy to compute and worth doing explicitly:

# group size 128, FP16 scale + FP16 zero_point = 32 bits per group
bits_per_weight = 4 + 32/128 = 4.25

# group size 64
bits_per_weight = 4 + 32/64  = 4.50

# group size 32
bits_per_weight = 4 + 32/32  = 5.00

# 8B params at 4.25 bpw:
8e9 * 4.25 / 8 = 4.25 GB

This is why quantized models advertise sizes like 4.25 or 4.65 bits per weight rather than a clean 4. Group size 128 is the usual choice because the accuracy gain from 128 to 64 is small while the storage cost is real, and since local decode is memory-bandwidth-bound, every extra byte per weight is a direct hit to tokens per second.

The two methods that close the remaining gap

Per-group round-to-nearest gets you most of the way at 4 bits and still loses measurable perplexity. Two approaches recover almost all of it, and they attack the problem from opposite directions.

GPTQ compensates for error it has already made.Instead of rounding all weights independently, it quantizes a layer column by column, and after each column it updates the remaining unquantized weights to absorb the error just introduced. The update direction comes from the inverse Hessian of the layer's reconstruction loss, approximated from a small calibration set, so it is a second-order method: it knows which remaining weights are most effective at cancelling the damage. The insight is that quantization error in one column is partly correctable by other columns, because what matters is the layer's output, not any individual weight. The GPTQ paper has the derivation.

AWQ protects the weights that matter before rounding. Its central observation is counterintuitive: the salient weights are identified by activation magnitude, not weight magnitude. A weight multiplied by a consistently large activation contributes more to the output than a large weight multiplied by near-zero activations, so the channels worth protecting are the ones the data lights up. AWQ measures per-channel activation scale on a calibration set, then applies an equivalent per-channel scaling that shrinks the dynamic range those salient weights have to share with their neighbors before quantizing. The AWQ paper reports that protecting only about 1% of channels recovers most of the loss, and because the method needs no backpropagation it does not overfit the calibration set the way reconstruction-based approaches can.

The falsifiable claim tying them together: quality at a given bit width is governed by the dynamic range within each shared-scale group, not by the bit width itself. Every technique above is a different way to shrink that range, and this predicts correctly that a 3-bit model with tight per-group ranges can beat a 4-bit model with one scale per tensor.

Quantization is not lossy compression of a model. It is a bit-allocation problem, and the interesting work is entirely in deciding where the bits go.

KV cache quantization is a different axis

Weight quantization is a fixed win: it shrinks a constant. KV cache quantization shrinks something that grows linearly with sequence length, and at long context it is often the bigger lever.

kv_bytes = 2 * layers * kv_heads * head_dim * seq_len * bytes_per_elem

# 8B-class model, 32 layers, 8 KV heads (GQA), head_dim 128, 32k context:
FP16: 2 * 32 * 8 * 128 * 32768 * 2 = ~4.3 GB
INT8: half that                    = ~2.1 GB
INT4: quarter                      = ~1.1 GB

At 32k tokens the FP16 cache rivals the 4-bit weights in size. Quantizing it to 8 bits frees roughly two gigabytes, which on a memory-constrained machine is the difference between a workable context length and an out-of-memory error. It is also the difference between fitting in fast unified memory and swapping, which costs far more than the quantization error ever will.

The caveat is that keys and values quantize differently. Keys have per-channel outlier structure similar to weights and want per-channel scales; values are better behaved and tolerate per-token scales. Most implementations that report bad quality from KV quantization applied the same scheme to both. INT8 is close to free in practice; INT4 keys are where degradation shows up on long-context retrieval tasks, which matters if the model is meant to find a specific fact buried in a 30k-token context.

The reason any of this touches a memory layer: a 4-bit local model with a quantized cache has less room for context, not more, so what gets loaded into that window has to be selected rather than dumped. Unimatrix exists to do that selection, returning the handful of relevant memories instead of a transcript. The retrieval side is in the MCP reference.