AWQ and NF4

INT4's sixteen levels are evenly spaced. Trained weights are not evenly spread: they cluster near zero in a roughly normal bell. NF4 (NormalFloat, from QLoRA) spends its sixteen codes where the weights are, so each code is used about equally often. QLoRA's recipe: take quantiles of the standard normal distribution, scale them to $[-1, 1]$, and quantise each block of 64 weights by dividing it by its absolute maximum and picking the nearest code.

The animation rebuilds bitsandbytes' table, the one QLoRA ships, from scratch: eight positive quantiles at probabilities evenly spaced from $\delta = 0.9677083$ down to $0.5$, seven negative ones, and an exact zero (so the type is asymmetric: zero gets a code of its own, and padding or pruned weights quantise exactly). Divided by the largest quantile, 1.848, the values match bitsandbytes' table to 1.08 × 10⁻⁷ (its table is built in 32-bit floats). bitsandbytes documents the offset as the outermost quantile boundary, about 1.85 standard deviations, chosen empirically.

Does it help? On 4,096 approximately normal values in blocks of 64, NF4's mean squared error is 0.008666 against 0.01172 for INT4 absmax with the same blocks, 1.352 times smaller (SQNR 20.7 dB against 19.4 dB).

AWQ (activation-aware weight quantisation, Lin et al.) starts from an observation: a small fraction of weights matter far more than the rest, and they are found by looking at the activations, not the weights. Keeping just 1% of the weight channels in FP16, chosen by activation size, recovers most of the accuracy that plain 4-bit rounding loses. Mixed precision is awkward for kernels, so AWQ gets the same effect with a scale instead: multiply the weights of input channel $i$ by $s_i > 1$ before rounding and divide its activations by $s_i$. The scaled weights see a relatively finer grid, so the important channels are rounded more accurately.

The scale is $s_i = \overline{|x_i|}^{\alpha}$, and AWQ searches $\alpha \in [0, 1]$ on calibration data. The animation runs that search over the layer of chapter 7, in eighths: the grey bars are each channel's mean activation, the orange bars the scale it gets. At INT3 in groups of 8, no scaling ($\alpha = 0$) gives an output error of 0.02271; the best eighth, $\alpha =$ 0.375, gives 0.01833, 1.239 times smaller. This layer's channels differ by less than an LLM's, so the gain is smaller than the paper's.

Now a real network. The animation above runs the Transformer Decoder Explainer's tiny transformer (two blocks, 16-wide, two heads, a 64-character vocabulary) in your browser, twice: once as is, once with the weights of every block's six linear layers quantised. It predicts the next character at all 32 positions of four prompts, and compares. The weights are random, so the text is gibberish, but every multiply is real.

Over the 128 predictions, with the token embedding (and the tied output head) kept at full precision:

Three lessons survive the model's smallness. Each bit removed roughly doubles the drift or more. With the same blocks of 64, NF4's grid beats INT4's, because these weights are normal. And GPTQ, calibrated on four other prompts, recovers part of what plain INT3 rounding loses. Pick a format and a prompt above and watch where predictions flip: they flip where the reference's top two candidates were close, and the reference's median gap between them is 1.884 logits.