Quantisation in hardware
Where the bits pay off: dequantising in registers, block-scaled formats in tensor cores, and the energy and area of each operation.
Loading the animation…
Concept
Everything so far has been about accuracy. The reason to accept any of it is hardware: narrow numbers are cheaper to store, to move and to multiply.
Moving. For a memory-bound kernel, such as decoding one token at a time, time is bytes over bandwidth, so 4-bit weights are up to four times faster than 16-bit ones. The kernel reads packed weights and dequantises them in registers, just before the multiply, and never writes the wide copy anywhere; GPU Kernels Explained animates exactly that, and shows when the gain disappears (a large batch makes the layer compute-bound).
Multiplying. A multiplier's cost grows roughly with the square of its width. In Horowitz's 45 nm figures (below), an 8-bit integer multiply-add costs 0.3 pJ against 4.6 pJ for FP32, 15.33 times less, and reading a 32-bit word from DRAM costs 640 pJ, more than a hundred FP32 multiply-adds. Narrow formats pay twice: in the arithmetic, and far more in the data movement.
Block scales in the datapath. The MX formats of chapter 4 were designed for this. The spec defines the dot product of two MX vectors as the product of their two block scales times the sum of the element products. The scales are powers of two, so applying them is an exponent add, and the reduction "only computes on the elements": narrow multipliers, an accumulator, and one scale step per block. Recent accelerators implement MXFP8, MXFP6 and MXFP4 (or close variants with finer scales) in their matrix units.
The animation computes one 32-element dot product four ways, one multiply-accumulate per step. FP32 ends 5.59 × 10⁻⁸ from the exact answer, FP16 (multiplying and adding in FP16) 0.2% off, INT8 with one scale per block 0.6%, and MXFP4 30.5%. Energy for the 32 multiply-accumulates: 147.2 pJ in FP32, 48 in FP16, 9.6 in INT8. MXFP4's single block shows why its users rely on many blocks, accumulate in FP32, and keep sensitive layers wider: one 4-bit block of a normal vector is a coarse picture of it.
Horowitz, ISSCC 2014, as tabulated by Gholami et al. (2021), Figure 7. Memory reads have no area entry.
Maths
An integer array multiplier for -bit operands adds partial products of bits, so its area and energy grow as about ; an adder grows as about . Horowitz's table follows that: an 8-bit multiply is 0.2 pJ and a 32-bit one 3.1 pJ, 15.5 times more for 4 times the width. A floating-point multiply is an integer multiply of the significands plus an exponent add and a normalisation, so a 32-bit float multiply (24-bit significand) costs about as much as a 32-bit integer one, and a float add costs much more than an integer add (alignment shifters and normalisation): 0.9 pJ against 0.1 pJ.
The MX dot product's error has two sources. The elements' rounding, relative to each block's scale, is at most half a step of the element format; and the sum of products, which the spec lets implementations compute at any internal precision. With the shared scales factored out, the products of two E2M1 elements are exact in a few bits, so a wide enough accumulator makes the reduction exact, as it is here.
Code
The four pipelines, one step each, cut from src/lib/num/model.ts:
ref += a[i]! * b[i]!;
s32 = roundTo(s32 + roundTo(a[i]! * b[i]!, "fp32"), "fp32");
s16 = roundTo(s16 + roundTo(a[i]! * b[i]!, "fp16"), "fp16");
const qa = quantInt(a[i]!, pa);
const qb = quantInt(b[i]!, pb);
acc8 += qa * qb;
const xa = ma.values[i]! / ea;
const xb = mb.values[i]! / eb;
accmx += xa * xb;