How few bits can a number have?

Models are trained in 16-bit formats and served in 8, 6 or 4. An FP8 E4M3 number has 4 significant bits and nothing above 448; FP4 has 2 and stops at 6. This site is about what that costs and the tricks that make it work: rounding, accumulating in wider formats, scales shared by blocks of numbers, and quantisers that choose their integers with care.

Each chapter is built around an animation, and every frame is computed by a small numerics library that reproduces numpy and ml_dtypes bit for bit and follows the IEEE 754 and OCP FP8 and MX specifications.