Quantization, finally explained
I spent way too long thinking quantization was some deep, scary thing. It isn't. The whole idea sits on top of one fact that nobody bothers to say out loud: an AI is secretly just a list of numbers. Once that clicks, the rest is basically rounding.
Let me build it up slowly. You can poke at each piece as we go.
It's just a big board of knobs
Say you're building an AI to decide whether an email is spam. It looks for clues. Each clue gets a number, its weight, that says how much that clue counts toward "yeah, this is spam." To make a call, the AI multiplies each clue by its weight and adds everything up.
Flip the clues on and off and watch the score move:
Those weights (0.9, 0.6, 0.3 and so on) are the AI. Change them and it behaves differently.
So where do those numbers come from? Nobody types them in. Training is just the computer finding good knob settings by itself: start random, make a guess, nudge every knob a tiny bit toward the right answer, then do that a few billion times. Whatever the knobs settle on is the finished model.
A spam detector needs four knobs. Something like ChatGPT needs billions of them. That's honestly the only difference.
Billions of long numbers get heavy
Here's the catch. Every knob is stored as a precise number like 0.6817. Now picture billions of those. The file gets so big it needs expensive, power-hungry GPUs just to sit in memory.
-0.0192, 0.1216, -0.7691, -0.5120, -0.1136, 0.0128, -0.8429, 0.7577, 0.1788, -0.1784, -0.4311, 0.5625, 0.3149, -0.8468, -0.0029, -0.5298, 0.3439, -0.6840, 0.1208, -0.0255, -0.6476, -0.8999, 0.8782, 0.9398, -0.7585, -0.1115, 0.9114, -0.6217, -0.1615, 0.1006, 0.3437, -0.4383, -0.6592, 0.9712, -0.1087, -0.5348, 0.0941, -0.3392, -0.6928, 0.5457, -0.2966, -0.2999, -0.8407, 0.8525, -0.5227, 0.7683, -0.0586, -0.5279, 0.2059, -0.1965, 0.5178, 0.3056, -0.7280, -0.5523, -0.0993, -0.8196, -0.9047, -0.1755, 0.3619, 0.2938, 0.9104, -0.3501, -0.1235, -0.0084, -0.1223, 0.6769, -0.0985, -0.1214, -0.4719, -0.5266, 0.4861, -0.7364, -0.4593, 0.5333, 0.7168, 0.9367, 0.8537, 0.7208, 0.4948, 0.2806, 0.1091, -0.9518, -0.3527, -0.2850, -0.5697, -0.2129, 0.3718, 0.4745, -0.4666, 0.9129, -0.9860, -0.2804, 0.1715, -0.1178, -0.8211, 0.9811, -0.0875, 0.7446, 0.1750, -0.0618, -0.7498, 0.5864, 0.7925, -0.4711, -0.9512, -0.6911, 0.4141, 0.3954, 0.3206, 0.7392, -0.5403, 0.6611, -0.6627, -0.9252, -0.8364, -0.9580, 0.0742, 0.6615, 0.6865, -0.7088, -0.0120, -0.8959, -0.4409, -0.0234, -0.8718, -0.2887, -0.5394, -0.4079, 0.5811, -0.7267, -0.6708, -0.9982, 0.2480, -0.8273, -0.1932, -0.7819, -0.0439, -0.1707, -0.9831, 0.5126, …
…and this goes on for billions more. Long, precise numbers are expensive to store.
Which raises a pretty obvious question. Do we actually need every single number to be that precise? Turns out, not really.
Quantization is just rounding
That's the whole move. Walk down the giant list of knobs and round each one to something shorter. Shorter numbers take up less room, so the model gets smaller and faster.
Smaller, faster, cheaper. A little less exact, but usually so little you never notice.
What "8-bit" and "4-bit" actually mean
There's one rule to hold onto: more bits means more allowed values, which means gentler rounding.
The easiest way I found to feel this is rounding a price. The fewer round numbers you're allowed to use, the further off you end up. Same thing happens to the weights.
Flip between 8-bit, 4-bit and 2-bit below. The same buttons also drive the size-versus-accuracy tradeoff, so you can watch what each choice actually costs:
8-bit is like rounding to the nearest dollar. Loads of options, barely any loss.
Half the original size.
Practically identical to the full model.
Here's the nice part. Across billions of weights, all those tiny rounding errors mostly cancel out, so 8-bit barely hurts while cutting the size in half. Push to 4-bit and you save more for a small drop in quality. Go all the way to 2-bit and the errors stop cancelling, and the model starts giving you nonsense.
So that's really it. No magic, no scary math.
An AI is a huge list of numbers. Quantization rounds them off to save room. Fewer bits means fewer values to round to, so you get something smaller and faster that eventually turns dumb if you push it too far.