quantization
To begin with
A model can weigh tens of gigabytes. To fit it into less memory, some of its numbers are written with less precision, like rounding a price.
The facts
Quantization represents some values with lower precision: for example weights stored as 8- or 4-bit integers with scales, instead of 16-bit floating-point numbers. The model gets smaller, but its values are no longer exactly the same. Speed gains depend on the hardware, and quality must be checked. Some methods use calibration.
Key takeaway
“4-bit” doesn’t say it all: you need to know what is quantized, and how. Smaller does not mean identical.
The stories about it
The notebook opens soon.The first stories are on their way.