It quantizes 32-bit floating point weights to 4-bit integers so that models fit into memory and run reasonably fast on a CPU by using the processor’s vector ext

It quantizes 32-bit floating point weights to 4-bit integers so that models fit into memory and run reasonably fast on a CPU by using the processor’s vector extensions.