He says this in the article: if you ever want to jam a multi-trillion-parameter model into a phone app or a Raspberry Pi, you must quantize. I've seen some quan

He says this in the article: if you ever want to jam a multi-trillion-parameter model into a phone app or a Raspberry Pi, you must quantize. I've seen some quantization go from doubles to bytes (64 bits to 8) per weight, reducing the RAM requirement by 8x. A simple quantization (I'm sure there are much better ones) is to round everything the nearest 1/255th of your number range, then multiply by 255. So your resolution is (max-min)/255. You also store the min and max so you can reverse it, of course. Say you're trying to quantize these sets of numbers: 1. { -1.4, 0.8, 2.7, 7.3 } : With a range of 8.7, you have a resolution of 0.034. This set quantizes to { 0, 64, 120, 255 }. 2. { -1400, 800, 2700, 7300 } : Resolution 34.1, quantizing to the same as the above { 0, 64, 120, 255 }. 3. { -0.008, -0.001, 0.009, 0.019 } : resolution 0.000106. This set quantizes to { 0, 66, 161, 255 }. 3. { -1.4, 0.8, 2.7, 7329 } : Resolution 28.7. This set quantizes to {0, 0, 0, 255 }. Oops -- we can no longer tell most of our weights apart. You can see how this quantization works really well when all the numbers are close together, regardless of their absolute scale. Major outliers completely mess up the entire system. You can make more and more complicated quantization algorithms, but those will always come with tradeoffs. The best option would be to tame your weights so that they are again close together.