3 ms·

Interesting read. As others have said, it will be much more convincing with some experimental numbers. I'm confused what his goal is though: I could imagine s

by ks2048 3y ago

Interesting read. As others have said, it will be much more convincing with some experimental numbers. I'm confused what his goal is though: I could imagine some theoretical reason to add a 1 there, but he starts by saying this can lead to smaller, more compactable models. Is he talking about the size the compressed weights? or pruning to a smaller model? or resistant more quantization? Parts of the essay seemed to throw me off track, because I'm not sure if they are relevant at all to the proposal (eg the of the initial embedding and how many bits it would take the store the vocab size, etc).

feoren 3y ago
He says this in the article: if you ever want to jam a multi-trillion-parameter model into a phone app or a Raspberry Pi, you must quantize. I've seen some quantization go from doubles to bytes (64 bits to 8) per weight, reducing the RAM requirement by 8x. A simple quantization (I'm sure there are much better ones) is to round everything the nearest 1/255th of your number range, then multiply by 255. So your resolution is (max-min)/255. You also store the min and max so you can reverse it, of course. Say you're trying to quantize these sets of numbers: 1. { -1.4, 0.8, 2.7, 7.3 } : With a range of 8.7, you have a resolution of 0.034. This set quantizes to { 0, 64, 120, 255 }. 2. { -1400, 800, 2700, 7300 } : Resolution 34.1, quantizing to the same as the above { 0, 64, 120, 255 }. 3. { -0.008, -0.001, 0.009, 0.019 } : resolution 0.000106. This set quantizes to { 0, 66, 161, 255 }. 3. { -1.4, 0.8, 2.7, 7329 } : Resolution 28.7. This set quantizes to {0, 0, 0, 255 }. Oops -- we can no longer tell most of our weights apart. You can see how this quantization works really well when all the numbers are close together, regardless of their absolute scale. Major outliers completely mess up the entire system. You can make more and more complicated quantization algorithms, but those will always come with tradeoffs. The best option would be to tame your weights so that they are again close together.
- ks2048 3y ago
  Thanks for the reply, that makes sense. It's not immediately clear why the modified softmax (allowing output of 0) will "tame the weights", but I need to read the blog post more closely and think about it...

by ks2048 3y ago

feoren 3y ago

He says this in the article: if you ever want to jam a multi-trillion-parameter model into a phone app or a Raspberry Pi, you must quantize. I've seen some quantization go from doubles to bytes (64 bits to 8) per weight, reducing the RAM requirement by 8x. A simple quantization (I'm sure there are much better ones) is to round everything the nearest 1/255th of your number range, then multiply by 255. So your resolution is (max-min)/255. You also store the min and max so you can reverse it, of course. Say you're trying to quantize these sets of numbers: 1. { -1.4, 0.8, 2.7, 7.3 } : With a range of 8.7, you have a resolution of 0.034. This set quantizes to { 0, 64, 120, 255 }. 2. { -1400, 800, 2700, 7300 } : Resolution 34.1, quantizing to the same as the above { 0, 64, 120, 255 }. 3. { -0.008, -0.001, 0.009, 0.019 } : resolution 0.000106. This set quantizes to { 0, 66, 161, 255 }. 3. { -1.4, 0.8, 2.7, 7329 } : Resolution 28.7. This set quantizes to {0, 0, 0, 255 }. Oops -- we can no longer tell most of our weights apart. You can see how this quantization works really well when all the numbers are close together, regardless of their absolute scale. Major outliers completely mess up the entire system. You can make more and more complicated quantization algorithms, but those will always come with tradeoffs. The best option would be to tame your weights so that they are again close together.

ks2048 3y ago
Thanks for the reply, that makes sense. It's not immediately clear why the modified softmax (allowing output of 0) will "tame the weights", but I need to read the blog post more closely and think about it...