3 ms·
Doesn't quantization harm model quality? It's hard to quantify how much damage quantization does, but natural language generation is very subtle.
by sillysaurusx 7y ago
Doesn't quantization harm model quality?
It's hard to quantify how much damage quantization does, but natural language generation is very subtle.
- p1esk 7y agoWhen you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for inference, 8 bits should be enough. It mostly depends on the model size and if there's enough learning capacity for the dataset. I suspect GPT2 1.5B is very overparametrized, so in the future it will probably be considered similar to VGG, which is highly compressible without any accuracy loss. I wouldn't be surprised if you can finetune GPT2 using only 3-4 bits for both weights and activations, with no quality loss. By 'finetune' here I mean finetuning for quantization, not transfer learning.