3 ms·
Re quantization - you should be fine with INT8. So that's 1.5GB. And after pruning you will probably be able to compress it down to 500MB. To clarify - we are t
by p1esk 7y ago
Re quantization - you should be fine with INT8. So that's 1.5GB. And after pruning you will probably be able to compress it down to 500MB. To clarify - we are talking about distributing a trained model, right? You don't need to train it in reduced precision. Both TF and Pytorch provide tools to quantize a model after training.
- sillysaurusx 7y agoDoesn't quantization harm model quality? It's hard to quantify how much damage quantization does, but natural language generation is very subtle.
- p1esk 7y agoWhen you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for inference, 8 bits should be enough. It mostly depends on the model size and if there's enough learning capacity for the dataset. I suspect GPT2 1.5B is very overparametrized, so in the future it will probably be considered similar to VGG, which is highly compressible without any accuracy loss. I wouldn't be surprised if you can finetune GPT2 using only 3-4 bits for both weights and activations, with no quality loss. By 'finetune' here I mean finetuning for quantization, not transfer learning.