2 ms·
Sorry for the confusing wording. The model is 1.5B. But every 1.5B model is 5.6GB of data. (1,558 million params * sizeof(float32) = 5.6GB) Quantization is a g
by sillysaurusx 7y ago
Sorry for the confusing wording. The model is 1.5B. But every 1.5B model is 5.6GB of data. (1,558 million params * sizeof(float32) = 5.6GB)
Quantization is a good idea, but I've had some bad experiences with bfloat16. (We couldn't get loss to decrease beyond a certain point when using bfloat16.) But that might have been an artifact of training. Still, I'd rather not harm the model if possible. It's hard to get a sense of perceptual quality, and I'm not convinced that validation tests are enough to capture the full nuances of GPT-2.
You're right that a single server would do the trick in this case. It's just so much easier to write a notebook with the logic in it than to set up a server, keep track of sessions, make read-eval-print logic, add error handling, etc. I was hoping to throw a couple hundred dollars at the problem of bandwidth hosting rather than spend time making a server.
A server also implies a fixed $300/mo cost, which is pretty expensive. After the initial demo, we'd probably end up turning the server off. I dislike the idea of the demo breaking after a month or two.
- p1esk 7y agoRe quantization - you should be fine with INT8. So that's 1.5GB. And after pruning you will probably be able to compress it down to 500MB. To clarify - we are talking about distributing a trained model, right? You don't need to train it in reduced precision. Both TF and Pytorch provide tools to quantize a model after training.
- sillysaurusx 7y agoDoesn't quantization harm model quality? It's hard to quantify how much damage quantization does, but natural language generation is very subtle.
- p1esk 7y agoWhen you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for inference, 8 bits should be enough. It mostly depends on the model size and if there's enough learning capacity for the dataset. I suspect GPT2 1.5B is very overparametrized, so in the future it will probably be considered similar to VGG, which is highly compressible without any accuracy loss. I wouldn't be surprised if you can finetune GPT2 using only 3-4 bits for both weights and activations, with no quality loss. By 'finetune' here I mean finetuning for quantization, not transfer learning.