When you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for

When you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for inference, 8 bits should be enough. It mostly depends on the model size and if there's enough learning capacity for the dataset. I suspect GPT2 1.5B is very overparametrized, so in the future it will probably be considered similar to VGG, which is highly compressible without any accuracy loss. I wouldn't be surprised if you can finetune GPT2 using only 3-4 bits for both weights and activations, with no quality loss. By 'finetune' here I mean finetuning for quantization, not transfer learning.