4 ms·
The article is right, 8-bit (and especially 4-bit) is atypical for deep learning models and highly depends on the amount of parameters (larger model can handle
by f_devd 3y ago
The article is right, 8-bit (and especially 4-bit) is atypical for deep learning models and highly depends on the amount of parameters (larger model can handle more quantization) and can even depend on specific training hyperparameters (mainly dropout & weight decay which can induce sparsity)
- int_19h 3y agoThing is, even when the impact from 4-bit is substantial, the larger parameter count it allows on the same hardware more than makes up for it. E.g. llama-30b is better at 4-bit than any derivative of llama-13b, no matter how fine-tuned or quantized.