Quantization aware training produces good results for fp8 quantization. Someone who is going all the way down to 4 bit quantization in a production environment

Quantization aware training produces good results for fp8 quantization. Someone who is going all the way down to 4 bit quantization in a production environment must be really desperate, because it makes batch processing harder. The additional dequantization operations slow the inference down. Dequantizing fp8 to bfloat16 is just a few bit shifts plus "and" masking.