The short answer is yes, it has a big impact on output quality, all other things being equal. There are techniques like "quantization-aware training" [1] which

The short answer is yes, it has a big impact on output quality, all other things being equal. There are techniques like "quantization-aware training" [1] which aim to reduce the impact. However, if your benchmark is "output quality for a fixed amount of compute" (e.g. running real time voice recognition on a smartphone) a larger model that's been quantized might perform better than a smaller model that hasn't. [1] https://www.tensorflow.org/model_optimization/guide/quantization/training https://www.tensorflow.org/model_optimization/guide/quantiza...