Correct, people in the model optimization world will claim all day that it only marginally impacts performance, but I’ve extensively played with quantization an

Correct, people in the model optimization world will claim all day that it only marginally impacts performance, but I’ve extensively played with quantization and pruning methods, and can report that they do cripple models in ways that the authors either didn’t notice or on purpose omitted from their SOTA benchmark chasing. Most claims of models being even equal to GPT-3.5 are also significantly overblown. I haven’t seen one yet below 70 billion parameters which even comes close. Nothing is free in this world.