This is not really true anymore, as the "consumer GPU" backends have very flexible quantization. Llama.cpp has like a dozen steps between 2 and 6 bit, and exLla

This is not really true anymore, as the "consumer GPU" backends have very flexible quantization. Llama.cpp has like a dozen steps between 2 and 6 bit, and exLlamav2 will literally do an arbitrary decimal bpw. It sort of matters with bigger models trying to squeeze into a server GPU, with the (currently) inflexible vLLM 4-bit quantization. I think its just a standard set by Llama.