It's because you can quantize to different extents, at the expense of a small amount of accuracy. The original model probably needs much more vRAM than most peo

It's because you can quantize to different extents, at the expense of a small amount of accuracy. The original model probably needs much more vRAM than most people will actually end up needing in prod.