You definitely want to use 4bit quants at minimum. https://arxiv.org/abs/2505.24832 https://arxiv.org/abs/2505.24832 LLMs usually have about 3.6 bits of data

You definitely want to use 4bit quants at minimum. https://arxiv.org/abs/2505.24832 https://arxiv.org/abs/2505.24832 LLMs usually have about 3.6 bits of data per parameter. You're losing a lot of information if quantized to 2 bits. 4 bit quants are the sweet spot where there's not much quality loss.