From what I can tell, the quantization being used there is extremely naive. If they were using techniques like GPTQ, presumably it would work much better. htt

From what I can tell, the quantization being used there is extremely naive. If they were using techniques like GPTQ, presumably it would work much better. https://arxiv.org/pdf/2210.17323.pdf https://arxiv.org/pdf/2210.17323.pdf