3 ms·
LLM quantization severely damages model quality and perplexity
- brucethemoose2 3y agoThat is just one quantization method, and its really Q3_K_L and smaller (the first four points on the graph for each model) that butcher the model.
- rahul07_ 3y agoCheck out this: https://youtube.com/playlist?list=PL80kAHvQbh-pT4lCkDT53zT8DKmhE0idB&si=eMwBFE3_hUmTED-L https://youtube.com/playlist?list=PL80kAHvQbh-pT4lCkDT53zT8D... might be helpful
- 0x008 3y agoWhat is this title? It's a pull request that contains a great investigation which outlines the effects which various mixes of quantization have in one metric. The article draws no conclusion and the title is "k-quants".