4 ms·
> how everyone is in this mad quantization rush but nobody's putting up benchmarks to show that it works (tinybox is resolutely supporting non quantized LLaMA)
by lhl 3y ago
> how everyone is in this mad quantization rush but nobody's putting up benchmarks to show that it works (tinybox is resolutely supporting non quantized LLaMA)
I don't think this is true. llama.cpp has historically been very conscientious about benchmarking perplexity. Here's a detailed chart of baseline FP16 vs the new k-quants: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
While most evals aren't currently evaluating performance between quantized models, there are two evals that are:
* Gotzmann LLM Score: https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro46YysCldKY7Ub2wcX5H1jCQ/edit#gid=0 https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro...
* llm-jeopardy: https://github.com/aigoopy/llm-jeopardy https://github.com/aigoopy/llm-jeopardy - You can see that the same Airoboros 65B model goes from a score of 81.62% to 80.00% going from an 8_0 to 5_1 quant, and 5_1 solidly beats out the 33B 8_0, as expected.
Also, GPTQ, SPQR, AWQ, SqueezeLLM all have arXiv papers and every single team is running their own perplexity tests.
Now, that being said, every code base seems to be calculating perplexity slightly differently. I recently have been working on trying to decode them all for apples-to-apples comparisons between implementations.
- georgehotz 3y agoI question if perplexity on wikitext is enough, I wonder if someone has checked how well perplexity correlates with https://github.com/google/BIG-bench https://github.com/google/BIG-bench Fundamentally, I'm not sure if perplexity on a dataset that's pretty much leaked everywhere will properly measure the task performance of LLMs. But open to being wrong on this.
- lhl 3y agoForgot to mention that Fabrice Bellard has run lm-eval on a range of functional tests (lambada, hellaswag, winogrande, piqa, coqa) for q4, q8 (and unquantized on smaller) models: https://bellard.org/ts_server/ https://bellard.org/ts_server/ Losses appear to be in-line (possibly less) than increases in perplexity. My intuition is that the dataset you use for testing perplexity (c4, wikitext2, wikitext103, whatever) shouldn't matter so much as long as its a representative sample of data it's read. You're calculating cross entropy loss, so you want to have data you know it's read to see how much each quantization gets more wrong.
- georgehotz 3y agoOh this is nice! Is this the same quantization method as GGML? It's just described as "Efficient custom 8 bit and 4 bit quantization" And thanks for the https://pypi.org/project/lm-eval/ https://pypi.org/project/lm-eval/ pointer, have to get this in tinygrad.