4 ms·
I question if perplexity on wikitext is enough, I wonder if someone has checked how well perplexity correlates with https://github.com/google/BIG-bench https://
by georgehotz 3y ago
I question if perplexity on wikitext is enough, I wonder if someone has checked how well perplexity correlates with https://github.com/google/BIG-bench https://github.com/google/BIG-bench
Fundamentally, I'm not sure if perplexity on a dataset that's pretty much leaked everywhere will properly measure the task performance of LLMs. But open to being wrong on this.
- lhl 3y agoForgot to mention that Fabrice Bellard has run lm-eval on a range of functional tests (lambada, hellaswag, winogrande, piqa, coqa) for q4, q8 (and unquantized on smaller) models: https://bellard.org/ts_server/ https://bellard.org/ts_server/ Losses appear to be in-line (possibly less) than increases in perplexity. My intuition is that the dataset you use for testing perplexity (c4, wikitext2, wikitext103, whatever) shouldn't matter so much as long as its a representative sample of data it's read. You're calculating cross entropy loss, so you want to have data you know it's read to see how much each quantization gets more wrong.
- georgehotz 3y agoOh this is nice! Is this the same quantization method as GGML? It's just described as "Efficient custom 8 bit and 4 bit quantization" And thanks for the https://pypi.org/project/lm-eval/ https://pypi.org/project/lm-eval/ pointer, have to get this in tinygrad.