Forgot to mention that Fabrice Bellard has run lm-eval on a range of functional tests (lambada, hellaswag, winogrande, piqa, coqa) for q4, q8 (and unquantized o

Forgot to mention that Fabrice Bellard has run lm-eval on a range of functional tests (lambada, hellaswag, winogrande, piqa, coqa) for q4, q8 (and unquantized on smaller) models: https://bellard.org/ts_server/ https://bellard.org/ts_server/ Losses appear to be in-line (possibly less) than increases in perplexity. My intuition is that the dataset you use for testing perplexity (c4, wikitext2, wikitext103, whatever) shouldn't matter so much as long as its a representative sample of data it's read. You're calculating cross entropy loss, so you want to have data you know it's read to see how much each quantization gets more wrong.