4 ms·
MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is
by bugglebeetle 3y ago
MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.
- hackerlight 3y agoHow can they avoid the contents from leaking into the training set somewhere in their internet scrape?
- jasonjmcghee 3y agoI still don't trust benchmarks, but they've come a long way. It's genuinely outperforming GPT4 in my manual tests.
- rafaelero 3y agoMMLU is garbage. A lot of incorrect answers there.
- bugglebeetle 3y agoAnd yet it’s still a good indicator of general performance. Any model that scores under GPT-4 on that benchmark, but above it in other, tends to be worse overall.