4 ms·
FWIW, the “HellaSwag” benchmark from the GPT-4 paper consists of many such “riddles” and it achieves ~95% accuracy (essentially par with humans). The next best
by namuol 4y ago
FWIW, the “HellaSwag” benchmark from the GPT-4 paper consists of many such “riddles” and it achieves ~95% accuracy (essentially par with humans). The next best model (LLM or otherwise) was around 85%. It is pretty remarkable.
GPT-4 paper: https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf
Original HellaSwag paper from 2019: https://arxiv.org/pdf/1905.07830.pdf https://arxiv.org/pdf/1905.07830.pdf
- mtlmtlmtlmtl 4y agoSure, papers are great. Happy to read papers. I just have no patience left for this endless stream of garbage from Twitter. It's so pointless.
- namuol 4y agoIf you’re an insider it’s frustrating to see a lot of “bad science” out there among laypeople but that’s what assimilation looks like for new technologies. But I couldn’t agree more about Twitter, in general… (I don’t consider myself an “insider” by any stretch, to be clear)
- YeGoblynQueenne 4y agoEvery time someone on HN invokes an LLM's scoring on a benchmark as if it meant something, I'll post a link to this paper: What Will it Take to Fix Benchmarking in Natural Language Understanding? https://aclanthology.org/2021.naacl-main.385/ https://aclanthology.org/2021.naacl-main.385/ To summarise, it's been quite some time now that language models have been beating NLU benchmarks left and right, but that still doesn't tell us anything about their true capabilities, because those benchmarks don't measure what they purport to be measuring. Or, to rephrase in a way that shows how much I love pattern matching: you keep saying this word, "benchmark". I don't think it means, what you think it means.
- namuol 4y agoThank you for the perspective; I'm an ML outsider so I'm not familiar with the debates going on within the field. That said, I was didn't mean to suggest that these benchmarks were measuring anything other than ...well, the performance of the benchmark. I was trying to give the parent commenter something less "subjective" to read that I think connects with the somewhat squishier Twitter discussion.
- YeGoblynQueenne 4y agoI think that's a great motivation. But now you can also read the paper I linked above and be more informed about the discussion around benchmarks for NLU (Natural Language Understanding). You can also follow the references in that paper. That will then help you know how to contribute to discussions, in particular when it comes to pointing interlocutors to results, without increasing the amount of noise. We sorely need that right now, so please consider becoming better informed.
- sebzim4500 4y agoThat paper makes some very fair criticisms, I think that GPT-4's performance in exams (bar exam, LSAT, AP bio, etc.) is far more impressive than its ability to beat benchmarks. My concern is that fragments of those exams might be found in the training set, and it is hard to see how to account for that. Even if we use exams ostensibily written more recently than the training data was collected, exam boards do tend to reuse questions (sometimes verbatim, sometimes slightly modified).
- YeGoblynQueenne 4y agoThis was posted on HN today (not by me!): https://news.ycombinator.com/item?id=35245626 https://news.ycombinator.com/item?id=35245626 It talks about bar exams and contamination, so spot on :)