3 ms·
Current generation LLMs get much easier problems wrong, problems that people have no trouble with.
by civilized 4y ago
Current generation LLMs get much easier problems wrong, problems that people have no trouble with.
- nl 4y agoI think it would be interesting to see benchmarks on that. I tested that example on chatGPT and it was correct, with a really good explanation even when I modified the question away from the example on Wikipedia. Some related results (eg https://arxiv.org/pdf/2206.14576.pdf https://arxiv.org/pdf/2206.14576.pdf - see "Causal reasoning: Interventions after passive observations" section) indicate it would be competitive.
- nl 4y agoActually https://arxiv.org/pdf/2207.07051.pdf https://arxiv.org/pdf/2207.07051.pdf is DeepMind doing Wason (and other) tests on Chincilla and they find it scores between 40% and 60% on "realistic" and "shuffled realistic" Wason tests (I think. It's hard to read. See Figure 5).