2 ms·
I think it would be interesting to see benchmarks on that. I tested that example on chatGPT and it was correct, with a really good explanation even when I modi
by nl 4y ago
I think it would be interesting to see benchmarks on that.
I tested that example on chatGPT and it was correct, with a really good explanation even when I modified the question away from the example on Wikipedia.
Some related results (eg https://arxiv.org/pdf/2206.14576.pdf https://arxiv.org/pdf/2206.14576.pdf - see "Causal reasoning: Interventions after passive observations" section) indicate it would be competitive.
- nl 4y agoActually https://arxiv.org/pdf/2207.07051.pdf https://arxiv.org/pdf/2207.07051.pdf is DeepMind doing Wason (and other) tests on Chincilla and they find it scores between 40% and 60% on "realistic" and "shuffled realistic" Wason tests (I think. It's hard to read. See Figure 5).