2 ms·

This is also not reproducible. Change the wording slightly and you get the 'right' answers from 3.5. https://twitter.com/iandanforth/status/1637277208828944385

• drew-y 4y ago
  I'm confused. The claim is that GPT-4 has a world model, not necessarily that GPT-3.5 does not. All that tweet shows is that GPT-3.5 can also give a correct answer. How does this invalidate the author's original claim?
  • ethanbond 4y ago
    It’s proof that it’s a bad test. No strong reason to think it tells you anything in either case.
• namuol 4y ago
  Here’s my anecdata: Tried it with a few different wordings and GPT-3.5 got it wrong every time. 0/3. GPT-4 got it right 2/3 of the times. But if you really want meaningful data I’d suggest reading the paper and specifically paying attention to the human-level performance (95%) of GPT-4 on the HellaSwag data set which includes a lot of these sort of “common sense reasoning” problems. No other LLMs or specialized models have been able to achieve this kind of performance; the state of the art at the time of publication was around 85%.