Here’s my anecdata: Tried it with a few different wordings and GPT-3.5 got it wrong every time. 0/3. GPT-4 got it right 2/3 of the times. But if you really want

Here’s my anecdata: Tried it with a few different wordings and GPT-3.5 got it wrong every time. 0/3. GPT-4 got it right 2/3 of the times. But if you really want meaningful data I’d suggest reading the paper and specifically paying attention to the human-level performance (95%) of GPT-4 on the HellaSwag data set which includes a lot of these sort of “common sense reasoning” problems. No other LLMs or specialized models have been able to achieve this kind of performance; the state of the art at the time of publication was around 85%.