3 ms·
> "GPT-4 performed best in zero-shot settings, reaching nearly 80% ToM accuracy, but still fell short of the 87% human accuracy on the test set. However, when s
by ftxbro 4y ago
> "GPT-4 performed best in zero-shot settings, reaching nearly 80% ToM accuracy, but still fell short of the 87% human accuracy on the test set. However,
when supplied with prompts for in-context learning, all RLHF-trained LLMs exceeded 80% ToM accuracy, with GPT-4 reaching 100%."
GPT-4 exceeded average adult human theory of mind when the simple prompt "Let's think step by step:" was added. It got every question right when the prompt included a couple of example puzzles.
Average human ability was from the following source: "To measure humans’ performance in ToM and Photo scenarios, we recruited 125 online participants through the Qualtrics platform. Participants were 18 to 65 years old, native English speakers, and located in the United States."
It looks like we will need more new benchmarks by the time GPT-5 comes!