2 ms·
Boosting Theory-of-Mind Performance in Large Language Models via Prompting
- ftxbro 4y ago> "GPT-4 performed best in zero-shot settings, reaching nearly 80% ToM accuracy, but still fell short of the 87% human accuracy on the test set. However, when supplied with prompts for in-context learning, all RLHF-trained LLMs exceeded 80% ToM accuracy, with GPT-4 reaching 100%." GPT-4 exceeded average adult human theory of mind when the simple prompt "Let's think step by step:" was added. It got every question right when the prompt included a couple of example puzzles. Average human ability was from the following source: "To measure humans’ performance in ToM and Photo scenarios, we recruited 125 online participants through the Qualtrics platform. Participants were 18 to 65 years old, native English speakers, and located in the United States." It looks like we will need more new benchmarks by the time GPT-5 comes!
- ftxbro 4y agoAn author of the paper is answering questions about it on twitter https://twitter.com/Shima_RM_/status/1651467500356538368 https://twitter.com/Shima_RM_/status/1651467500356538368
- ftxbro 4y agoIf I would have put the title as "When prompted with 'Let's think step by step:', GPT-4 exceeds human theory of mind" then it would have a hundred upvotes and comments by now. I'm still working on my clickbait game.