3 ms·
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
by sebzim4500 3y ago
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
- whymauri 3y agoIs it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
- nopinsight 3y agoMost likely, it’s less generally smart than the top 2-4% of US high school students. It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance. Since the test questions were unseen, the result still suggests the person has some intelligence though. Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.