4 ms·

AMC 10, AMC 12 (2023) results in Table 2 suggest Claude 3 Opus is better than the average high school students who participate in these math competitions. These

AMC 10, AMC 12 (2023) results in Table 2 suggest Claude 3 Opus is better than the average high school students who participate in these math competitions. These math problems are not straightforward and cannot be solve by simply memorizing formulas. Most of the students are also quite good at math. The student averages are 64.4 and 61.5 respectively, while Opus 3 scores are 72 and 63. Probably fewer than 100,000 students take part in AMC 12 out of possibly 3-4 million grade-12 students. Assume just half of the top US students participate, the average score of AMC would represent the top 2-4% of US high school students. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf#page7 https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...

usaar333 3y ago
Interestingly, math olympiad problems (using ones I wrote myself years ago so outside training data) seem to be better in Claude 3. Almost everything else though I've tested seems better in GPT-4.
sebzim4500 3y ago
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
- whymauri 3y ago
  Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
- nopinsight 3y ago
  Most likely, it’s less generally smart than the top 2-4% of US high school students. It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance. Since the test questions were unseen, the result still suggests the person has some intelligence though. Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.

usaar333 3y ago

Interestingly, math olympiad problems (using ones I wrote myself years ago so outside training data) seem to be better in Claude 3. Almost everything else though I've tested seems better in GPT-4.

sebzim4500 3y ago

The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.

whymauri 3y ago
Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
nopinsight 3y ago
Most likely, it’s less generally smart than the top 2-4% of US high school students. It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance. Since the test questions were unseen, the result still suggests the person has some intelligence though. Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.