5 ms·

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descripti

The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... Table 1: Evaluation results (more datasets than in the blog post) https://twitter.com/karinanguyen_/status/1764666528220557320 https://twitter.com/karinanguyen_/status/1764666528220557320 APPS dataset: https://huggingface.co/datasets/codeparrot/apps https://huggingface.co/datasets/codeparrot/apps APPS dataset paper: https://arxiv.org/abs/2105.09938v3 https://arxiv.org/abs/2105.09938v3

eschluntz 3y ago
(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)
- RivieraKid 3y ago
  What's your estimate of how much does it increase a typical programmer's productivity?
- bwanab 3y ago
  Sounds almost recursive.
  - deleted 3y ago
    [deleted]
- zellyn 3y ago
  Do y'all have an explanation for why Haiku outperforms Sonnet for code?
  - razodactyl 3y ago
    Seems like they optimised this model with coding datasets for use in Copilot-like assistants with the low latency advantage. Additionally, I wonder if an alternate dataset is provided based on model size as to not run into issues with model forgetting.
- mkakkori 3y ago
  Interested to try this out as well! What is your setup for integrating Opus to you development workflow?
nopinsight 3y ago
“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitter.com/idavidrein/status/1764675668175094169 https://twitter.com/idavidrein/status/1764675668175094169 GPQA: A Graduate-Level Google-Proof Q&A Benchmark https://arxiv.org/abs/2311.12022 https://arxiv.org/abs/2311.12022
- deleted 3y ago
  [deleted]
- wbarber 3y ago
  What's to say this isn't just a demonstration of memorization capabilities? For example, rephrasing the logic of the question or even just simple randomizing the order of the multiple choice answers to these questions often dramatically impacts performance. For example, every model in the Claude 3 family repeats the memorized solution to the lion, goat, wolf riddle regardless of how I modify the riddle.
  - apsec112 3y ago
    If the answers were Googleable, presumably smart humans with Internet access wouldn't do barely better than chance?
  - msikora 3y ago
    GPT-4 used to have the same issue with this puzzle early on but they've fixed since then (the fix was like mid 2023).
    - Jensson 3y ago
      The fix is to train it on this puzzle and variants of it, meaning it memorized this pattern. It still fail similar puzzles if given in a different structure, until they feed it that structure as well. LLMs is more like programming than human intelligence, they need to program in the solution to these riddles very much like we did expert systems in the past. The main new thing we get here is natural language compatibility, but other than that the programming seems to be the same or weaker than old programming of expert systems. The other big thing is that there is already a ton of solutions on the web coded in natural language, such as all the tutorials etc, so you get all of those programs for free. But other than that these LLMs seems to have exactly the same problems and limitations and strengths as expert systems. They don't generalize in a flexible enough manner to solve problems like a human.
vok 3y ago
APPS has 3 subsets by difficulty level: introductory, interview, and competition. It isn't clear which subset Claude 3 was benchmarked on. Even if it is just "introductory" it is still pretty good, but it would be good to know.
- nopinsight 3y ago
  Since they don’t state it, does it mean they tested it on the whole test set? If that’s the case, and we assume for simplicity that Opus solves all Intro problems and none of the Competition problems, it’d have solved 83%+ of the Interview level problems. (There are 1000/3000/1000 problems in the test set in each level). It’d be great if someone from Anthropic provides an answer though.
nopinsight 3y ago
AMC 10, AMC 12 (2023) results in Table 2 suggest Claude 3 Opus is better than the average high school students who participate in these math competitions. These math problems are not straightforward and cannot be solve by simply memorizing formulas. Most of the students are also quite good at math. The student averages are 64.4 and 61.5 respectively, while Opus 3 scores are 72 and 63. Probably fewer than 100,000 students take part in AMC 12 out of possibly 3-4 million grade-12 students. Assume just half of the top US students participate, the average score of AMC would represent the top 2-4% of US high school students. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf#page7 https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...
- usaar333 3y ago
  Interestingly, math olympiad problems (using ones I wrote myself years ago so outside training data) seem to be better in Claude 3. Almost everything else though I've tested seems better in GPT-4.
- sebzim4500 3y ago
  The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
  - whymauri 3y ago
    Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
  - nopinsight 3y ago
    Most likely, it’s less generally smart than the top 2-4% of US high school students. It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance. Since the test questions were unseen, the result still suggests the person has some intelligence though. Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.
AndyNemmity 3y ago
I saw the benchmarks, and everyone repeating how amazing it is, so I signed up for pro today. It was a complete and total disaster for my normal workflows. Compared to ChatGPT4, it is orders of magnitude worse. I get that people are impressed by the benchmarks, and press released, but actually using it, it feels like a large step backward in time.

eschluntz 3y ago

(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)

RivieraKid 3y ago
What's your estimate of how much does it increase a typical programmer's productivity?
bwanab 3y ago
Sounds almost recursive.
- deleted 3y ago
  [deleted]
zellyn 3y ago
Do y'all have an explanation for why Haiku outperforms Sonnet for code?
- razodactyl 3y ago
  Seems like they optimised this model with coding datasets for use in Copilot-like assistants with the low latency advantage. Additionally, I wonder if an alternate dataset is provided based on model size as to not run into issues with model forgetting.
mkakkori 3y ago
Interested to try this out as well! What is your setup for integrating Opus to you development workflow?

nopinsight 3y ago

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitter.com/idavidrein/status/1764675668175094169 https://twitter.com/idavidrein/status/1764675668175094169 GPQA: A Graduate-Level Google-Proof Q&A Benchmark https://arxiv.org/abs/2311.12022 https://arxiv.org/abs/2311.12022

deleted 3y ago
[deleted]
wbarber 3y ago
What's to say this isn't just a demonstration of memorization capabilities? For example, rephrasing the logic of the question or even just simple randomizing the order of the multiple choice answers to these questions often dramatically impacts performance. For example, every model in the Claude 3 family repeats the memorized solution to the lion, goat, wolf riddle regardless of how I modify the riddle.
- apsec112 3y ago
  If the answers were Googleable, presumably smart humans with Internet access wouldn't do barely better than chance?
- msikora 3y ago
  GPT-4 used to have the same issue with this puzzle early on but they've fixed since then (the fix was like mid 2023).
  - Jensson 3y ago
    The fix is to train it on this puzzle and variants of it, meaning it memorized this pattern. It still fail similar puzzles if given in a different structure, until they feed it that structure as well. LLMs is more like programming than human intelligence, they need to program in the solution to these riddles very much like we did expert systems in the past. The main new thing we get here is natural language compatibility, but other than that the programming seems to be the same or weaker than old programming of expert systems. The other big thing is that there is already a ton of solutions on the web coded in natural language, such as all the tutorials etc, so you get all of those programs for free. But other than that these LLMs seems to have exactly the same problems and limitations and strengths as expert systems. They don't generalize in a flexible enough manner to solve problems like a human.

vok 3y ago

APPS has 3 subsets by difficulty level: introductory, interview, and competition. It isn't clear which subset Claude 3 was benchmarked on. Even if it is just "introductory" it is still pretty good, but it would be good to know.

nopinsight 3y ago
Since they don’t state it, does it mean they tested it on the whole test set? If that’s the case, and we assume for simplicity that Opus solves all Intro problems and none of the Competition problems, it’d have solved 83%+ of the Interview level problems. (There are 1000/3000/1000 problems in the test set in each level). It’d be great if someone from Anthropic provides an answer though.

nopinsight 3y ago

AMC 10, AMC 12 (2023) results in Table 2 suggest Claude 3 Opus is better than the average high school students who participate in these math competitions. These math problems are not straightforward and cannot be solve by simply memorizing formulas. Most of the students are also quite good at math. The student averages are 64.4 and 61.5 respectively, while Opus 3 scores are 72 and 63. Probably fewer than 100,000 students take part in AMC 12 out of possibly 3-4 million grade-12 students. Assume just half of the top US students participate, the average score of AMC would represent the top 2-4% of US high school students. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf#page7 https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...

usaar333 3y ago
Interestingly, math olympiad problems (using ones I wrote myself years ago so outside training data) seem to be better in Claude 3. Almost everything else though I've tested seems better in GPT-4.
sebzim4500 3y ago
The benchmark would suggest that but if you actually try asking it questions it is much worse than a bright high school student.
- whymauri 3y ago
  Is it possible they are using some sort of specialized prompting for these? I'm not familiar with how prompting optimization might work in LLM benchmarks.
- nopinsight 3y ago
  Most likely, it’s less generally smart than the top 2-4% of US high school students. It’s more like someone who trains really hard on many, many math problems, even though most of them are not the replicas of the test questions, and get to that level of performance. Since the test questions were unseen, the result still suggests the person has some intelligence though. Note that there’s some transfer learning in LLMs. Training on math and coding yields better reasoning capabilities as well.

AndyNemmity 3y ago

I saw the benchmarks, and everyone repeating how amazing it is, so I signed up for pro today. It was a complete and total disaster for my normal workflows. Compared to ChatGPT4, it is orders of magnitude worse. I get that people are impressed by the benchmarks, and press released, but actually using it, it feels like a large step backward in time.