Anthropic just released their new Claude 3 models with evals showing better performance on coding tasks. With that in mind, I’ve been benchmarking the new model

Anthropic just released their new Claude 3 models with evals showing better performance on coding tasks. With that in mind, I’ve been benchmarking the new models using Aider’s code editing benchmark suite. Claude 3 Opus outperforms all of OpenAI’s models, making it the best available model for pair programming with AI. The new claude-3-opus-20240229 model got the highest score ever on this benchmark, completing 68.4% of the tasks. While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use.