4 ms·
Gemini 2.5 Pro gets 64% on SWE-bench verified. Sonnet 3.7 gets 70% They are reporting that GPT-4.1 gets 55%.
by dmd 2y ago
Gemini 2.5 Pro gets 64% on SWE-bench verified.
Sonnet 3.7 gets 70%
They are reporting that GPT-4.1 gets 55%.
- hmottestad 2y agoAre those with «thinking» or without?
- energy123 2y agoWith
- chaos_emergent 2y agobased on their release cadence, I suspect that o4-mini will compete on price, performance, and context length with the rest of these models.
- hecticjeff 2y agoo4-mini, not to be confused with 4o-mini
- sanxiyn 2y agoSonnet 3.7's 70% is without thinking, see https://www.anthropic.com/news/claude-3-7-sonnet https://www.anthropic.com/news/claude-3-7-sonnet
- aledalgrande 2y agoThe thinking tokens (even just 1024) make a massive difference in real world tasks with 3.7 in my experience
- egeozcan 2y agoVery interesting. For my use cases, Gemini's responses beat Sonnet 3.7's like 80% of the time (gut feeling, didn't collect actual data). It beats Sonnet 100% of the time when the context gets above 120k.
- int_19h 2y agoAs usual with LLMs. In my experience, all those metrics are useful mainly to tell which models are definitely bad, but doesn't tell you much about which ones are good, and especially not how the good ones stack against each other in real world use cases. Andrej Karpathy famously quipped that he only trusts two LLM evals: Chatbot Arena (which has humans blindly compare and score responses), and the r/LocalLLaMA comment section.