5 ms·

No benchmark comparisons to other models, especially Gemini 2.5 Pro, is telling.

by ZeroCool2u 2y ago

dmd 2y ago
Gemini 2.5 Pro gets 64% on SWE-bench verified. Sonnet 3.7 gets 70% They are reporting that GPT-4.1 gets 55%.
- hmottestad 2y ago
  Are those with «thinking» or without?
  - energy123 2y ago
    With
  - chaos_emergent 2y ago
    based on their release cadence, I suspect that o4-mini will compete on price, performance, and context length with the rest of these models.
    - hecticjeff 2y ago
      o4-mini, not to be confused with 4o-mini
  - sanxiyn 2y ago
    Sonnet 3.7's 70% is without thinking, see https://www.anthropic.com/news/claude-3-7-sonnet https://www.anthropic.com/news/claude-3-7-sonnet
  - aledalgrande 2y ago
    The thinking tokens (even just 1024) make a massive difference in real world tasks with 3.7 in my experience
- egeozcan 2y ago
  Very interesting. For my use cases, Gemini's responses beat Sonnet 3.7's like 80% of the time (gut feeling, didn't collect actual data). It beats Sonnet 100% of the time when the context gets above 120k.
  - int_19h 2y ago
    As usual with LLMs. In my experience, all those metrics are useful mainly to tell which models are definitely bad, but doesn't tell you much about which ones are good, and especially not how the good ones stack against each other in real world use cases. Andrej Karpathy famously quipped that he only trusts two LLM evals: Chatbot Arena (which has humans blindly compare and score responses), and the r/LocalLLaMA comment section.
    - ezyang 2y ago
      Lmarena isn't that useful anymore lol
      - int_19h 2y ago
        I actually agree with that, but it's generally better than other scores. Also, the quote is like a year old at this point. In practice you have to evaluate the models yourself for any non-trivial task.
poormathskills 2y ago
Go look at their past blog posts. OpenAI only ever benchmarks against their own models. This is pretty common across industries. The leader doesn’t compare themselves to the competition.
- dimitrios1 2y ago
  There is no uniform tactic for this type of marketing. They will compare against whomever they need to to suit their marketing goals.
- oofbaroomf 2y ago
  Leader is debatable, especially given the actual comparisons...
- swyx 2y ago
  also sometimes if you get it wrong you catch unnecessary flak
- kweingar 2y ago
  That would make sense if OAI were the leader.
- christianqchung 2y ago
  Okay, it's common across other industries, but not this one. Here is Google, Facebook, and Anthropic comparing their frontier models to others[1][2][3]. [1] https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-pro https://blog.google/technology/google-deepmind/gemini-model-... [2] https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/ [3] https://www.anthropic.com/claude/sonnet https://www.anthropic.com/claude/sonnet
  - poormathskills 2y ago
    Right. Those labs aren’t leading the industry.
deleted 2y ago
[deleted]

No benchmark comparisons to other models, especially Gemini 2.5 Pro, is telling.

dmd 2y ago

Gemini 2.5 Pro gets 64% on SWE-bench verified. Sonnet 3.7 gets 70% They are reporting that GPT-4.1 gets 55%.

hmottestad 2y ago
Are those with «thinking» or without?
- energy123 2y ago
  With
- chaos_emergent 2y ago
  based on their release cadence, I suspect that o4-mini will compete on price, performance, and context length with the rest of these models.
  - hecticjeff 2y ago
    o4-mini, not to be confused with 4o-mini
- sanxiyn 2y ago
  Sonnet 3.7's 70% is without thinking, see https://www.anthropic.com/news/claude-3-7-sonnet https://www.anthropic.com/news/claude-3-7-sonnet
- aledalgrande 2y ago
  The thinking tokens (even just 1024) make a massive difference in real world tasks with 3.7 in my experience
egeozcan 2y ago
Very interesting. For my use cases, Gemini's responses beat Sonnet 3.7's like 80% of the time (gut feeling, didn't collect actual data). It beats Sonnet 100% of the time when the context gets above 120k.
- int_19h 2y ago
  As usual with LLMs. In my experience, all those metrics are useful mainly to tell which models are definitely bad, but doesn't tell you much about which ones are good, and especially not how the good ones stack against each other in real world use cases. Andrej Karpathy famously quipped that he only trusts two LLM evals: Chatbot Arena (which has humans blindly compare and score responses), and the r/LocalLLaMA comment section.
  - ezyang 2y ago
    Lmarena isn't that useful anymore lol
    - int_19h 2y ago
      I actually agree with that, but it's generally better than other scores. Also, the quote is like a year old at this point. In practice you have to evaluate the models yourself for any non-trivial task.

poormathskills 2y ago

Go look at their past blog posts. OpenAI only ever benchmarks against their own models. This is pretty common across industries. The leader doesn’t compare themselves to the competition.

dimitrios1 2y ago
There is no uniform tactic for this type of marketing. They will compare against whomever they need to to suit their marketing goals.
oofbaroomf 2y ago
Leader is debatable, especially given the actual comparisons...
swyx 2y ago
also sometimes if you get it wrong you catch unnecessary flak
kweingar 2y ago
That would make sense if OAI were the leader.
christianqchung 2y ago
Okay, it's common across other industries, but not this one. Here is Google, Facebook, and Anthropic comparing their frontier models to others[1][2][3]. [1] https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-pro https://blog.google/technology/google-deepmind/gemini-model-... [2] https://ai.meta.com/blog/llama-4-multimodal-intelligence/ https://ai.meta.com/blog/llama-4-multimodal-intelligence/ [3] https://www.anthropic.com/claude/sonnet https://www.anthropic.com/claude/sonnet
- poormathskills 2y ago
  Right. Those labs aren’t leading the industry.

deleted 2y ago

[deleted]