6 ms·

I still feel like the difference between Sonnet and Opus is a bit unclear. Somewhere on Anthropic's website it says that Opus is the most advanced, but on other

by diggan 2y ago

I still feel like the difference between Sonnet and Opus is a bit unclear. Somewhere on Anthropic's website it says that Opus is the most advanced, but on other parts it says Sonnet is the most advanced and also the fastest. The UI doesn't make the distinction clear either. Then on Perplexity, Perplexity says that Opus is the most advanced, compared to Sonnet. And finally, in the table in the blogpost, Opus isn't even included? It seems to me like Opus is the best model they have, but they don't want people to default using it, maybe the ROI is lower on Opus or something? When I manually tested it, I feel like Opus gives slightly better replies compared to Sonnet, but I'm not 100% it's just placebo.

smallerize 2y ago
Opus has been stuck on 3.0, so Sonnet 3.5 is better for most things as well as cheaper.
- diggan 2y ago
  > Opus has been stuck on 3.0, so Sonnet 3.5 is better So for example, Perplexity is wrong here implying that Opus is better than Sonnet? https://i.imgur.com/N58I4PC.png https://i.imgur.com/N58I4PC.png
  - hobofan 2y ago
    I think as of this announcement that is indeed outdated information.
    - diggan 2y ago
      So Opus that costs $15.00/$75.00 for 1mil tokens (input/output) is now worse than the model that costs $3.00/$15.00? That's according to https://docs.anthropic.com/en/docs/about-claude/models https://docs.anthropic.com/en/docs/about-claude/models which has "claude-3-5-sonnet-20241022" as the latest model (today's date)
      - hobofan 2y ago
        Yes, you will find similar things at essentially all other model providers. The older/bigger GPT4 runs at $30/$60 and peforms about on par with GPT4o-mini which costs only $0.15/$0.60. If you are currently, or have been integrating AI models in the past ~2 years, you should definitely keep up with model capability/pricing development. If you are staying on old models you are certainly overpaying/leaving performance on the table. It's essentially a tax on agility.
        diggan 2y ago
        > The older/bigger GPT4 runs at $30/$60 and peforms about on par with GPT4o-mini which costs only $0.15/$0.60. I don't think GPT-4o Mini has comparable performance to GPT-4 at all, where are you finding the benchmarks claiming this? Everywhere I look says GPT-4 is more powerful, but GPT-4o Mini is most cost-effective, if you're OK with worse performance. Even OpenAI themselves about GPT-4o Mini: > Our affordable and intelligent small model for fast, lightweight tasks. GPT-4o mini is cheaper and more capable than GPT-3.5 Turbo. If it was "on par" with GPT-4 they would surely say this. > should definitely keep up with model capability/pricing development Yeah, I mean that's why we're both here and why we're discussing this very topic, right? :D
        hobofan 2y ago
        > Yeah, I mean that's why we're both here and why we're discussing this very topic, right? :D That wasn't specifically directed at "you", but more as a plea to everyone reading that comment ;) I looked at a few benchmarks, comparing the two, which like in the case of Opus 3 vs Sonnet 3.5 is hard, as the benchmarks the wider community is interested in shifts over time. I think this page[0] provides the best overview I can link to. Yes, GPT4 is better in the MMLU benchmark, but in all other benchmarks and the LMSys Chatbot Arena scores[1], GPT4o-mini comes out ahead. Overall, the margin between is so thin that it falls under my definition of "on par". I think OpenAI is generally a bit more conservative with the messaging here (which is understandable), and they only advertise a model as "more capable", if one model beats the other one in every benchmark they track, which AFAIK is the case when it comes to 4o mini vs 3.5 Turbo. [0]: https://context.ai/compare/gpt-4o-mini/gpt-4 https://context.ai/compare/gpt-4o-mini/gpt-4 [1]: https://artificialanalysis.ai/models?models_selected=gpt-4o-mini%2Cgpt-4 https://artificialanalysis.ai/models?models_selected=gpt-4o-...
  - apsec112 2y ago
    Basically yeah
wmf 2y ago
Opus is a larger and more expensive model. Presumably 3.5 Opus will be the best but it hasn't been released. 3.5 Sonnet is better than 3.0 Opus kind of like how a newer i5 midrange processor is faster and cheaper than an old high-end i7.
- gwd 2y ago
  Makes me wonder if perhaps they do have 3.5 Opus trained, but that they're not releasing it because 3.5 Sonnet is already enough to beat the competition, and some combination of "don't want to contribute to an arms race" and "it has some scary capabilities they weren't sure were ready to publish yet".
hobofan 2y ago
Opus hasn't yet gotten an update from 3 to 3.5, and if you line up the benchmarks, the Sonnet "3.5 New" model seems to beat it everywhere. I think they originally announced that Opus would get a 3.5 update, but with every product update they are doing I'm doubting it more and more. It seems like their strategy is to beat the competition on a smaller model that they can train/tune more nimbly and pair it with outside-the-model product features, and it honestly seems to be working.
- diggan 2y ago
  > Opus hasn't yet gotten an update from 3 to 3.5, and if you line up the benchmarks, the Sonnet "3.5 New" model seems to beat it everywhere Why isn't Anthropic clearer about Sonnet being better then? Why isn't it included in the benchmark if new Sonnet beats Opus? Why are they so ambiguous with their language? For example, https://www.anthropic.com/api https://www.anthropic.com/api says: > Sonnet - Our best combination of performance and speed for efficient, high-throughput tasks. > Opus - Our highest-performing model, which can handle complex analysis, longer tasks with many steps, and higher-order math and coding tasks. And Opus is above/after Sonnet. That to me implies that Opus is indeed better than Sonnet. But then you go to https://docs.anthropic.com/en/docs/about-claude/models https://docs.anthropic.com/en/docs/about-claude/models and it says: > Claude 3.5 Sonnet - Most intelligent model - Claude 3 Opus - Powerful model for highly complex tasks Does that mean Sonnet 3.5 is better than Opus for even highly complex tasks, since it's the "most intelligent model"? Or just for everything except "highly complex tasks" I don't understand why this seems purposefully ambiguous?
  - hobofan 2y ago
    > I don't understand why this seems purposefully ambiguous? I wouldn't attribute this to malice when it can also be explained by incompetence. Sonnet 3.5 New > Opus 3 > Sonnet 3.5 is generally how they stack up against each other when looking at the total benchmarks. "Sonnet 3.5 New" has just been announced, and they likely just haven't updated the marketing copy across the whole page yet, and maybe also haven't figured out how to graple with the fact that their new Sonnet model was ready faster than their next Opus model. At the same time I think they want to keep their options open to either: A) drop a Opus 3.5 soon that will bring the logic back in order again B) potentially phase out Opus, and instead introduce new branding for what they called a "reasoning model" like OpenAI did with o1(-preview)
bloedsinnig 2y ago
Big models / huge models take weeks / month longer than the smaller ones. Thats why they release them with that skew
- a9dhalaan 2y ago
  I don't think that's quite it. They had it on their website before this, that opus 3.5 was coming soon, now they've removed that from the webpage. Also, Gemini ultra 1.0, was released like 8 months ago, 1.5 pro released soon after, with this wording "The first Gemini 1.5 model we’re releasing for early testing is Gemini 1.5 Pro" Still no ultra 1.5, despite many mid and small sized models being released in that time frame. This isn't just an issue of "the training time takes longer", or a "skew" to release dates. There's a better theory to explain why all SoTA LLM companies have not released a heavy model in many months.
JamesBarney 2y ago
Sonnet is better for most things. But I do prefer Opus's writing style to Sonnet.
kalkin 2y ago
By reputation -- I can't vouch for this personally, and I don't know if it'll still be true with this update -- Opus is still often better for things like creative writing and conversations about emotional or political topics.
- aoeusnth1 2y ago
  Yes, (old) 3.5 Sonnet is distinctly worse at emotional intelligence, flexibility, expressiveness and poetry.
  - a9dhalaan 2y ago
    Are you also implying that new 3.5 sonnet is better at those things?
    - aoeusnth1 2y ago
      No, Opus is better. I have no experience with 3.5.new.
HarHarVeryFunny 2y ago
Anthropic use the names Haiku/Sonnet/Opus for the small/medium/large versions of each generation of their models, so within-generation that is also their performance (& cost) order. Evidentially Sonnet 3.5 outperforms Opus 3.0 on at least some tasks, but that is not a same-generation comparison. I'm wondering at this point if they are going to release Opus 3.5 at all, or maybe skip it and go straight to 4.0. It's possible that Haiku 3.5 is a distillation of Opus 3.5.
karmasimida 2y ago
Opus the biggest and slowest and most expensive one Not most advanced
kqr 2y ago
The models "3.5 Sonnet" and "3 Opus" are in my experience nearly at the same level. Once in my last 250 prompts did I run into a problem that 3 Opus was able to solve, but 3.5 Sonnet could not. (I forget the details but it was a combination of logic and trivia knowledge. It is highly likely 3.5 Sonnet would have done a better job with better prompting and richer context, but this was a problem where also I lacked the context and understanding to prompt well.) Given that 3.5 Sonnet is cheaper and faster than 3 Opus, I default to 3.5 Sonnet so I don't know what the number for the reverse is. How many problems do 3.5 Sonnet get which 3 Opus does not? ¯\_(ツ)_/¯ My best guess would be that it's something in the same kind of range.
inquisitor27552 2y ago
yes it baffles they cant semver the shit out of them properly (anthtopic, meta, openai, lol)

by diggan 2y ago

smallerize 2y ago

Opus has been stuck on 3.0, so Sonnet 3.5 is better for most things as well as cheaper.

diggan 2y ago
> Opus has been stuck on 3.0, so Sonnet 3.5 is better So for example, Perplexity is wrong here implying that Opus is better than Sonnet? https://i.imgur.com/N58I4PC.png https://i.imgur.com/N58I4PC.png
- hobofan 2y ago
  I think as of this announcement that is indeed outdated information.
  - diggan 2y ago
    So Opus that costs $15.00/$75.00 for 1mil tokens (input/output) is now worse than the model that costs $3.00/$15.00? That's according to https://docs.anthropic.com/en/docs/about-claude/models https://docs.anthropic.com/en/docs/about-claude/models which has "claude-3-5-sonnet-20241022" as the latest model (today's date)
    - hobofan 2y ago
      Yes, you will find similar things at essentially all other model providers. The older/bigger GPT4 runs at $30/$60 and peforms about on par with GPT4o-mini which costs only $0.15/$0.60. If you are currently, or have been integrating AI models in the past ~2 years, you should definitely keep up with model capability/pricing development. If you are staying on old models you are certainly overpaying/leaving performance on the table. It's essentially a tax on agility.
      - diggan 2y ago
        > The older/bigger GPT4 runs at $30/$60 and peforms about on par with GPT4o-mini which costs only $0.15/$0.60. I don't think GPT-4o Mini has comparable performance to GPT-4 at all, where are you finding the benchmarks claiming this? Everywhere I look says GPT-4 is more powerful, but GPT-4o Mini is most cost-effective, if you're OK with worse performance. Even OpenAI themselves about GPT-4o Mini: > Our affordable and intelligent small model for fast, lightweight tasks. GPT-4o mini is cheaper and more capable than GPT-3.5 Turbo. If it was "on par" with GPT-4 they would surely say this. > should definitely keep up with model capability/pricing development Yeah, I mean that's why we're both here and why we're discussing this very topic, right? :D
        hobofan 2y ago
        > Yeah, I mean that's why we're both here and why we're discussing this very topic, right? :D That wasn't specifically directed at "you", but more as a plea to everyone reading that comment ;) I looked at a few benchmarks, comparing the two, which like in the case of Opus 3 vs Sonnet 3.5 is hard, as the benchmarks the wider community is interested in shifts over time. I think this page[0] provides the best overview I can link to. Yes, GPT4 is better in the MMLU benchmark, but in all other benchmarks and the LMSys Chatbot Arena scores[1], GPT4o-mini comes out ahead. Overall, the margin between is so thin that it falls under my definition of "on par". I think OpenAI is generally a bit more conservative with the messaging here (which is understandable), and they only advertise a model as "more capable", if one model beats the other one in every benchmark they track, which AFAIK is the case when it comes to 4o mini vs 3.5 Turbo. [0]: https://context.ai/compare/gpt-4o-mini/gpt-4 https://context.ai/compare/gpt-4o-mini/gpt-4 [1]: https://artificialanalysis.ai/models?models_selected=gpt-4o-mini%2Cgpt-4 https://artificialanalysis.ai/models?models_selected=gpt-4o-...
- apsec112 2y ago
  Basically yeah

wmf 2y ago

Opus is a larger and more expensive model. Presumably 3.5 Opus will be the best but it hasn't been released. 3.5 Sonnet is better than 3.0 Opus kind of like how a newer i5 midrange processor is faster and cheaper than an old high-end i7.

gwd 2y ago
Makes me wonder if perhaps they do have 3.5 Opus trained, but that they're not releasing it because 3.5 Sonnet is already enough to beat the competition, and some combination of "don't want to contribute to an arms race" and "it has some scary capabilities they weren't sure were ready to publish yet".

hobofan 2y ago

Opus hasn't yet gotten an update from 3 to 3.5, and if you line up the benchmarks, the Sonnet "3.5 New" model seems to beat it everywhere. I think they originally announced that Opus would get a 3.5 update, but with every product update they are doing I'm doubting it more and more. It seems like their strategy is to beat the competition on a smaller model that they can train/tune more nimbly and pair it with outside-the-model product features, and it honestly seems to be working.

diggan 2y ago
> Opus hasn't yet gotten an update from 3 to 3.5, and if you line up the benchmarks, the Sonnet "3.5 New" model seems to beat it everywhere Why isn't Anthropic clearer about Sonnet being better then? Why isn't it included in the benchmark if new Sonnet beats Opus? Why are they so ambiguous with their language? For example, https://www.anthropic.com/api https://www.anthropic.com/api says: > Sonnet - Our best combination of performance and speed for efficient, high-throughput tasks. > Opus - Our highest-performing model, which can handle complex analysis, longer tasks with many steps, and higher-order math and coding tasks. And Opus is above/after Sonnet. That to me implies that Opus is indeed better than Sonnet. But then you go to https://docs.anthropic.com/en/docs/about-claude/models https://docs.anthropic.com/en/docs/about-claude/models and it says: > Claude 3.5 Sonnet - Most intelligent model - Claude 3 Opus - Powerful model for highly complex tasks Does that mean Sonnet 3.5 is better than Opus for even highly complex tasks, since it's the "most intelligent model"? Or just for everything except "highly complex tasks" I don't understand why this seems purposefully ambiguous?
- hobofan 2y ago
  > I don't understand why this seems purposefully ambiguous? I wouldn't attribute this to malice when it can also be explained by incompetence. Sonnet 3.5 New > Opus 3 > Sonnet 3.5 is generally how they stack up against each other when looking at the total benchmarks. "Sonnet 3.5 New" has just been announced, and they likely just haven't updated the marketing copy across the whole page yet, and maybe also haven't figured out how to graple with the fact that their new Sonnet model was ready faster than their next Opus model. At the same time I think they want to keep their options open to either: A) drop a Opus 3.5 soon that will bring the logic back in order again B) potentially phase out Opus, and instead introduce new branding for what they called a "reasoning model" like OpenAI did with o1(-preview)

bloedsinnig 2y ago

Big models / huge models take weeks / month longer than the smaller ones. Thats why they release them with that skew

a9dhalaan 2y ago
I don't think that's quite it. They had it on their website before this, that opus 3.5 was coming soon, now they've removed that from the webpage. Also, Gemini ultra 1.0, was released like 8 months ago, 1.5 pro released soon after, with this wording "The first Gemini 1.5 model we’re releasing for early testing is Gemini 1.5 Pro" Still no ultra 1.5, despite many mid and small sized models being released in that time frame. This isn't just an issue of "the training time takes longer", or a "skew" to release dates. There's a better theory to explain why all SoTA LLM companies have not released a heavy model in many months.

JamesBarney 2y ago

Sonnet is better for most things. But I do prefer Opus's writing style to Sonnet.

kalkin 2y ago

By reputation -- I can't vouch for this personally, and I don't know if it'll still be true with this update -- Opus is still often better for things like creative writing and conversations about emotional or political topics.

aoeusnth1 2y ago
Yes, (old) 3.5 Sonnet is distinctly worse at emotional intelligence, flexibility, expressiveness and poetry.
- a9dhalaan 2y ago
  Are you also implying that new 3.5 sonnet is better at those things?
  - aoeusnth1 2y ago
    No, Opus is better. I have no experience with 3.5.new.

HarHarVeryFunny 2y ago

Anthropic use the names Haiku/Sonnet/Opus for the small/medium/large versions of each generation of their models, so within-generation that is also their performance (& cost) order. Evidentially Sonnet 3.5 outperforms Opus 3.0 on at least some tasks, but that is not a same-generation comparison. I'm wondering at this point if they are going to release Opus 3.5 at all, or maybe skip it and go straight to 4.0. It's possible that Haiku 3.5 is a distillation of Opus 3.5.

karmasimida 2y ago

Opus the biggest and slowest and most expensive one Not most advanced

kqr 2y ago

The models "3.5 Sonnet" and "3 Opus" are in my experience nearly at the same level. Once in my last 250 prompts did I run into a problem that 3 Opus was able to solve, but 3.5 Sonnet could not. (I forget the details but it was a combination of logic and trivia knowledge. It is highly likely 3.5 Sonnet would have done a better job with better prompting and richer context, but this was a problem where also I lacked the context and understanding to prompt well.) Given that 3.5 Sonnet is cheaper and faster than 3 Opus, I default to 3.5 Sonnet so I don't know what the number for the reverse is. How many problems do 3.5 Sonnet get which 3 Opus does not? ¯\_(ツ)_/¯ My best guess would be that it's something in the same kind of range.

inquisitor27552 2y ago

yes it baffles they cant semver the shit out of them properly (anthtopic, meta, openai, lol)