6 ms·

> Qwen 3.5 397B-A17B is a good comparison It is not. It's a terrible comparison. Qwen, deepseek and other Chinese models are known for their 10x or even better

by hirako2000 8mo ago

> Qwen 3.5 397B-A17B is a good comparison It is not. It's a terrible comparison. Qwen, deepseek and other Chinese models are known for their 10x or even better efficiency compared to Anthropic's. That's why the difference between open router prices and those official providers isn't that different. Plus who knows what open routed providers do in term quantization. They may be getting 100x better efficiency, thus the competitive price. That being said not all users max out their plan, so it's not like each user costs anthropic 5,000 USD. The hemoragy would be so brutal they would be out of business in months

simianwords 8mo ago
>It is not. It's a terrible comparison. Qwen, deepseek and other Chinese models are known for their 10x or even better efficiency compared to Anthropic's. I find it a good comparison because it is a good baseline since we have zero insider knowledge of Anthropic. They give me an idea that a certain size of a model has a certain cost associated. I don't buy the 10x efficiency thing: they are just lagging behind the performance of current SOTA models. They perform much worse than the current models while also costing much less - exactly what I would expect. Current Qwen models perform as good as Sonnet 3 I think. 2 years later when Chinese models catchup with enough distillation attacks, they would be as good as Sonnet 4.6 and still be profitable.
- coldtea 8mo ago
  > I don't buy the 10x efficiency thing: they are just lagging behind the performance of current SOTA models. They perform much worse than the current models while also costing much less - exactly what I would expect. Define "much worse". +--------------------------------------+-------------+-----------+------------------+ | Benchmark | Claude Opus | DeepSeek | DeepSeek vs Opus | +--------------------------------------+-------------+-----------+------------------+ | SWE-Bench Verified (coding) | 80.9% | 73.1% | ~90% | | MMLU (knowledge) | ~91 | ~88.5 | ~97% | | GPQA (hard science reasoning) | ~79–80 | ~75–76 | ~95% | | MATH-500 (math reasoning) | ~78 | ~90 | ~115% | +--------------------------------------+-------------+-----------+------------------+
  - Filligree 8mo ago
    Everyone who's used Opus knows it's better than the others in a way that isn't captured by the benchmarks. I would describe it as taste. Lots of models get really close on benchmarks, but benchmarks only tell us how good they are at solving a defined problem. Opus is far better at solving ill-defined ones.
    - coldtea 8mo ago
      >Everyone who's used Opus knows it's better than the others in a way that isn't captured by the benchmarks. I would describe it as taste. Ah, the "trust me bro" advantage. Couldn't it just be brand identity and familiarity?
      - vidarh 8mo ago
        I have a project where we've had Opus, Sonnet, Deepseek, Kimi, Qwen create and execute an aggregate total of about 350 plans so far, and the quality difference as measured in plans where the agent failed to complete the tasks on the first run is high enough that it comes out several times higher than Anthropics subscription prices, but probably cheaper than the API prices once we have improved the harness further - at present the challenge is that too much human intervention for the cheaper models drives up the cost. My dashboard goes from all green to 50/50 green/red for our agents whenever I switch from Claude to one of the cheaper agents... This is after investing a substantial amount of effort in "dumbing down" the prompts - e.g. adding a lot of extra wording to convince the dumber models to actually follow instructions - that is not necessary for Sonnet or Opus. I buy the benchmarks. The problem is that a 10% difference in the benchmarks makes the difference between barely usable and something that can consistently deliver working code unilaterally and require few review interventions. Basically, the starting point for "usable" on these benchmarks is already very far up the scale for a lot of tasks. I do strongly believe the moat is narrow - With 4.6 I switched from defaulting to Opus to defaulting to Sonnet for most tasks. I can fully see myself moving substantial workloads to a future iteration of Kimi, Qwen or Deepseek in 6-12 months once they actually start approaching Sonnet 4.5 level. But for my use at least, currently, they're at best competing with Athropics 3.x models in terms of real-world ability. That said, even now, I think if we were stuck with current models for 12 months, we might well also be able to build our way around this and get to a point where Deepseek and Kimi would be cheaper than Sonnet. Eventually we'll converge on good enough harnesses to get away with cheaper models for most uses, and the remaining appeal for the frontier models will be complex planning and actual hard work.
- crooked-v 8mo ago
  > distillation "attacks" I find it really funny that anyone can call it this with a straight face when all the American models are based on heaps of illegally pirated books and TOS-breaking website scraping in the first place.
  - hirako2000 8mo ago
    Looters calling the police for being looted. Plus, Chines made distillation did good to the overall internet infrastructure. Millions of small Joe's WordPress website, paid and maintained out of good will, getting hammered by AI mining vs making the VC loaded pirates pay for what they seeded, I find the latter more fair.
jychang 8mo ago
That's a tautology. People think chinese models are 10x more efficient because they're 10x cheaper, and then you use that to claim that they're 10x more efficient. Opus isn't that expensive to host. Look at Amazon Bedrock's t/s numbers for Opus 4.5 vs other chinese models. They're around the same order of magnitude- which means that Opus has roughly the same amount of active params as the chinese models. Also, you can select BF16 or Q8 providers on openrouter.
- re-thc 8mo ago
  > That's a tautology. People think chinese models are 10x more efficient because they're 10x cheaper They do have different infrastructure / electricity costs and they might not run on nvidia hardware. It's not just the models.
  - jychang 8mo ago
    Except there are providers that serve both chinese models AND opus as well. On the same hardware. Namely, Amazon Bedrock and Google Vertex. That means normalized infrastructure costs, normalized electricity costs, and normalized hardware performance. Normalized inference software stack, even (most likely). It's about a close of a 1 to 1 comparison as you can get. Both Amazon and Google serve Opus at roughly ~1/2 the speed of the chinese models. Note that they are not incentivized to slow down the serving of Opus or the chinese models! So that tells you the ratio of active params for Opus and for the chinese models.
    - Shakahs 8mo ago
      AWS and GCP both have their own custom inference chips, so a better example for hosting Opus on commodity hardware would be Digital Ocean.
    - giancarlostoro 8mo ago
      And Microsoft's Azure. It's on all 3 major cloud providers. Which tells me, they can make profit from these cloud providers without having to pay for any hardware. They just take a small enough cut. https://code.claude.com/docs/en/microsoft-foundry https://code.claude.com/docs/en/microsoft-foundry https://www.anthropic.com/news/claude-in-microsoft-foundry https://www.anthropic.com/news/claude-in-microsoft-foundry
lelanthran 8mo ago
> That being said not all users max out their plan, These are not cell phone plans which the average joe takes, they are plans purchased with the explicit goal of software development. I would guess that 99 out of every 100 plans are purchased with the explicit goal of maxing them out.
- solumunus 8mo ago
  There’s absolutely no way that’s true.
- serial_dev 8mo ago
  I’m not maxing them out… I have issues that I need to fix, features I need to develop, and I have things I want to learn. When I have a feeling that these tools will speed me up, I use them. My client pays for a couple of these tools in an enterprise deal, and I suspect most of us on the team work like that. If my goal was to max out every tool my client pays, I’d be working 24hrs a day and see no sunlight ever. I guess it’s like the all you can eat buffet. Everybody eats a lot, but if you eat so much that you throw up and get sick, you are special.
  - bloppe 8mo ago
    [flagged]
- rustystump 8mo ago
  In saas this is not true. Most saas is highly profitable or was i suppose because they knew that most of their customers would never max out their plans.
- Ginden 8mo ago
  My employer bought me a Claude Max subscription. On heavy weeks I use 80% of the subscription. And among software engineers that I know, I'm a relatively heavy user. Why? Because in my experience, the bottleneck is in shareholders approving new features, not my ability to dish out code.
- raihansaputra 8mo ago
Weaver_zhu 8mo ago
Agree, but I guess the Opus 4.6 is 10x larger, rather than Chinese models being 10x more efficient. It is said that GPT-4 is already a 1.6T model, and Llama 4 behemoth is also much bigger than Chinese open-weight models. Chinese tech companies are short of frontier GPUs, but they did a lot of innovations on inference efficiency (Deepseek CEO Liang himself shows up in the author list of the related published papers).
- jychang 8mo ago
  No, Opus cannot be 10x larger than the chinese models. If Opus was 10x larger than the chinese models, then Google Vertex/Amazon Bedrock would serve it 10x slower than Deepseek/Kimi/etc. That's not the case. They're in the same order of magnitude of speed.
  - bakugo 8mo ago
    I agree that Opus almost definitely isn't anywhere near that big, but AWS throughput might not be a great way to measure model size. According to OpenRouter, AWS serves the latest Opus and Sonnet at roughly the same speed. It's likely that they simply allocate hardware differently per model.
    - jychang 8mo ago
      The numbers look about right. Opus 4.5 is about 1.5x the size of Sonnet 4.6, and Opus 4/4.1 is about 5x the size of Sonnet 4.5/4.6. Note that Opus 4.5 is about 1/3 the size of Opus 4/4.1 (and 1/3 the price in the API)
  - Filligree 8mo ago
    They serve it about 2x slower. So it must have about 2x the active parameters. It could still be 10x larger overall, though that would not make it 10x more expensive.
    - jychang 8mo ago
      Yes, but I highly doubt they would increase sparsity much vs the chinese models. That's how you get Llama 4. Pretty much every major lab settled on ~3-5% sparsity for a reason.
Havoc 8mo ago
> Plus who knows what open routed providers do in term quantization The quantisation is shown on the provider section.
- hirako2000 8mo ago
  Fair point.
grayxu 8mo ago
Actually, Opus might achieve a lower cost with the help of TPUs.
DanielHall 8mo ago
Comparing open-source models like Qwen against Anthropic’s models is absolutely foolish. First of all, Anthropic has never disclosed the actual parameter count or architecture of their models. Second, it’s well known that these open-source models more or less distill from other models and use MoE, which allows them to run at much lower computational costs. Using Qwen as a comparison point only proves the blog post author is foolish. The article devoted such a large portion to discussing Qwen on OpenRouter, I find it hard to believe.
- yorwba 8mo ago
  Anthropic is obviously also aware of the benefits of MoE and distilling a larger model into a smaller one, so they could run a model of the same size as Alibaba's for the same inference cost if they want to. Or they can run a slightly larger model for slightly higher cost. They definitely aren't running a much larger model (except potentially as a teacher for distillation training) because then they wouldn't be able to hit the output speeds they're hitting.
  - hirako2000 8mo ago
    They are fully aware, but are playing a different game, R&D isn't something you flip a parameter and you get what the efficient oriented pipelines do. Chinese models were built on constraints. As we know limitations lead to innovation. So the "Chinese" R&D invested in optimisations. Teacher models were already there so they likely built the best distillation processes, along with the best MoE. Actually they published many of these works. Nuance, sure. Anthropic/OpenAI could revise their philosophy to adopt efficiency. But momentum can't be underestimated. Plus, dollar per optimisations is a different math altogether, it's not only about access to the latest Nvidia GPUs. At $400k the engineer pop a year, health coverage, pension contribution. Hardware efficiency doesn't weigh as much as making sure engineering focuses on.. the raw power factor, I suppose.
    - yorwba 8mo ago
      Every company is subject to constraints. A bigger budget is not an infinite budget. And there is no tradeoff between efficiency and raw power. An optimization that lets you build a similarly powerful model for less money also lets you build a more powerful model for the same amount of money. Honestly, I wonder what you think closed LLM companies do R&D on if not optimizations. And the nature of research is that most ideas that sound good turn out duds, so they already need to have an established process for testing many ideas quickly. Now if somebody publishes a new idea they haven't tried yet, setting up an experiment to try it out is just a routine task... But they aren't going to tell anybody the results, just quietly integrate it if it works.
tom_m 8mo ago
No they wouldn't. They have tons of funding. They absolutely can and do absorb costs like this. Don't think anyone is ever gonna tell you precise numbers (and it also varies based on workload of course)...but this is literally the business model of AI providers. They're goal (similar to Uber, DoorDash, Robin Hood, etc.) is to get mass adoption. Their business models only work at this kind of scale. It's completely impossible to have consumers pay $20-60/mo and be a profitable business without mass adoption where some are not using it as much as others...and, perhaps more importantly, the masses put pressure on their employers to pay for their tooling. This is why pricing does not need to come down. Quite literally I have engineers spending over $1,000/mo on Opus. That's the goal.

simianwords 8mo ago

>It is not. It's a terrible comparison. Qwen, deepseek and other Chinese models are known for their 10x or even better efficiency compared to Anthropic's. I find it a good comparison because it is a good baseline since we have zero insider knowledge of Anthropic. They give me an idea that a certain size of a model has a certain cost associated. I don't buy the 10x efficiency thing: they are just lagging behind the performance of current SOTA models. They perform much worse than the current models while also costing much less - exactly what I would expect. Current Qwen models perform as good as Sonnet 3 I think. 2 years later when Chinese models catchup with enough distillation attacks, they would be as good as Sonnet 4.6 and still be profitable.

coldtea 8mo ago
> I don't buy the 10x efficiency thing: they are just lagging behind the performance of current SOTA models. They perform much worse than the current models while also costing much less - exactly what I would expect. Define "much worse". +--------------------------------------+-------------+-----------+------------------+ | Benchmark | Claude Opus | DeepSeek | DeepSeek vs Opus | +--------------------------------------+-------------+-----------+------------------+ | SWE-Bench Verified (coding) | 80.9% | 73.1% | ~90% | | MMLU (knowledge) | ~91 | ~88.5 | ~97% | | GPQA (hard science reasoning) | ~79–80 | ~75–76 | ~95% | | MATH-500 (math reasoning) | ~78 | ~90 | ~115% | +--------------------------------------+-------------+-----------+------------------+
- Filligree 8mo ago
  Everyone who's used Opus knows it's better than the others in a way that isn't captured by the benchmarks. I would describe it as taste. Lots of models get really close on benchmarks, but benchmarks only tell us how good they are at solving a defined problem. Opus is far better at solving ill-defined ones.
  - coldtea 8mo ago
    >Everyone who's used Opus knows it's better than the others in a way that isn't captured by the benchmarks. I would describe it as taste. Ah, the "trust me bro" advantage. Couldn't it just be brand identity and familiarity?
    - vidarh 8mo ago
      I have a project where we've had Opus, Sonnet, Deepseek, Kimi, Qwen create and execute an aggregate total of about 350 plans so far, and the quality difference as measured in plans where the agent failed to complete the tasks on the first run is high enough that it comes out several times higher than Anthropics subscription prices, but probably cheaper than the API prices once we have improved the harness further - at present the challenge is that too much human intervention for the cheaper models drives up the cost. My dashboard goes from all green to 50/50 green/red for our agents whenever I switch from Claude to one of the cheaper agents... This is after investing a substantial amount of effort in "dumbing down" the prompts - e.g. adding a lot of extra wording to convince the dumber models to actually follow instructions - that is not necessary for Sonnet or Opus. I buy the benchmarks. The problem is that a 10% difference in the benchmarks makes the difference between barely usable and something that can consistently deliver working code unilaterally and require few review interventions. Basically, the starting point for "usable" on these benchmarks is already very far up the scale for a lot of tasks. I do strongly believe the moat is narrow - With 4.6 I switched from defaulting to Opus to defaulting to Sonnet for most tasks. I can fully see myself moving substantial workloads to a future iteration of Kimi, Qwen or Deepseek in 6-12 months once they actually start approaching Sonnet 4.5 level. But for my use at least, currently, they're at best competing with Athropics 3.x models in terms of real-world ability. That said, even now, I think if we were stuck with current models for 12 months, we might well also be able to build our way around this and get to a point where Deepseek and Kimi would be cheaper than Sonnet. Eventually we'll converge on good enough harnesses to get away with cheaper models for most uses, and the remaining appeal for the frontier models will be complex planning and actual hard work.
crooked-v 8mo ago
> distillation "attacks" I find it really funny that anyone can call it this with a straight face when all the American models are based on heaps of illegally pirated books and TOS-breaking website scraping in the first place.
- hirako2000 8mo ago
  Looters calling the police for being looted. Plus, Chines made distillation did good to the overall internet infrastructure. Millions of small Joe's WordPress website, paid and maintained out of good will, getting hammered by AI mining vs making the VC loaded pirates pay for what they seeded, I find the latter more fair.

jychang 8mo ago

That's a tautology. People think chinese models are 10x more efficient because they're 10x cheaper, and then you use that to claim that they're 10x more efficient. Opus isn't that expensive to host. Look at Amazon Bedrock's t/s numbers for Opus 4.5 vs other chinese models. They're around the same order of magnitude- which means that Opus has roughly the same amount of active params as the chinese models. Also, you can select BF16 or Q8 providers on openrouter.

re-thc 8mo ago
> That's a tautology. People think chinese models are 10x more efficient because they're 10x cheaper They do have different infrastructure / electricity costs and they might not run on nvidia hardware. It's not just the models.
- jychang 8mo ago
  Except there are providers that serve both chinese models AND opus as well. On the same hardware. Namely, Amazon Bedrock and Google Vertex. That means normalized infrastructure costs, normalized electricity costs, and normalized hardware performance. Normalized inference software stack, even (most likely). It's about a close of a 1 to 1 comparison as you can get. Both Amazon and Google serve Opus at roughly ~1/2 the speed of the chinese models. Note that they are not incentivized to slow down the serving of Opus or the chinese models! So that tells you the ratio of active params for Opus and for the chinese models.
  - Shakahs 8mo ago
    AWS and GCP both have their own custom inference chips, so a better example for hosting Opus on commodity hardware would be Digital Ocean.
  - giancarlostoro 8mo ago
    And Microsoft's Azure. It's on all 3 major cloud providers. Which tells me, they can make profit from these cloud providers without having to pay for any hardware. They just take a small enough cut. https://code.claude.com/docs/en/microsoft-foundry https://code.claude.com/docs/en/microsoft-foundry https://www.anthropic.com/news/claude-in-microsoft-foundry https://www.anthropic.com/news/claude-in-microsoft-foundry

lelanthran 8mo ago

> That being said not all users max out their plan, These are not cell phone plans which the average joe takes, they are plans purchased with the explicit goal of software development. I would guess that 99 out of every 100 plans are purchased with the explicit goal of maxing them out.

solumunus 8mo ago
There’s absolutely no way that’s true.
serial_dev 8mo ago
I’m not maxing them out… I have issues that I need to fix, features I need to develop, and I have things I want to learn. When I have a feeling that these tools will speed me up, I use them. My client pays for a couple of these tools in an enterprise deal, and I suspect most of us on the team work like that. If my goal was to max out every tool my client pays, I’d be working 24hrs a day and see no sunlight ever. I guess it’s like the all you can eat buffet. Everybody eats a lot, but if you eat so much that you throw up and get sick, you are special.
- bloppe 8mo ago
  [flagged]
rustystump 8mo ago
In saas this is not true. Most saas is highly profitable or was i suppose because they knew that most of their customers would never max out their plans.
Ginden 8mo ago
My employer bought me a Claude Max subscription. On heavy weeks I use 80% of the subscription. And among software engineers that I know, I'm a relatively heavy user. Why? Because in my experience, the bottleneck is in shareholders approving new features, not my ability to dish out code.
raihansaputra 8mo ago

Weaver_zhu 8mo ago

Agree, but I guess the Opus 4.6 is 10x larger, rather than Chinese models being 10x more efficient. It is said that GPT-4 is already a 1.6T model, and Llama 4 behemoth is also much bigger than Chinese open-weight models. Chinese tech companies are short of frontier GPUs, but they did a lot of innovations on inference efficiency (Deepseek CEO Liang himself shows up in the author list of the related published papers).

jychang 8mo ago
No, Opus cannot be 10x larger than the chinese models. If Opus was 10x larger than the chinese models, then Google Vertex/Amazon Bedrock would serve it 10x slower than Deepseek/Kimi/etc. That's not the case. They're in the same order of magnitude of speed.
- bakugo 8mo ago
  I agree that Opus almost definitely isn't anywhere near that big, but AWS throughput might not be a great way to measure model size. According to OpenRouter, AWS serves the latest Opus and Sonnet at roughly the same speed. It's likely that they simply allocate hardware differently per model.
  - jychang 8mo ago
    The numbers look about right. Opus 4.5 is about 1.5x the size of Sonnet 4.6, and Opus 4/4.1 is about 5x the size of Sonnet 4.5/4.6. Note that Opus 4.5 is about 1/3 the size of Opus 4/4.1 (and 1/3 the price in the API)
- Filligree 8mo ago
  They serve it about 2x slower. So it must have about 2x the active parameters. It could still be 10x larger overall, though that would not make it 10x more expensive.
  - jychang 8mo ago
    Yes, but I highly doubt they would increase sparsity much vs the chinese models. That's how you get Llama 4. Pretty much every major lab settled on ~3-5% sparsity for a reason.

Havoc 8mo ago

> Plus who knows what open routed providers do in term quantization The quantisation is shown on the provider section.

hirako2000 8mo ago
Fair point.

grayxu 8mo ago

Actually, Opus might achieve a lower cost with the help of TPUs.

DanielHall 8mo ago

Comparing open-source models like Qwen against Anthropic’s models is absolutely foolish. First of all, Anthropic has never disclosed the actual parameter count or architecture of their models. Second, it’s well known that these open-source models more or less distill from other models and use MoE, which allows them to run at much lower computational costs. Using Qwen as a comparison point only proves the blog post author is foolish. The article devoted such a large portion to discussing Qwen on OpenRouter, I find it hard to believe.

yorwba 8mo ago
Anthropic is obviously also aware of the benefits of MoE and distilling a larger model into a smaller one, so they could run a model of the same size as Alibaba's for the same inference cost if they want to. Or they can run a slightly larger model for slightly higher cost. They definitely aren't running a much larger model (except potentially as a teacher for distillation training) because then they wouldn't be able to hit the output speeds they're hitting.
- hirako2000 8mo ago
  They are fully aware, but are playing a different game, R&D isn't something you flip a parameter and you get what the efficient oriented pipelines do. Chinese models were built on constraints. As we know limitations lead to innovation. So the "Chinese" R&D invested in optimisations. Teacher models were already there so they likely built the best distillation processes, along with the best MoE. Actually they published many of these works. Nuance, sure. Anthropic/OpenAI could revise their philosophy to adopt efficiency. But momentum can't be underestimated. Plus, dollar per optimisations is a different math altogether, it's not only about access to the latest Nvidia GPUs. At $400k the engineer pop a year, health coverage, pension contribution. Hardware efficiency doesn't weigh as much as making sure engineering focuses on.. the raw power factor, I suppose.
  - yorwba 8mo ago
    Every company is subject to constraints. A bigger budget is not an infinite budget. And there is no tradeoff between efficiency and raw power. An optimization that lets you build a similarly powerful model for less money also lets you build a more powerful model for the same amount of money. Honestly, I wonder what you think closed LLM companies do R&D on if not optimizations. And the nature of research is that most ideas that sound good turn out duds, so they already need to have an established process for testing many ideas quickly. Now if somebody publishes a new idea they haven't tried yet, setting up an experiment to try it out is just a routine task... But they aren't going to tell anybody the results, just quietly integrate it if it works.

tom_m 8mo ago

No they wouldn't. They have tons of funding. They absolutely can and do absorb costs like this. Don't think anyone is ever gonna tell you precise numbers (and it also varies based on workload of course)...but this is literally the business model of AI providers. They're goal (similar to Uber, DoorDash, Robin Hood, etc.) is to get mass adoption. Their business models only work at this kind of scale. It's completely impossible to have consumers pay $20-60/mo and be a profitable business without mass adoption where some are not using it as much as others...and, perhaps more importantly, the masses put pressure on their employers to pay for their tooling. This is why pricing does not need to come down. Quite literally I have engineers spending over $1,000/mo on Opus. That's the goal.