7 ms·

I genuinely do not understand the evaluations of the US AI industry. The chinese models are so close and far cheaper

by jodleif 10mo ago

beastman82 10mo ago
Then you should short the market
newyankee 10mo ago
Yet tbh if the US industry had not moved ahead and created the race with FOMO it would not had been easier for Chinese strategy to work either. The nature of the race may change as yet though, and I am unsure if the devil is in the details, as in very specific edge cases that will work only with frontier models ?
jazzyjackson 10mo ago
Valuation is not based on what they have done but what they might do, I agree tho it's investment made with very little insight into Chinese research. I guess it's counting on deepseek being banned and all computers in America refusing to run open software by the year 2030 /snark
- bilbo0s 10mo ago
  >I guess it's counting on deepseek being banned And the people making the bets are in a position to make sure the banning happens. The US government system being what it is. Not that our leaders need any incentive to ban Chinese tech in this space. Just pointing out that it's not necessarily a "bet". "Bet" imply you don't know the outcome and you have no influence over the outcome. Even "investment" implies you don't know the outcome. I'm not sure that's the case with these people?
  - coliveira 10mo ago
    Exactly. "Business investment" these days means that the people involved will have at least some amount of power to determine the winning results.
- jodleif 10mo ago
  > Valuation is not based on what they have done but what they might do Exactly what I’m thinking. Chinese models catching rapidly. Soon to be on-par with the big dogs.
  - ksynwa 10mo ago
    Even if they do continue to lag behind they are a good bet against monopolisation by proprietary vendors.
    - coliveira 10mo ago
      They would if corporations were allowed to run these models. I fully expect the US government to prohibit corporations from doing anything useful with Chinese models (full censorship). It's the same game they use with chips.
jasonsb 10mo ago
It's all about the hardware and infrastructure. If you check OpenRouter, no provider offers a SOTA chinese model matching the speed of Claude, GPT or Gemini. The chinese models may benchmark close on paper, but real-world deployment is different. So you either buy your own hardware in order to run a chinese model at 150-200tps or give up an use one of the Big 3. The US labs aren't just selling models, they're selling globally distributed, low-latency infrastructure at massive scale. That's what justifies the valuation gap. Edit: It looks like Cerebras is offering a very fast GLM 4.6
- csomar 10mo ago
  According to OpenRouter, z.ai is 50% faster than Anthropic; which matches my experience. z.ai does have frequent downtimes but so does Claude.
- jodleif 10mo ago
  Assuming your hardware premise is right (and lets be honest, nobody really wants to send their data to chinese providers) You can use a provider like Cerebras, Groq?
- observationist 10mo ago
  The network effects of using consistently behaving models and maintaining API coverage between updates is valuable, too - presumably the big labs are including their own domains of competence in the training, so Claude is likely to remain being very good at coding, and behave in similar ways, informed and constrained by their prompt frameworks, so that interactions will continue to work in predictable ways even after major new releases occur, and upgrades can be clean. It'll probably be a few years before all that stuff becomes as smooth as people need, but OAI and Anthropic are already doing a good job on that front. Each new Chinese model requires a lot of testing and bespoke conformance to every task you want to use it for. There's a lot of activity and shared prompt engineering, and some really competent people doing things out in the open, but it's generally going to take a lot more expert work getting the new Chinese models up to snuff than working with the big US labs. Their product and testing teams do a lot of valuable work.
isamuel 10mo ago
There is a great deal of orientalism --- it is genuinely unthinkable to a lot of American tech dullards that the Chinese could be better at anything requiring what they think of as "intelligence." Aren't they Communist? Backward? Don't they eat weird stuff at wet markets? It reminds me, in an encouraging way, of the way that German military planners regarded the Soviet Union in the lead-up to Operation Barbarossa. The Slavs are an obviously inferior race; their Bolshevism dooms them; we have the will to power; we will succeed. Even now, when you ask questions like what you ask of that era, the answers you get are genuinely not better than "yes, this should have been obvious at the time if you were not completely blinded by ethnic and especially ideological prejudice."
- newyankee 10mo ago
  but didn't Chinese already surpass the rest of the world in Solar, batteries, EVs among other things ?
  - cyberlimerence 10mo ago
    They did, but the goalposts keep moving, so to speak. We're approximately here : advanced semiconductors, artificial intelligence, reusable rockets, quantum computing, etc. Chinese will never catch up. /s
- mosselman 10mo ago
  Back when deepseek came out and people were tripping over themselves shouting it was so much better than what was out there, it just wasn’t good. It might be this model is super good, I haven’t tried it, but to say the Chinese models are better is just not true. What I really love though is that I can run them (open models) on my own machine. The other day I categorised images locally using Qwen, what a time to be alive. Further even than local hardware, open models make it possible to run on providers of choice, such as European ones. Which is great! So I love everything about the competitive nature of this.
  - CamperBob2 10mo ago
espadrine 10mo ago
Two aspects to consider: 1. Chinese models typically focus on text. US and EU models also bear the cross of handling image, often voice and video. Supporting all those is additional training costs not spent on further reasoning, tying one hand in your back to be more generally useful. 2. The gap seems small, because so many benchmarks get saturated so fast. But towards the top, every 1% increase in benchmarks is significantly better. On the second point, I worked on a leaderboard that both normalizes scores, and predicts unknown scores to help improve comparisons between models on various criteria: https://metabench.organisons.com/ https://metabench.organisons.com/ You can notice that, while Chinese models are quite good, the gap to the top is still significant. However, the US models are typically much more expensive for inference, and Chinese models do have a niche on the Pareto frontier on cheaper but serviceable models (even though US models also eat up the frontier there).
- jodleif 10mo ago
  1. Have you seen the Qwen offerings? They have great multi-modality, some even SOTA.
  - brabel 10mo ago
    Qwen Image and Image Edit were among the best image models until Nano Banana Pro came along. I have tried some open image models and can confirm , the Chinese models are easily the best or very close to the best, but right now the Google model is even better... we'll see if the Chinese catch up again.
    - BoorishBears 10mo ago
      I'd say Google still hasn't caught up on the smaller model side at all, but we've all been (rightfully) wowed enough by Pro to ignore that for now. Nano Banano Pro starts at 15 cents per image at <2k resolution, and is not strictly better than Seedream 4.0: yet the latter does 4K for 3 cents per image. Add in the power of fine-tuning on their open weight models and I don't know if China actually needs to catch up. I finetuned Qwen Image on 200 generations from Seedream 4.0 that were cleaned up with Nano Banana Pro, and got results that were as good and more reliable than either model could achieve otherwise.
Bolwin 10mo ago
Third party providers rarely support caching. With caching the expensive US models end up being like 2x the price (e.g sonnet) and often much cheaper (e.g gpt-5 mini) If they start caching then US companies will be completely out priced.
fastball 10mo ago
They're not that close (on things like LMArena) and being cheaper is pretty meaningless when we are not yet at the point where LLMs are good enough for autonomy.
mrinterweb 10mo ago
I would expect one of the motivations for making these LLM model weights open is to undermine the valuation of other players in the industry. Open models like this must diminish the value prop of the frontier focused companies if other companies can compete with similar results at competitive prices.
rprend 10mo ago
People pay for products, not models. OpenAI and Anthropic make products (ChatGPT, Claude Code).
Plaoo 10mo ago
[dead]

I genuinely do not understand the evaluations of the US AI industry. The chinese models are so close and far cheaper

by jodleif 10mo ago

I genuinely do not understand the evaluations of the US AI industry. The chinese models are so close and far cheaper

beastman82 10mo ago

Then you should short the market

newyankee 10mo ago

Yet tbh if the US industry had not moved ahead and created the race with FOMO it would not had been easier for Chinese strategy to work either. The nature of the race may change as yet though, and I am unsure if the devil is in the details, as in very specific edge cases that will work only with frontier models ?

jazzyjackson 10mo ago

Valuation is not based on what they have done but what they might do, I agree tho it's investment made with very little insight into Chinese research. I guess it's counting on deepseek being banned and all computers in America refusing to run open software by the year 2030 /snark

bilbo0s 10mo ago
>I guess it's counting on deepseek being banned And the people making the bets are in a position to make sure the banning happens. The US government system being what it is. Not that our leaders need any incentive to ban Chinese tech in this space. Just pointing out that it's not necessarily a "bet". "Bet" imply you don't know the outcome and you have no influence over the outcome. Even "investment" implies you don't know the outcome. I'm not sure that's the case with these people?
- coliveira 10mo ago
  Exactly. "Business investment" these days means that the people involved will have at least some amount of power to determine the winning results.
jodleif 10mo ago
> Valuation is not based on what they have done but what they might do Exactly what I’m thinking. Chinese models catching rapidly. Soon to be on-par with the big dogs.
- ksynwa 10mo ago
  Even if they do continue to lag behind they are a good bet against monopolisation by proprietary vendors.
  - coliveira 10mo ago
    They would if corporations were allowed to run these models. I fully expect the US government to prohibit corporations from doing anything useful with Chinese models (full censorship). It's the same game they use with chips.

jasonsb 10mo ago

It's all about the hardware and infrastructure. If you check OpenRouter, no provider offers a SOTA chinese model matching the speed of Claude, GPT or Gemini. The chinese models may benchmark close on paper, but real-world deployment is different. So you either buy your own hardware in order to run a chinese model at 150-200tps or give up an use one of the Big 3. The US labs aren't just selling models, they're selling globally distributed, low-latency infrastructure at massive scale. That's what justifies the valuation gap. Edit: It looks like Cerebras is offering a very fast GLM 4.6

csomar 10mo ago
According to OpenRouter, z.ai is 50% faster than Anthropic; which matches my experience. z.ai does have frequent downtimes but so does Claude.
jodleif 10mo ago
Assuming your hardware premise is right (and lets be honest, nobody really wants to send their data to chinese providers) You can use a provider like Cerebras, Groq?
observationist 10mo ago
The network effects of using consistently behaving models and maintaining API coverage between updates is valuable, too - presumably the big labs are including their own domains of competence in the training, so Claude is likely to remain being very good at coding, and behave in similar ways, informed and constrained by their prompt frameworks, so that interactions will continue to work in predictable ways even after major new releases occur, and upgrades can be clean. It'll probably be a few years before all that stuff becomes as smooth as people need, but OAI and Anthropic are already doing a good job on that front. Each new Chinese model requires a lot of testing and bespoke conformance to every task you want to use it for. There's a lot of activity and shared prompt engineering, and some really competent people doing things out in the open, but it's generally going to take a lot more expert work getting the new Chinese models up to snuff than working with the big US labs. Their product and testing teams do a lot of valuable work.

isamuel 10mo ago

There is a great deal of orientalism --- it is genuinely unthinkable to a lot of American tech dullards that the Chinese could be better at anything requiring what they think of as "intelligence." Aren't they Communist? Backward? Don't they eat weird stuff at wet markets? It reminds me, in an encouraging way, of the way that German military planners regarded the Soviet Union in the lead-up to Operation Barbarossa. The Slavs are an obviously inferior race; their Bolshevism dooms them; we have the will to power; we will succeed. Even now, when you ask questions like what you ask of that era, the answers you get are genuinely not better than "yes, this should have been obvious at the time if you were not completely blinded by ethnic and especially ideological prejudice."

newyankee 10mo ago
but didn't Chinese already surpass the rest of the world in Solar, batteries, EVs among other things ?
- cyberlimerence 10mo ago
  They did, but the goalposts keep moving, so to speak. We're approximately here : advanced semiconductors, artificial intelligence, reusable rockets, quantum computing, etc. Chinese will never catch up. /s
mosselman 10mo ago
Back when deepseek came out and people were tripping over themselves shouting it was so much better than what was out there, it just wasn’t good. It might be this model is super good, I haven’t tried it, but to say the Chinese models are better is just not true. What I really love though is that I can run them (open models) on my own machine. The other day I categorised images locally using Qwen, what a time to be alive. Further even than local hardware, open models make it possible to run on providers of choice, such as European ones. Which is great! So I love everything about the competitive nature of this.
- CamperBob2 10mo ago

espadrine 10mo ago

Two aspects to consider: 1. Chinese models typically focus on text. US and EU models also bear the cross of handling image, often voice and video. Supporting all those is additional training costs not spent on further reasoning, tying one hand in your back to be more generally useful. 2. The gap seems small, because so many benchmarks get saturated so fast. But towards the top, every 1% increase in benchmarks is significantly better. On the second point, I worked on a leaderboard that both normalizes scores, and predicts unknown scores to help improve comparisons between models on various criteria: https://metabench.organisons.com/ https://metabench.organisons.com/ You can notice that, while Chinese models are quite good, the gap to the top is still significant. However, the US models are typically much more expensive for inference, and Chinese models do have a niche on the Pareto frontier on cheaper but serviceable models (even though US models also eat up the frontier there).

jodleif 10mo ago
1. Have you seen the Qwen offerings? They have great multi-modality, some even SOTA.
- brabel 10mo ago
  Qwen Image and Image Edit were among the best image models until Nano Banana Pro came along. I have tried some open image models and can confirm , the Chinese models are easily the best or very close to the best, but right now the Google model is even better... we'll see if the Chinese catch up again.
  - BoorishBears 10mo ago
    I'd say Google still hasn't caught up on the smaller model side at all, but we've all been (rightfully) wowed enough by Pro to ignore that for now. Nano Banano Pro starts at 15 cents per image at <2k resolution, and is not strictly better than Seedream 4.0: yet the latter does 4K for 3 cents per image. Add in the power of fine-tuning on their open weight models and I don't know if China actually needs to catch up. I finetuned Qwen Image on 200 generations from Seedream 4.0 that were cleaned up with Nano Banana Pro, and got results that were as good and more reliable than either model could achieve otherwise.

Bolwin 10mo ago

Third party providers rarely support caching. With caching the expensive US models end up being like 2x the price (e.g sonnet) and often much cheaper (e.g gpt-5 mini) If they start caching then US companies will be completely out priced.

fastball 10mo ago

They're not that close (on things like LMArena) and being cheaper is pretty meaningless when we are not yet at the point where LLMs are good enough for autonomy.

mrinterweb 10mo ago

I would expect one of the motivations for making these LLM model weights open is to undermine the valuation of other players in the industry. Open models like this must diminish the value prop of the frontier focused companies if other companies can compete with similar results at competitive prices.

rprend 10mo ago

People pay for products, not models. OpenAI and Anthropic make products (ChatGPT, Claude Code).

Plaoo 10mo ago

[dead]