7 ms·

On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or AS

On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment. The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies. The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.

chvid 5d ago
Huawei uses their own non-standard HBM called HiZQ probably not produced by CXMT.
- deleted 5d ago
  [deleted]
andy_ppp 6d ago
Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.
- cubefox 5d ago
  According to SemiAnalysis, both inference and post-training (RLVR) is mostly memory bandwidth bound. Only pre-training is compute bound, but it now only takes a small share of overall data center capacity. https://x.com/EugeneNg/status/2099315982959616369 https://x.com/EugeneNg/status/2099315982959616369
- darig 5d ago
  [dead]
- martinald 5d ago
  Not quite, it's got quite a bit more complicated with agentic use cases. Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result. Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound. Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
  - com2kid 5d ago
    I can easily use close to 100 million input tokens a day. A few million output tokens but at the end of the day maybe a thousand or so lines of code get written.
  - andy_ppp 5d ago
    This is why I was careful to specify tokens per second not time to first token which is the prefill step you’re talking about. Clearly to run these models well you need both but as I said adding more chips or compute units can give you more latency where as overall memory bandwidth (throughput) is limited by access to fast memory.
6510 6d ago
I read HBM yields are 25-30% (vs 80-90%) making them 3 to 5 times as expensive. They are 4-5 years behind, that probably means 1-2 in Chinese time.
- throwaway85825 6d ago
  Does that include yield from packaging?
  - 6510 5d ago
    The 3 to 5 times more expensive does.
vatsachak 6d ago
China should invest in an analog inference chip. It's a hail mary but why not.
- geysersam 5d ago
  They can probably afford to do both.
- sroussey 5d ago
  That would be like skipping land line phones for mobile...
  - clipsy 5d ago
    Developing countries have done exactly that in many cases.
    - mitxela 5d ago
      And skipped DSL for fiber, and skipped credit cards for banking apps.
- bell-cot 5d ago
  I'd assume that they have - but will keep mum 'till they have a major breakthrough or large-scale operational deployment to announce.
- danielheath 5d ago
  The two challenges there are firstly - that analog design has been a separate electrical engineering school for most of a century, so there are few who could design it - and secondly - that every single chip will have subtle variations in its computations, necessitating some sort of model finetuning per chip. Possibly the chip could be characterised at the factory, and ship with the characterisation data burned into a controller rom or something, but if that doesn’t pan out the whole thing is likely a non-starter. If it could be made to work, you could run a fable-grade model in tens of watts.
jacquesm 5d ago
The only thing the West is achieving here is that sooner or later China will be able to compete on their own terms rather than ours. It may buy some time but the end result is very predictable.
- stingraycharles 5d ago
  It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
  - mrheosuper 5d ago
    they expect the Chinese to bend their knee and beg for the sweet, sweet chip
  - qaq 5d ago
    they were sold an idea the AGI is right around the corner so even a slight delay might be critical to secure the "win"
  - adrianN 5d ago
    They expect to retire with a decent amount of money before things go down hill.
  - odo1242 5d ago
    They probably weren't expecting AI foundation models / model research to be quite as fungible as they are
    - safety1st 5d ago
      Yes. Over the last 20 years the economy has become dysfunctional. It no longer really resembles a free market; monopolies have established barriers to entry everywhere. And the biggest investors are awash with helicopter money that's been doled out for favors by the political class. So those investors have tons of cash to burn and surprisingly few opportunities. Even within Silicon Valley/VC there are surprisingly few who seem to really understand the fundamental economics of software. Or perhaps those economics just aren't that important when you have billions of dollars on hand and cash is obviously not going to get you a return. Any whisper of possible exponential growth is worth throwing money at. Crypto? Why not. AI? Why not. Datacenters? Why not. Tulips? Why not. This is by all definitions an empire in decline. Everything is broken or fake. Everyone is afraid to do what needs to be done. Power forbids it. So we're all just waiting for the other shoe to drop. Our secret police aren't as bad as the late stage USSR's yet, but hold Uncle Sam's beer... (That's not a recommendation to try and time the nadir, by the way, as it could easily be 50 years away.)
KurSix 5d ago
China has had a very strong incentive to recreate lithography and fab equipment for years, but a lot of the specialized HBM packaging chain was probably much lower priority until relatively recently
FooBarWidget 5d ago
The "abysmal" yield using DUV is an overblown statement by western commentators who don't look closer at the development. Certainly yield is worse than with EUV, but after multiple iterations of development, yield has become pretty good, within economically acceptable bounds, still making scaling possible. Volume is still ramping up. A lot of capacity will come online in 2027. The removal of western middlemen such as Mediatek actually improved the economics. Further yield improvements are still coming. Volume is already so high that domestically made phones and NPUs are making a real impact, yet are also selling like hot cakes.
- HarHarVeryFunny 5d ago
  It depends on what you are trying to make, but there are workarounds for most things, and it becomes more of a cost and volume issue than a showstopper. If your yield is low and you still want the volume, then run more wafers, but then you need access to more DUV machines and more wafers. If your defect density is too high, then design smaller chips with a higher chance of avoiding defects, which is what Hauwei are doing - split processor into multiple chiplets (NVidia do this too, to raise yields and reduce cost). HBM3E usually has 5-6000 vias, but doing this with multi-pattern DUV without defects is tough, so CXMT currently drop that to 3000, at a cost of some loss of thermal and voltage stability. HBM:processor production ratios aren't what Huawei would like, so they mitigate this at system level by putting an optical memory bus on the GPU chip and sharing memory across the system. Not everything is a huge LLM - smaller models like recommendation systems don't have so many parameters and need so much memory, so why waste HBM on them? ByteDance use their SeedChip acelerator for this, currently made by TSMC (using an older sanctions-approved 28mm process), which instead etches dense RRAM on die beside the processor. There is also a time dynamic to this, with China still using pre-sanctions equipment and chips as their domestic alternatives ramp up to replace them. One interesting part of this is HBM ... HBM is very demanding to make, and everyone had problems with it, with initially only SK Hynix being successful. Samsung and Micron took a year or so to catch up, and during this time Samsung had made a ton of HBM that didn't meet NVidia's specifications, so ended up, pre-sanctions, selling it to China, where it has acted as a stockpile to carry them over as CXMT's domestic capacity ramps up. China seems to be doing fine. Of course they would like sanctions lifted, moreso for HBM than anything else, but all that sanctions have really achieved is accelerating their semiconductor independence. They are still building 1T+ SOTA models, standing up 100K clusters of domestic AI accelerators, etc, etc.
toxik 5d ago
This competition will be great for us consumers for a while. Until China undercuts everybody else for long enough to essentially monopolize the market. This pattern has repeated itself across many markets already.

chvid 5d ago

Huawei uses their own non-standard HBM called HiZQ probably not produced by CXMT.

deleted 5d ago
[deleted]

andy_ppp 6d ago

Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.

cubefox 5d ago
According to SemiAnalysis, both inference and post-training (RLVR) is mostly memory bandwidth bound. Only pre-training is compute bound, but it now only takes a small share of overall data center capacity. https://x.com/EugeneNg/status/2099315982959616369 https://x.com/EugeneNg/status/2099315982959616369
darig 5d ago
[dead]
martinald 5d ago
Not quite, it's got quite a bit more complicated with agentic use cases. Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result. Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound. Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
- com2kid 5d ago
  I can easily use close to 100 million input tokens a day. A few million output tokens but at the end of the day maybe a thousand or so lines of code get written.
- andy_ppp 5d ago
  This is why I was careful to specify tokens per second not time to first token which is the prefill step you’re talking about. Clearly to run these models well you need both but as I said adding more chips or compute units can give you more latency where as overall memory bandwidth (throughput) is limited by access to fast memory.

6510 6d ago

I read HBM yields are 25-30% (vs 80-90%) making them 3 to 5 times as expensive. They are 4-5 years behind, that probably means 1-2 in Chinese time.

throwaway85825 6d ago
Does that include yield from packaging?
- 6510 5d ago
  The 3 to 5 times more expensive does.

vatsachak 6d ago

China should invest in an analog inference chip. It's a hail mary but why not.

geysersam 5d ago
They can probably afford to do both.
sroussey 5d ago
That would be like skipping land line phones for mobile...
- clipsy 5d ago
  Developing countries have done exactly that in many cases.
  - mitxela 5d ago
    And skipped DSL for fiber, and skipped credit cards for banking apps.
bell-cot 5d ago
I'd assume that they have - but will keep mum 'till they have a major breakthrough or large-scale operational deployment to announce.
danielheath 5d ago
The two challenges there are firstly - that analog design has been a separate electrical engineering school for most of a century, so there are few who could design it - and secondly - that every single chip will have subtle variations in its computations, necessitating some sort of model finetuning per chip. Possibly the chip could be characterised at the factory, and ship with the characterisation data burned into a controller rom or something, but if that doesn’t pan out the whole thing is likely a non-starter. If it could be made to work, you could run a fable-grade model in tens of watts.

jacquesm 5d ago

The only thing the West is achieving here is that sooner or later China will be able to compete on their own terms rather than ours. It may buy some time but the end result is very predictable.

stingraycharles 5d ago
It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
- mrheosuper 5d ago
  they expect the Chinese to bend their knee and beg for the sweet, sweet chip
- qaq 5d ago
  they were sold an idea the AGI is right around the corner so even a slight delay might be critical to secure the "win"
- adrianN 5d ago
  They expect to retire with a decent amount of money before things go down hill.
- odo1242 5d ago
  They probably weren't expecting AI foundation models / model research to be quite as fungible as they are
  - safety1st 5d ago
    Yes. Over the last 20 years the economy has become dysfunctional. It no longer really resembles a free market; monopolies have established barriers to entry everywhere. And the biggest investors are awash with helicopter money that's been doled out for favors by the political class. So those investors have tons of cash to burn and surprisingly few opportunities. Even within Silicon Valley/VC there are surprisingly few who seem to really understand the fundamental economics of software. Or perhaps those economics just aren't that important when you have billions of dollars on hand and cash is obviously not going to get you a return. Any whisper of possible exponential growth is worth throwing money at. Crypto? Why not. AI? Why not. Datacenters? Why not. Tulips? Why not. This is by all definitions an empire in decline. Everything is broken or fake. Everyone is afraid to do what needs to be done. Power forbids it. So we're all just waiting for the other shoe to drop. Our secret police aren't as bad as the late stage USSR's yet, but hold Uncle Sam's beer... (That's not a recommendation to try and time the nadir, by the way, as it could easily be 50 years away.)

KurSix 5d ago

China has had a very strong incentive to recreate lithography and fab equipment for years, but a lot of the specialized HBM packaging chain was probably much lower priority until relatively recently

FooBarWidget 5d ago

The "abysmal" yield using DUV is an overblown statement by western commentators who don't look closer at the development. Certainly yield is worse than with EUV, but after multiple iterations of development, yield has become pretty good, within economically acceptable bounds, still making scaling possible. Volume is still ramping up. A lot of capacity will come online in 2027. The removal of western middlemen such as Mediatek actually improved the economics. Further yield improvements are still coming. Volume is already so high that domestically made phones and NPUs are making a real impact, yet are also selling like hot cakes.

HarHarVeryFunny 5d ago
It depends on what you are trying to make, but there are workarounds for most things, and it becomes more of a cost and volume issue than a showstopper. If your yield is low and you still want the volume, then run more wafers, but then you need access to more DUV machines and more wafers. If your defect density is too high, then design smaller chips with a higher chance of avoiding defects, which is what Hauwei are doing - split processor into multiple chiplets (NVidia do this too, to raise yields and reduce cost). HBM3E usually has 5-6000 vias, but doing this with multi-pattern DUV without defects is tough, so CXMT currently drop that to 3000, at a cost of some loss of thermal and voltage stability. HBM:processor production ratios aren't what Huawei would like, so they mitigate this at system level by putting an optical memory bus on the GPU chip and sharing memory across the system. Not everything is a huge LLM - smaller models like recommendation systems don't have so many parameters and need so much memory, so why waste HBM on them? ByteDance use their SeedChip acelerator for this, currently made by TSMC (using an older sanctions-approved 28mm process), which instead etches dense RRAM on die beside the processor. There is also a time dynamic to this, with China still using pre-sanctions equipment and chips as their domestic alternatives ramp up to replace them. One interesting part of this is HBM ... HBM is very demanding to make, and everyone had problems with it, with initially only SK Hynix being successful. Samsung and Micron took a year or so to catch up, and during this time Samsung had made a ton of HBM that didn't meet NVidia's specifications, so ended up, pre-sanctions, selling it to China, where it has acted as a stockpile to carry them over as CXMT's domestic capacity ramps up. China seems to be doing fine. Of course they would like sanctions lifted, moreso for HBM than anything else, but all that sanctions have really achieved is accelerating their semiconductor independence. They are still building 1T+ SOTA models, standing up 100K clusters of domestic AI accelerators, etc, etc.

toxik 5d ago

This competition will be great for us consumers for a while. Until China undercuts everybody else for long enough to essentially monopolize the market. This pattern has repeated itself across many markets already.