19 ms·

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

523 points by hd4 1y ago

Paper: https://dl.acm.org/doi/10.1145/3731569.3764815 https://dl.acm.org/doi/10.1145/3731569.3764815

hunglee2 1y ago
The US attempt to slow down China's technological development succeeds on the basis of preventing China from directly following the same path, but may backfire in the sense it forces innovation by China in a different direction. The overall outcome for us all may be increase efficiency as a result of this forced innovation, especially if Chinese companies continue to open source their advances, so we may in the end have reason to thank the US for their civilisational gate keeping
- rzerowan 1y ago
  Fingers crossed for convergence rather than divergence in the technical standards.Although the way hings are going it looks like the 2 stacks will diverge sooner rather than later , with the US+ banning the use of CHN models while simultaneosly banning the export of it quasi-open models. We may very well end up in a situation like the old PAL vs NTSC video standard where the PAL(EU/Asia/AFrica) and NTSC(America's/Japan) gradually converged with the adoption of digital formats. Instead here would be a divergence based on geopolitical considerations.
  - hunglee2 1y ago
    positive take: a bifurcated tech tree might give us (humanity) a better chance of faster advancement, as it would be a persistent A/B test in live environment. Where I would join you in the crossing of fingers is to ensure such A/B testing is competitive but not destructive. We may even evolve to a situation of complementarity, an American Ying vs the Chinese Yang. Lets hope so!
- reliabilityguy 1y ago
  Tbh this whole situation reminds of how Japan excelled in making a lot more with a lot less after WW2, e.g., fuel-efficient engines, light cars, etc. these constraints were not present in the US (and to some extent in Europe), and resulted in US cars being completely not competitive in non-US markets.
  - dataviz1000 1y ago
    I've been in Chile, Peru, Colombia, Panama, and Costa Rica. The streets are flooded with cheap Chinese cars and I see more BYD than American cars. If the car wasn't made in Japan or Korea which probably account for most of the cars, it was likely made in China. Moreover, I haven't been in countries with the closest ties to China.
    - sofixa 1y ago
      > The streets are flooded with cheap Chinese cars and I see more BYD than American cars This isn't surprising in any way, American "cars" (quotes because the vast majority of what American manufacturers pump out isn't cars, it's trucks) haven't been competitive in decades. The only globally competitive vehicles were developed in Europe by GM Europe (Opel, since sold to PSA now Stellantis) or Ford Europe (which axed all models bar the Puma). The rest is too big, expensive and inefficient from the vast majority of uses. Tariffs and good marketing keep American car manufacturers in business in the US, but those don't work in most other markets. The more appropriate comparison is with European automakers such as VW Group, Stellantis (Peugeot, Citroën, DS, Fiat, Chrysler, Dodge, Ram), Renault. And there too BYD is winning as well in mosy countries, but at least there's a comparison possible.
      - Workaccount2 1y ago
        The US couldn't compete anyway. The US is an advanced economy and will struggle to compete in non-advanced categories. It's like trying to level your MMORPG character to 100 by only farming in lvl 30-40 mob areas. It's really not worth it and mostly forced.
- segmondy 1y ago
  may backfire? it's a bit too late for that. go to 2024, western labs were crushing it. it's now 2025, and from china, we have deepseek, qwen, kimi, glm, ernie and many more capable models keeping up with western labs. there are actually now more chinese labs releasing sota models than western labs.
  - hunglee2 1y ago
    too early to call a winner, though it is disappointing to see US withdrawal from open source. Still the main outcome of open source is distribution / diffusion of the idea, so it will inevitably mean US open source will come back, hopefully via some grass roots maniac, there will be a Linus-like character emerge at some point
    - segmondy 1y ago
      i'm not calling a winner, i'm just saying that the chinese have caught up despite the embargo. google, openai & anthrophic have phenomenal models. i stopped using openai & anthropic after they called for open weight/source regulation. i use google because they offer gemma and i got a year gemini-pro subscription for free, use openai gpt-oss-120b since i can run it at home, and the only model i currently pay for is a chinese model.
    - mixologist 1y ago
      user growth has slowed. the technology that should help users is only being pushed from the top, while users refuse to use it. openai pivoted to porn. does it really feel like they have a chance to recover all the expenses in the future? crypto grifters pivoted to ai and, same as last time, normal people don’t want to have anything to do with them. considering the amount of money burned on this garbage, i think we can at least declare a looser.
  - Workaccount2 1y ago
    But they aren't keeping up They are lauded for the ability to cost ratio, or their ability to parameter ratio, but virtually everyone using LLMs for productive work are using ChatGPT/Gemini/Claude. They are kind of like Huffy bicycles. Good value, work well, but if you go to any serious event, no one will be riding one.
- dlisboa 1y ago
  History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years. In many senses there's hubris in the western* view of China accomplishments: most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the names of AI researchers there's a strong pattern even if some are currently plying their trade in the west. --- * I hate the term "western" because some "westeners" use it to separated what they think are "civilized" from "uncivilized", hence for them LATAM is not "western" even though everything about LATAM countries is western.
  - achierius 1y ago
    > most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the names of AI researchers there's a strong pattern even if some are currently plying their trade in the west. While I don't disagree with your overall point, it's important to recognize that this is only a phenomenon of the last ~30 years, and to avoid falling into the trapn of Han racial chauvinism. E.g. there were ~no Chinese scientists in Germany in the 70s but they were heavily innovating nevertheless.
    - dlisboa 1y ago
      Absolutely. China obviously has a longer history with innovation but they like to make it seem everything was invented by them at some point in the past. I'd say newer technology is where China has had a bigger impact. Consequently newer tech is precisely where global cooperation is most required so no country can really do it by themselves. We could even say no country, western or otherwise, has been doing it on their own for the past 500 years or so but alas...
      - tsunamifury 1y ago
        It’s more helpful to think of China as an accelerant rather than an innovator in this position.
- notepad0x90 1y ago
  I think anti-immigrant rhetoric will have the most impact against the US. A lot of the people innovating on this stuff are being maligned and leaving in droves. Aside from geography, attracting talent from all over the world is the one edge the US has a nation over countries like China. But now the US is trying to be xenophobic like China, restrict tech import/export like China but compete against 10x population and lack of similar levels of internal strife and fissures. The world, even Europe is looking for a new country to take on a leader/superpower role. China isn't there yet, but it might get there in a few years after their next-gen fighter jets and catching up to ASML. But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india.
  - dlisboa 1y ago
    > But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india. That's a strength. Them not having interest in global domination and regime change other than their backyard is what allows them to easily make partners in Africa and LATAM, the most important regions for raw materials.
    - notepad0x90 1y ago
      You would think so, but historically that's why they never became more than a regional power. Empires for millennia craved trade with China but only the mongols from that region made it all the way to western europe in their invasions. It is a strength, if their goal is to have a stable and prosperous country long term, and that seems to be what they want. good for them. But nature abhors a vacuum, so there will always be an empire at the top of the food chain. Such empires want to maximize wealth for their people and secure them against threats, that's why invasions and exploitation of weaker countries happens. That game hasn't changed. Friendly relations work, until you need a lot of resources from a country that doesn't want to give it up. Or, like with the US, when they're opening up military bases next to your borders and you need a buffer state. Or, when naval blockades and sanctions are being enforced against your country for not complying with extra-sovereign demands. History shows that countries content with what they have collapse or weaken very quickly. China will have a population crisis in a few decades for example, and it won't have the large manufacturing base and its people will be too used to luxuries to go back to slaving for western countries for pennies. Keep in mind that the current china itself is so great and prosperous because of all the invasions it did against western china and satellite states like Vietnam and north Korea (the US isn't special in this regard).
- IT4MD 1y ago
  I believe this is an Pollyanna take on AI. There is nothing about humans that tells us humans will bring AI to fruition for the other humans and a mountain of evidence showing how it will be used to abuse humans instead....for profits/power/whatever horse shit the masters of the universe have decided upon.
- amelius 1y ago
  Another outcome may be that we now have to learn Chinese to understand their datasheets ...
  - anonzzzies 1y ago
    I was doing this in the 70-80s with electronics from Hong Kong and Japan. The nice cheap stuff ( I was very young ) was all sheets in things I basically had to pattern match against notes of others on BBS and meetups.
- myth_drannon 1y ago
  China's innovation relies on the stolen western IP, without it, China is nothing. Also USSR/Russia is no longer a scientific powerhouse that can supply China with some military innovation. A dictatorship combined with cheap labour it 100% guarantees that the country's innovation is stunted, no matter what the Chinese propaganda claims.
  - Hikikomori 1y ago
    And the US has never stolen IP?
    - myth_drannon 1y ago
      Corporate espionage is ever present but it is criminalized. The only time US as a country did that you can say "stole IP" was after WII when it took Nazi rocket scientists and technology. China is the opposite; stealing tech is done by the state apparatus (same was done by USSR and reverse engineering computers for example). Frankly I'm not surprised that this is done, probably if US was so behind it would have done the same to reduce the gap. Everyone is trying to survive and outsmart and outwit the other, instead of collaborating.
      - currydove 1y ago
        Negative. https://interestingengineering.com/innovation/the-spies-who-launched-americas-industrial-revolution https://interestingengineering.com/innovation/the-spies-who-... https://yawboadu.substack.com/p/stealing-success-how-ip-theft-built https://yawboadu.substack.com/p/stealing-success-how-ip-thef... https://ipwatchdog.com/2017/07/05/americas-industrial-revolution-based-trade-secret-theft/ https://ipwatchdog.com/2017/07/05/americas-industrial-revolu...
      - g8oz 1y ago
        You are mistaken about American intellectual property theft. They engaged in extensive IP theft from Britain in the 18th and 19th century with the encouragement of the government. See https://interestingengineering.com/innovation/the-spies-who-launched-americas-industrial-revolution https://interestingengineering.com/innovation/the-spies-who-... Also during World War I the American government seized German chemical patents thereby launching the American chemical industry. So that is an example of theft by the state apparatus.
- belter 1y ago
  China is a nation of engineers...The US has been relying in on H-1B immigrants. Science is under attack. The truth is the US already lost: https://youtu.be/whVlI6H4d-4 https://youtu.be/whVlI6H4d-4
- archerx 1y ago
  I want China to release GPUs with a ton of VRAM, 128gb - 256gb. It doesn’t matter if they are half as fast as Nvidia because having a big model at a reasonable speed is better than not being to run them at all. AMD could have done this and have had a massive impact on nvidia’s market share but they choose not to because reasons.
- lesuorac 1y ago
  The US isn't slowing China anymore. China has an import ban on chips [1] so its irrelevant what the US does. [1]: https://www.cnbc.com/2025/09/17/nvidia-ceo-disappointed-after-reports-china-has-banned-its-ai-chips.html https://www.cnbc.com/2025/09/17/nvidia-ceo-disappointed-afte...
  - unethical_ban 1y ago
    Would they have done that if the US had been more "reliable" in providing the chips and didn't cut them off in the first place? The point still stands that the US instigated the split.
  - xadhominemx 1y ago
    The US is certainly slowing down China considerably. China would certainly not have an import ban on Blackwell GPUs if they were made available. And upstream, the ban on EUV and other high end semiconductor production equipment has severely limited china’s capacity to produce logic and DRAM (including HBM).
    - FooBarWidget 1y ago
      You severely underestimate what they can do with alternative tech paths. You don't have to chase nanometers for good AI system outcomes. Their current, and very viable, strategy is to build a ton of slower chips and to pump in a humongous amount of power. And to optimize the software stack, e.g. more efficient architectures. Unlike the west, they have a lot of cheap power (think solar panels in the desert) and excellent transmission. It also means they'll have to innovate on powder delivery and cooling systems to handle that sort of scale, but that's still easier than building EUV. Huawei has already done it with their phone from last year: they put in a more power-hungry chip, but they innovated hugely woth passive heat dissipation and bigger batteries, so the end product is still something consumers want. And with these Chinese AI models you're already seeing how they're reducing costs so they can run more on fewer chips.
      - xadhominemx 1y ago
- knowitnone3 1y ago
  It's much easier to copy what others are doing instead of spending the time and money for research and engineering. It's also much easier if you steal the tech. I could never have invented a bicycle but I can sure make a copy of one.
  - FpUser 1y ago
    "... instead of spending the time and money for research and engineering..." China has plenty of R&D and science now.
- sspiff 1y ago
  Their are signs that China is not open sourcing their SOTA models anymore. Both Huawei and Qwen (Qwen-Max, WAN 2.5) and have launched flagship models which are yet to be opensourced.
  - camel_Snake 1y ago
    Small counterpoint but there are also 2 new players putting out SOTA open source models (Moonshots Kimi and zhipus GLM) so we're still seeing the same number of models overall, just via newer entrants.
  - natrys 1y ago
    Qwen's max series had always been closed weight, it's not a policy change like you are alluding. What exactly is Huawei's flagship series anyway? Because their PanGu line is open-weight, but Huawei is as of yet not in the LLM making business, their models are only meant to signal that it's possible to do training and inference on their hardware, that's all. No one actually uses those models.
- deleted 1y ago
  [deleted]
- downrightmike 1y ago
  That's how it usually goes, fully expected
- coliveira 1y ago
  You mean, thank the US for their FAILED "civilizational" gate keeping.
- narrator 1y ago
  Peaceful competition is a good thing. It's better than a unified one world government throttling everybody.
majke 1y ago
better link https://www.tomshardware.com/tech-industry/semiconductors/alibaba-says-new-pooling-system-cut-nvidia-gpu-use-by-82-percent https://www.tomshardware.com/tech-industry/semiconductors/al... paper https://dl.acm.org/doi/10.1145/3731569.3764815 https://dl.acm.org/doi/10.1145/3731569.3764815
- dang 1y ago
  Ok, we've changed the URL above (from https://www.scmp.com/business/article/3329450/alibaba-cloud-claims-slash-nvidia-gpu-use-82-new-pooling-system https://www.scmp.com/business/article/3329450/alibaba-cloud-...), and will put the link to the paper in the top text. Thanks!
  - hd4 1y ago
    Feel like I've made it
braza 1y ago
Does someone know if there's some equivalent of those engineering/research blogs for Chinese companies? I used to follow the ones from Western companies, but honestly, after some point in time, I would like to see some cases from what I consider is a good benchmark for everyone that does not work in FAANG in terms of engineering.
- supriyo-biswas 1y ago
  The company blogs of Chinese companies will often do articles like this[1] talking about a new innovation or optimization that they did, but this will be often just mixed in with marketing articles too. I would also assume there's a lot of content in the native Chinese forums, which unfortunately, as an English-speaking person, I wouldn't be able to easily refer to :( [1] https://www.alibabacloud.com/blog/how-does-alibaba-ensure-the-performance-of-system-components-in-a-10000-node-kubernetes-cluster_595469 https://www.alibabacloud.com/blog/how-does-alibaba-ensure-th...
- wsfung2008 1y ago
  [dead]
djoldman 1y ago
Key paragraph: > However, a small handful of models such as Alibaba’s Qwen and DeepSeek are most popular for inference, with most other models only sporadically called upon. This leads to resource inefficiency, with 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found.
- make3 1y ago
  these other models are likely much smaller
throwaway48476 1y ago
Its easy enough for a a well resourced entity to take a pre trained model and deploy it on new hardware to save on the NVDA tax. It's far less likely for research and model training to happen outside the mature NVDA ecosystem.
muddi900 1y ago
[flagged]
- throawayonthe 1y ago
  scmp is kinda the opposite of state media lol
  - muddi900 1y ago
    It is owned buy AliBaba. Can anybody independently verify any of this?
- larus_canus 1y ago
  Ah, yes, the American media environment, which is internationally famous for not lying.
  - muddi900 1y ago
    So we are in agreement, we should not take every thing at face value.
    - larus_canus 1y ago
      I just hope you extend this skepticism consistently.
      - muddi900 1y ago
        I do. I don't know what gave people the impression that I was born an American and went to their schools. In "open societies" like the US, the form information control are rather sophisticated, but they are there. However, I grew up in a one TV channel country. I learned to never trust the news. This immunizes me from state propaganda a little bit. I think I was more referring to general abundance of credulity that is thrown at all AI stories at HN. There are a lot of skeptics when an Ed Zitron blog is shared. But every time a "ChatGPT was worried about being shut down" story is shared, it gets hundreds of upvotes. This story is just as ridiculous. AliBaba claims to reduce usage of equipment from a business that is being banned by their government. Magically. And during a trade war.
kilotaras 1y ago
Alibaba Cloud claims to reduce Nvidia GPU used for serving unpopular models by 82% (emphasis mine) > 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found Instead of 1192 GPUs they now use 213 for serving those requests.
- yorwba 1y ago
  Not really, Figure 1(a) of the paper says that the 17.7% are relative to a total of 30k GPUs (i.e. 5310 GPUs for handling those 1.35% of requests) and the reduction is measured in a smaller beta deployment with only 47 different models (vs. the 733 "cold" models overall.) Naïve extrapolation by model count suggests they would need 3321 GPUs to serve all cold models, a 37.5% reduction to before. (Or 6.6% reduction of the full 30k-GPU cluster.)
  - somerandomdude2 1y ago
    Really: "A paper presented at SOSP 2025 details how token-level scheduling helped one GPU serve multiple LLMs, reducing demand from 1,192 to 213 H20s." Which, if you scale it, matches the GPs statement.
    - yorwba 1y ago
      From the SCMP article you might get the impression that the various figures all refer to the same GPU cluster, but in the paper itself it's very clear that this is not the case, i.e. the 213 GPUs in the smaller cluster are not serving 1.35% of the requests in the larger cluster. Then if you want to scale it, you have a choice of different numbers you could scale, and each would get different results. Since they're constrained by the limited number of different models a single GPU can serve, I think scaling by the number of models is the most realistic option.
- MangoCoffee 1y ago
  In the past, software and computer engineers would tackle problems head-on, designing algorithms and finding creative solutions. thanks to the US restrictions on semiconductor industry (Chinese), Chinese engineers are being forced to innovate and find their own ways to overcome challenges like the old school engineers (What Silicon Valley used to be)
checker659 1y ago
They are working with tiny models. Not sure how well it'd scale to bigger models (if at all).
- CaptainOfCoit 1y ago
  They're all LLMs, so no, not tiny, but not exactly huge either: > Our current deployment runs in a cross-region cluster comprising 213 H20 GPUs, serving twenty-eight 1.8–7B models (TP=1) and nineteen 32–72B models (TP=4).
mighmi 1y ago
To what extent is this practice applicable to other loads?
- wsfung2008 1y ago
  This is for platforms that serve many different models, most of which have very low usage. e.g. huggingface, civitai
ibejoeb 1y ago
Sounds like this virtual GPU is a separate scheduler. I wonder what kind of latency is introduced by marshaling all that data around.
wslh 1y ago
How feasible is that in an horizon of 5 years new optimized "equations" will cut the need for more GPUs?
- aoeusnth1 1y ago
  Not feasible.
lnxg33k1 1y ago
Lots of shareholders here, move along, there is nothing to read
catigula 1y ago
Sounds like they stopped doing something stupid.
shoeb00m 1y ago
Would this make cloud providers running low volume fine-tuned models more economically viable?
jeffybefffy519 1y ago
I still think nVidia has the most to loose in the AI race, optimisations like this will continue coupled with better ASIC's.
OhioMan2943 1y ago
Is this another nail in the gpu/ai stock market bubble coffin?
nickysielicki 1y ago
> Distributed executor: Inference engines support model parallelism via distributed executors (e.g., Ray [32] and NCCL [9]), whose initialization takes tens of seconds. I mean, it really shouldn't take tens of seconds for those initialization(s) to occur. There's no good fundamental reason that it should take that long. It's just bloat.
ddelnano 1y ago
Does anyone know how their KV cache sync mechanism compares to newer P2P communication layers like nixl, uccl p2p, etc.? The authors mention that NCCL and Ray initialization were too slow (see quote below), but from the description it sounds like they’ve reimplemented a layer that’s increasingly being standardized by frameworks like nixl and uccl. > Distributed executor: Inference engines support model parallelism via distributed executors (e.g., Ray [32] and NCCL [9]), whose initialization takes tens of seconds.

523 points by hd4 1y ago

Paper: https://dl.acm.org/doi/10.1145/3731569.3764815 https://dl.acm.org/doi/10.1145/3731569.3764815

The US attempt to slow down China's technological development succeeds on the basis of preventing China from directly following the same path, but may backfire in the sense it forces innovation by China in a different direction. The overall outcome for us all may be increase efficiency as a result of this forced innovation, especially if Chinese companies continue to open source their advances, so we may in the end have reason to thank the US for their civilisational gate keeping

rzerowan 1y ago
Fingers crossed for convergence rather than divergence in the technical standards.Although the way hings are going it looks like the 2 stacks will diverge sooner rather than later , with the US+ banning the use of CHN models while simultaneosly banning the export of it quasi-open models. We may very well end up in a situation like the old PAL vs NTSC video standard where the PAL(EU/Asia/AFrica) and NTSC(America's/Japan) gradually converged with the adoption of digital formats. Instead here would be a divergence based on geopolitical considerations.
- hunglee2 1y ago
  positive take: a bifurcated tech tree might give us (humanity) a better chance of faster advancement, as it would be a persistent A/B test in live environment. Where I would join you in the crossing of fingers is to ensure such A/B testing is competitive but not destructive. We may even evolve to a situation of complementarity, an American Ying vs the Chinese Yang. Lets hope so!
reliabilityguy 1y ago
Tbh this whole situation reminds of how Japan excelled in making a lot more with a lot less after WW2, e.g., fuel-efficient engines, light cars, etc. these constraints were not present in the US (and to some extent in Europe), and resulted in US cars being completely not competitive in non-US markets.
- dataviz1000 1y ago
  I've been in Chile, Peru, Colombia, Panama, and Costa Rica. The streets are flooded with cheap Chinese cars and I see more BYD than American cars. If the car wasn't made in Japan or Korea which probably account for most of the cars, it was likely made in China. Moreover, I haven't been in countries with the closest ties to China.
  - sofixa 1y ago
    > The streets are flooded with cheap Chinese cars and I see more BYD than American cars This isn't surprising in any way, American "cars" (quotes because the vast majority of what American manufacturers pump out isn't cars, it's trucks) haven't been competitive in decades. The only globally competitive vehicles were developed in Europe by GM Europe (Opel, since sold to PSA now Stellantis) or Ford Europe (which axed all models bar the Puma). The rest is too big, expensive and inefficient from the vast majority of uses. Tariffs and good marketing keep American car manufacturers in business in the US, but those don't work in most other markets. The more appropriate comparison is with European automakers such as VW Group, Stellantis (Peugeot, Citroën, DS, Fiat, Chrysler, Dodge, Ram), Renault. And there too BYD is winning as well in mosy countries, but at least there's a comparison possible.
    - Workaccount2 1y ago
      The US couldn't compete anyway. The US is an advanced economy and will struggle to compete in non-advanced categories. It's like trying to level your MMORPG character to 100 by only farming in lvl 30-40 mob areas. It's really not worth it and mostly forced.
segmondy 1y ago
may backfire? it's a bit too late for that. go to 2024, western labs were crushing it. it's now 2025, and from china, we have deepseek, qwen, kimi, glm, ernie and many more capable models keeping up with western labs. there are actually now more chinese labs releasing sota models than western labs.
- hunglee2 1y ago
  too early to call a winner, though it is disappointing to see US withdrawal from open source. Still the main outcome of open source is distribution / diffusion of the idea, so it will inevitably mean US open source will come back, hopefully via some grass roots maniac, there will be a Linus-like character emerge at some point
  - segmondy 1y ago
    i'm not calling a winner, i'm just saying that the chinese have caught up despite the embargo. google, openai & anthrophic have phenomenal models. i stopped using openai & anthropic after they called for open weight/source regulation. i use google because they offer gemma and i got a year gemini-pro subscription for free, use openai gpt-oss-120b since i can run it at home, and the only model i currently pay for is a chinese model.
  - mixologist 1y ago
    user growth has slowed. the technology that should help users is only being pushed from the top, while users refuse to use it. openai pivoted to porn. does it really feel like they have a chance to recover all the expenses in the future? crypto grifters pivoted to ai and, same as last time, normal people don’t want to have anything to do with them. considering the amount of money burned on this garbage, i think we can at least declare a looser.
- Workaccount2 1y ago
  But they aren't keeping up They are lauded for the ability to cost ratio, or their ability to parameter ratio, but virtually everyone using LLMs for productive work are using ChatGPT/Gemini/Claude. They are kind of like Huffy bicycles. Good value, work well, but if you go to any serious event, no one will be riding one.
dlisboa 1y ago
History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years. In many senses there's hubris in the western* view of China accomplishments: most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the names of AI researchers there's a strong pattern even if some are currently plying their trade in the west. --- * I hate the term "western" because some "westeners" use it to separated what they think are "civilized" from "uncivilized", hence for them LATAM is not "western" even though everything about LATAM countries is western.
- achierius 1y ago
  > most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the names of AI researchers there's a strong pattern even if some are currently plying their trade in the west. While I don't disagree with your overall point, it's important to recognize that this is only a phenomenon of the last ~30 years, and to avoid falling into the trapn of Han racial chauvinism. E.g. there were ~no Chinese scientists in Germany in the 70s but they were heavily innovating nevertheless.
  - dlisboa 1y ago
    Absolutely. China obviously has a longer history with innovation but they like to make it seem everything was invented by them at some point in the past. I'd say newer technology is where China has had a bigger impact. Consequently newer tech is precisely where global cooperation is most required so no country can really do it by themselves. We could even say no country, western or otherwise, has been doing it on their own for the past 500 years or so but alas...
    - tsunamifury 1y ago
      It’s more helpful to think of China as an accelerant rather than an innovator in this position.
notepad0x90 1y ago
I think anti-immigrant rhetoric will have the most impact against the US. A lot of the people innovating on this stuff are being maligned and leaving in droves. Aside from geography, attracting talent from all over the world is the one edge the US has a nation over countries like China. But now the US is trying to be xenophobic like China, restrict tech import/export like China but compete against 10x population and lack of similar levels of internal strife and fissures. The world, even Europe is looking for a new country to take on a leader/superpower role. China isn't there yet, but it might get there in a few years after their next-gen fighter jets and catching up to ASML. But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india.
- dlisboa 1y ago
  > But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india. That's a strength. Them not having interest in global domination and regime change other than their backyard is what allows them to easily make partners in Africa and LATAM, the most important regions for raw materials.
  - notepad0x90 1y ago
    You would think so, but historically that's why they never became more than a regional power. Empires for millennia craved trade with China but only the mongols from that region made it all the way to western europe in their invasions. It is a strength, if their goal is to have a stable and prosperous country long term, and that seems to be what they want. good for them. But nature abhors a vacuum, so there will always be an empire at the top of the food chain. Such empires want to maximize wealth for their people and secure them against threats, that's why invasions and exploitation of weaker countries happens. That game hasn't changed. Friendly relations work, until you need a lot of resources from a country that doesn't want to give it up. Or, like with the US, when they're opening up military bases next to your borders and you need a buffer state. Or, when naval blockades and sanctions are being enforced against your country for not complying with extra-sovereign demands. History shows that countries content with what they have collapse or weaken very quickly. China will have a population crisis in a few decades for example, and it won't have the large manufacturing base and its people will be too used to luxuries to go back to slaving for western countries for pennies. Keep in mind that the current china itself is so great and prosperous because of all the invasions it did against western china and satellite states like Vietnam and north Korea (the US isn't special in this regard).
IT4MD 1y ago
I believe this is an Pollyanna take on AI. There is nothing about humans that tells us humans will bring AI to fruition for the other humans and a mountain of evidence showing how it will be used to abuse humans instead....for profits/power/whatever horse shit the masters of the universe have decided upon.
amelius 1y ago
Another outcome may be that we now have to learn Chinese to understand their datasheets ...
- anonzzzies 1y ago
  I was doing this in the 70-80s with electronics from Hong Kong and Japan. The nice cheap stuff ( I was very young ) was all sheets in things I basically had to pattern match against notes of others on BBS and meetups.
myth_drannon 1y ago
China's innovation relies on the stolen western IP, without it, China is nothing. Also USSR/Russia is no longer a scientific powerhouse that can supply China with some military innovation. A dictatorship combined with cheap labour it 100% guarantees that the country's innovation is stunted, no matter what the Chinese propaganda claims.
- Hikikomori 1y ago
  And the US has never stolen IP?
  - myth_drannon 1y ago
    Corporate espionage is ever present but it is criminalized. The only time US as a country did that you can say "stole IP" was after WII when it took Nazi rocket scientists and technology. China is the opposite; stealing tech is done by the state apparatus (same was done by USSR and reverse engineering computers for example). Frankly I'm not surprised that this is done, probably if US was so behind it would have done the same to reduce the gap. Everyone is trying to survive and outsmart and outwit the other, instead of collaborating.
    - currydove 1y ago
      Negative. https://interestingengineering.com/innovation/the-spies-who-launched-americas-industrial-revolution https://interestingengineering.com/innovation/the-spies-who-... https://yawboadu.substack.com/p/stealing-success-how-ip-theft-built https://yawboadu.substack.com/p/stealing-success-how-ip-thef... https://ipwatchdog.com/2017/07/05/americas-industrial-revolution-based-trade-secret-theft/ https://ipwatchdog.com/2017/07/05/americas-industrial-revolu...
    - g8oz 1y ago
      You are mistaken about American intellectual property theft. They engaged in extensive IP theft from Britain in the 18th and 19th century with the encouragement of the government. See https://interestingengineering.com/innovation/the-spies-who-launched-americas-industrial-revolution https://interestingengineering.com/innovation/the-spies-who-... Also during World War I the American government seized German chemical patents thereby launching the American chemical industry. So that is an example of theft by the state apparatus.
belter 1y ago
China is a nation of engineers...The US has been relying in on H-1B immigrants. Science is under attack. The truth is the US already lost: https://youtu.be/whVlI6H4d-4 https://youtu.be/whVlI6H4d-4
archerx 1y ago
I want China to release GPUs with a ton of VRAM, 128gb - 256gb. It doesn’t matter if they are half as fast as Nvidia because having a big model at a reasonable speed is better than not being to run them at all. AMD could have done this and have had a massive impact on nvidia’s market share but they choose not to because reasons.
lesuorac 1y ago
The US isn't slowing China anymore. China has an import ban on chips [1] so its irrelevant what the US does. [1]: https://www.cnbc.com/2025/09/17/nvidia-ceo-disappointed-after-reports-china-has-banned-its-ai-chips.html https://www.cnbc.com/2025/09/17/nvidia-ceo-disappointed-afte...
- unethical_ban 1y ago
  Would they have done that if the US had been more "reliable" in providing the chips and didn't cut them off in the first place? The point still stands that the US instigated the split.
- xadhominemx 1y ago
  The US is certainly slowing down China considerably. China would certainly not have an import ban on Blackwell GPUs if they were made available. And upstream, the ban on EUV and other high end semiconductor production equipment has severely limited china’s capacity to produce logic and DRAM (including HBM).
  - FooBarWidget 1y ago
    You severely underestimate what they can do with alternative tech paths. You don't have to chase nanometers for good AI system outcomes. Their current, and very viable, strategy is to build a ton of slower chips and to pump in a humongous amount of power. And to optimize the software stack, e.g. more efficient architectures. Unlike the west, they have a lot of cheap power (think solar panels in the desert) and excellent transmission. It also means they'll have to innovate on powder delivery and cooling systems to handle that sort of scale, but that's still easier than building EUV. Huawei has already done it with their phone from last year: they put in a more power-hungry chip, but they innovated hugely woth passive heat dissipation and bigger batteries, so the end product is still something consumers want. And with these Chinese AI models you're already seeing how they're reducing costs so they can run more on fewer chips.
    - xadhominemx 1y ago
knowitnone3 1y ago
It's much easier to copy what others are doing instead of spending the time and money for research and engineering. It's also much easier if you steal the tech. I could never have invented a bicycle but I can sure make a copy of one.
- FpUser 1y ago
  "... instead of spending the time and money for research and engineering..." China has plenty of R&D and science now.
sspiff 1y ago
Their are signs that China is not open sourcing their SOTA models anymore. Both Huawei and Qwen (Qwen-Max, WAN 2.5) and have launched flagship models which are yet to be opensourced.
- camel_Snake 1y ago
  Small counterpoint but there are also 2 new players putting out SOTA open source models (Moonshots Kimi and zhipus GLM) so we're still seeing the same number of models overall, just via newer entrants.
- natrys 1y ago
  Qwen's max series had always been closed weight, it's not a policy change like you are alluding. What exactly is Huawei's flagship series anyway? Because their PanGu line is open-weight, but Huawei is as of yet not in the LLM making business, their models are only meant to signal that it's possible to do training and inference on their hardware, that's all. No one actually uses those models.
deleted 1y ago
[deleted]
downrightmike 1y ago
That's how it usually goes, fully expected
coliveira 1y ago
You mean, thank the US for their FAILED "civilizational" gate keeping.
narrator 1y ago
Peaceful competition is a good thing. It's better than a unified one world government throttling everybody.

majke 1y ago

better link https://www.tomshardware.com/tech-industry/semiconductors/alibaba-says-new-pooling-system-cut-nvidia-gpu-use-by-82-percent https://www.tomshardware.com/tech-industry/semiconductors/al... paper https://dl.acm.org/doi/10.1145/3731569.3764815 https://dl.acm.org/doi/10.1145/3731569.3764815

dang 1y ago
Ok, we've changed the URL above (from https://www.scmp.com/business/article/3329450/alibaba-cloud-claims-slash-nvidia-gpu-use-82-new-pooling-system https://www.scmp.com/business/article/3329450/alibaba-cloud-...), and will put the link to the paper in the top text. Thanks!
- hd4 1y ago
  Feel like I've made it

braza 1y ago

Does someone know if there's some equivalent of those engineering/research blogs for Chinese companies? I used to follow the ones from Western companies, but honestly, after some point in time, I would like to see some cases from what I consider is a good benchmark for everyone that does not work in FAANG in terms of engineering.

supriyo-biswas 1y ago
The company blogs of Chinese companies will often do articles like this[1] talking about a new innovation or optimization that they did, but this will be often just mixed in with marketing articles too. I would also assume there's a lot of content in the native Chinese forums, which unfortunately, as an English-speaking person, I wouldn't be able to easily refer to :( [1] https://www.alibabacloud.com/blog/how-does-alibaba-ensure-the-performance-of-system-components-in-a-10000-node-kubernetes-cluster_595469 https://www.alibabacloud.com/blog/how-does-alibaba-ensure-th...
wsfung2008 1y ago
[dead]

djoldman 1y ago

Key paragraph: > However, a small handful of models such as Alibaba’s Qwen and DeepSeek are most popular for inference, with most other models only sporadically called upon. This leads to resource inefficiency, with 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found.

make3 1y ago
these other models are likely much smaller

throwaway48476 1y ago

Its easy enough for a a well resourced entity to take a pre trained model and deploy it on new hardware to save on the NVDA tax. It's far less likely for research and model training to happen outside the mature NVDA ecosystem.

muddi900 1y ago

[flagged]

throawayonthe 1y ago
scmp is kinda the opposite of state media lol
- muddi900 1y ago
  It is owned buy AliBaba. Can anybody independently verify any of this?
larus_canus 1y ago
Ah, yes, the American media environment, which is internationally famous for not lying.
- muddi900 1y ago
  So we are in agreement, we should not take every thing at face value.
  - larus_canus 1y ago
    I just hope you extend this skepticism consistently.
    - muddi900 1y ago
      I do. I don't know what gave people the impression that I was born an American and went to their schools. In "open societies" like the US, the form information control are rather sophisticated, but they are there. However, I grew up in a one TV channel country. I learned to never trust the news. This immunizes me from state propaganda a little bit. I think I was more referring to general abundance of credulity that is thrown at all AI stories at HN. There are a lot of skeptics when an Ed Zitron blog is shared. But every time a "ChatGPT was worried about being shut down" story is shared, it gets hundreds of upvotes. This story is just as ridiculous. AliBaba claims to reduce usage of equipment from a business that is being banned by their government. Magically. And during a trade war.

kilotaras 1y ago

Alibaba Cloud claims to reduce Nvidia GPU used for serving unpopular models by 82% (emphasis mine) > 17.7 per cent of GPUs allocated to serve only 1.35 per cent of requests in Alibaba Cloud’s marketplace, the researchers found Instead of 1192 GPUs they now use 213 for serving those requests.

yorwba 1y ago
Not really, Figure 1(a) of the paper says that the 17.7% are relative to a total of 30k GPUs (i.e. 5310 GPUs for handling those 1.35% of requests) and the reduction is measured in a smaller beta deployment with only 47 different models (vs. the 733 "cold" models overall.) Naïve extrapolation by model count suggests they would need 3321 GPUs to serve all cold models, a 37.5% reduction to before. (Or 6.6% reduction of the full 30k-GPU cluster.)
- somerandomdude2 1y ago
  Really: "A paper presented at SOSP 2025 details how token-level scheduling helped one GPU serve multiple LLMs, reducing demand from 1,192 to 213 H20s." Which, if you scale it, matches the GPs statement.
  - yorwba 1y ago
    From the SCMP article you might get the impression that the various figures all refer to the same GPU cluster, but in the paper itself it's very clear that this is not the case, i.e. the 213 GPUs in the smaller cluster are not serving 1.35% of the requests in the larger cluster. Then if you want to scale it, you have a choice of different numbers you could scale, and each would get different results. Since they're constrained by the limited number of different models a single GPU can serve, I think scaling by the number of models is the most realistic option.
MangoCoffee 1y ago
In the past, software and computer engineers would tackle problems head-on, designing algorithms and finding creative solutions. thanks to the US restrictions on semiconductor industry (Chinese), Chinese engineers are being forced to innovate and find their own ways to overcome challenges like the old school engineers (What Silicon Valley used to be)

checker659 1y ago

They are working with tiny models. Not sure how well it'd scale to bigger models (if at all).

CaptainOfCoit 1y ago
They're all LLMs, so no, not tiny, but not exactly huge either: > Our current deployment runs in a cross-region cluster comprising 213 H20 GPUs, serving twenty-eight 1.8–7B models (TP=1) and nineteen 32–72B models (TP=4).

mighmi 1y ago

To what extent is this practice applicable to other loads?

wsfung2008 1y ago
This is for platforms that serve many different models, most of which have very low usage. e.g. huggingface, civitai

ibejoeb 1y ago

Sounds like this virtual GPU is a separate scheduler. I wonder what kind of latency is introduced by marshaling all that data around.

wslh 1y ago

How feasible is that in an horizon of 5 years new optimized "equations" will cut the need for more GPUs?

aoeusnth1 1y ago
Not feasible.

lnxg33k1 1y ago

Lots of shareholders here, move along, there is nothing to read

catigula 1y ago

Sounds like they stopped doing something stupid.

shoeb00m 1y ago

Would this make cloud providers running low volume fine-tuned models more economically viable?

jeffybefffy519 1y ago

I still think nVidia has the most to loose in the AI race, optimisations like this will continue coupled with better ASIC's.

OhioMan2943 1y ago

Is this another nail in the gpu/ai stock market bubble coffin?

nickysielicki 1y ago

> Distributed executor: Inference engines support model parallelism via distributed executors (e.g., Ray [32] and NCCL [9]), whose initialization takes tens of seconds. I mean, it really shouldn't take tens of seconds for those initialization(s) to occur. There's no good fundamental reason that it should take that long. It's just bloat.

ddelnano 1y ago

Does anyone know how their KV cache sync mechanism compares to newer P2P communication layers like nixl, uccl p2p, etc.? The authors mention that NCCL and Ray initialization were too slow (see quote below), but from the description it sounds like they’ve reimplemented a layer that’s increasingly being standardized by frameworks like nixl and uccl. > Distributed executor: Inference engines support model parallelism via distributed executors (e.g., Ray [32] and NCCL [9]), whose initialization takes tens of seconds.