4 ms·
Interesting if true - that Moonshot can train a ~3T SOTA model on only 20K NVIDIA GPUs, while others like Musk (who freely admits to distilling OpenAI's models)
by HarHarVeryFunny 3mo ago
Interesting if true - that Moonshot can train a ~3T SOTA model on only 20K NVIDIA GPUs, while others like Musk (who freely admits to distilling OpenAI's models) struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs).
I guess at least partly a reflection of all the optimizations in the Kimi 3 architecture.
In the recent leaked DeepSeek investor meeting, they also mentioned only having a 20K GPU cluster (unclear if NVIDIA, or Huawei).
- cubefox 3mo ago> Musk (who freely admits to distilling OpenAI's models) Source?
- HarHarVeryFunny 3mo agohttps://www.forbes.com/sites/antoniopequenoiv/2026/04/30/elon-musk-admits-xai-distilled-openai-data-to-train-models-heres-what-that-means/ https://www.forbes.com/sites/antoniopequenoiv/2026/04/30/elo...
- gyanchawdhary 3mo agoSounds fair, since Sam can run a for profit NGO
- corranh 3mo agoI’m still baffled that OpenAI can complain about this with a straight face while these models are literally trained on everything regardless of copyright or permission.
- deleted 3mo ago[deleted]
- nomel 3mo agoRaw stolen data that is in no way related to AI vs a very very expensive transformative compilation of that raw stolen data that results in usable AI. The frustration is completely understandable to me, since distillation skips much of that "very very expensive" part. And yes, I understand the stolen data was expensive to make, so I understand the owners of it are also frustrated, but that's partly a problem with current law. Would the authors of the world be rich if OpenAI bought a single copy of their book to legally scan? For best sellers, that's somewhere around pennies, so no. Should the authors get a share in OpenAI? Current laws says, unambiguously, "no". Frustration all around is reasonable.
- HarHarVeryFunny 3mo agoNot really, in this context the word "distillation" is really being abused, or at least used in a different sense than when it was originally introduced in the Hinton et al "Distilling the Knowledge in a Neural Network" paper, where it was essentially referring to knowledge compression. The way Anthropic are using "distillation" is just in a very broad vague sense to claim that some data generated by their model was used to help train another one. They are not talking about something like internal logits, expensive to derive, that would be useful to train a smaller model, but rather about any output from their model, even outputs with redacted reasoning (i.e. incomplete outputs that do NOT reflect the underlying knowledge of the source model). Given the way Anthropic are using the word, IMO it's better just to think of this as cheap training data, and indeed very similar to the way Anthropic themselves got cheap training data just by taking it (even in cases where that was illegal - copyright). The alternative for Moonshot would be to pay for human generated reasoning data, just as the alternative for Anthropic would have been to pay human developers for coding data etc, not just take it from wherever they could lay their hands on it. So, I guess Moonshot may have violated Anthropic's TOS, in using Anthropic output to compete against Anthropic, but unlike Anthropic they at least didn't break copyright law since model output is not copyright, and technically they may not have even violated Anthropic's Terms of Service unless they owned the accounts used to access Anthropic's models (perhaps not - they may have used one of the anonymizing Chinese token resellers). So yeah - pot calls kettle black.
- 3mo ago
- infecto 3mo agoGenuine question. Is there a time factor part of that equation?
- HarHarVeryFunny 3mo agoIt's obviously a question - did they just train for 10x as long due to having 10x fewer GPUs, but then that spoils the claim that they distilled Fable which was only recently introduced. Now doubt they did use some training data generated from older US models though.
- upbeat_general 3mo agoThis is just not how it works - it is perfectly plausible (in fact the most likely) that pretraining was well finished by the time Fable was released. A bit of extra distillation takes far less compute. Moreover, it’s very plausible (and expected) to use multiple clusters and GPU types for RL rollouts which could very well not be included in this count. No part of this pipeline is fixed in stone.
- infecto 3mo agoRelatedly I also wonder how much the orchestrated distillation bot nets that Anthropic uses against China are real. Not saying it does not happen but I have never seen dating showing how pervasive it is and I always imagine other western labs would absolutely be doing the same.
- theblazehen 3mo agoThere's significant amounts of traffic going through transfer stations [1], which is then re-sold to the model labs. For a while the DeepSeek API was serving Fable as the DS 4 Pro model even. [2] [1] https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens-in https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens... [2] https://xcancel.com/synthwavedd/status/2078514339552628880 https://xcancel.com/synthwavedd/status/2078514339552628880
- 3mo ago
- ux266478 3mo agoIt's not that surprising to me. Most of the innovation in Chinese models has been in efficiency gains and optimizations. K3 coming from the factory in MXFP4 weights is a pretty relevant factor. Big performance gap probably also due to Moonshot doing QAT. Throw in the fact that Musk has easier access to compute, and I think you have your answer on the disparity.
- nextaccountic 3mo agoBut perf improvements not only mean you can run the same thing on cheaper hardware, you can also run more capable models in the same hardware In this sense, any advance in intelligence is a performance improvement and vice versa
- qeternity 3mo agoI don't follow this. Clearly all of the frontier labs are doing these things. When OAI released gpt-oss it was released as an mxfp4 checkpoint. OAI, Ant, et al are also obviously employing QAT.
- SubiculumCode 3mo agoOne does wonder whether there has been an expiration of the actual weights of Opus at one point.
- HawtAds 3mo ago> Musk (who freely admits to distilling OpenAI's models) struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs) Are you sure they are using all of their compute on training? Didn't they rent out a ton to other AI companies?
- foolswisdom 3mo agoThe renting out is a pretty recent development.
- dzonga 3mo agoeverything about the current 'a.i' models - east vs west points the wastefulness of western labs - whether in terms of gpu clusters being ran, cost of inference. the only way is down for the massive valuations and 'a.i' revenue projections.
- nekusar 3mo agoThe wastefulness is akin to having multiple private jets on standby, or workers coming in to a fishbowl office to be gawked at, or massive datacenters to be used as social "proof". Its all a fucking capitalistic farce to display to other rich elite that "Look at how much clout I have! I can make these peons dance around and do my bidding! Im a slave-owner!" https://infosec.exchange/@david_chisnall/116991627711001827 https://infosec.exchange/@david_chisnall/116991627711001827 He noted that that Silicon Valley doesnt really want to SOLVE problems. They want to find already-solved problems with problem matching. And of course, we just throw more people and more compute instead of optimization and understanding. The Chinese are being actively constrained with bullshit politics around a second Red Scare moment. And, well, they're winning. A lot.
- otabdeveloper4 3mo agoNo, it's an attempt to form an artificial moat by buying all the world's GPUs and memory.
- nekusar 3mo agoTrue. But check out Ebay right now. Networking gear is CHEAP. I just bought 2 switches, 48 port 10GbE with 4x QSFP+ at 40Gb fiber. $110 each. You can even get 24 port QSFP+ @100Gb networking devices for $350. Yeah while ram and gfx is $$$$$, networking is rock bottom prices..
- subscribed 3mo agoTry getting something new from Arista, Juniper or Fortigate. Lead times are sad. (new in terms of the most recent models, not just "something with X number of ports")
- yorwba 3mo agoFrom the article: > A person familiar with Moonshot’s procurement strategy confirmed that the company does indeed have a channel for accessing Blackwell processors via Southeast Asia. They didn’t specify whether this was a rental channel — which is legal, in most cases — or direct purchases, which are a breach of US regulations. The Information reported this week that Moonshot is seeking additional Blackwell processors to train its next model. > The 20,000 chips Moonshot accesses via Alibaba, meanwhile, are from Nvidia’s earlier generation of Hopper products, the people familiar with the agreement said. So the 20k GPUs from Alibaba is only a lower bound on how many you need to train a model like Kimi K3. The advantage of having more GPUs in any case is not so much that you can train bigger models, but that the turnaround time is faster, so you can run more experiments to dial in training choices. It's entirely possible that Musk has more than enough compute, but can't hire the talent to run all those experiments. (That would also explain why he has excess capacity he can rent to Google.)
- gpugreg 3mo ago> they also mentioned only having a 20K GPU cluster (unclear if NVIDIA, or Huawei). A few quotes from the transcript: > Our current computing capacity is approximately 20,000 H-equivalent units, most of which have just arrived within the past month or two > Regarding the Huawei 950, Huawei currently provides us with 16,000 SIM cards > A Huawei 950 [cluster] with 16,000 cards is equivalent to only a B-series card [cluster] with 4,000 cards.
- andy_ppp 3mo agoDo we believe them? It seems there’s no reason with the resources OpenAI and Anthropic have they wouldn’t be incentivised to build equally efficient pipelines? I think it’s mostly the fine tuning with distillation that gives Chinese labs their big advantage. However I’m pretty sure all the main labs are poisoning the output now when they detect Chinese activity so while K3 looks frontier on tests when you use it it’s miles off Sol and Fable.
- UqWBcuFx6NV4r 3mo ago“The only reason that China is good is because it steals from us” spoken from atop the unambiguously crumbling has-been empire. I’m not American nor Chinese, and I’m from a much more US-aligned country. But, Christ, you lot really are asking for it.
- andy_ppp 3mo agoI don’t think you can steal someone’s work if it’s based entirely off theft tbh, I see distillation as smart - are you denying it’s a huge part of why Chinese models are competitive or do you believe the narrative yourself that they are doing this with several orders of magnitude less compute and the US labs are profligate and fond of burning money rather than optimising? The truth is probably Anthropic/OpenAI/Google are pretty efficient but less efficient than the Chinese labs, the Chinese labs probably have more compute than they say to undermine US spending and distillation is quite efficient at bridging the gap in compute.
- HarHarVeryFunny 3mo agoThe way you make an LLM smarter is by training it on more data, which means you need to make it bigger, which means it costs more to train. High quality data is expensive. Synthetic data will get you so far, but after that you need to start paying experts to create data for you, which has been going on for a long time. The latest thing is paying for human written LLM-as-judge AI-output evaluation "rubrics", trying to extend RLVR into areas where "looks like it checks the boxes" is the best you can do. When anyone, Chinese or not (Elon Musk cheerfully admits to distilling OpenAI models) uses the output of someone else's model to train their own, then what they are primarily getting is cheap training data, but you still need to train your model on this data! You may have reduced the cost/speed of training data acquisition, but if you are training a 3T param model (Kimi 3) then you still need the compute to do that - that did not change. There was an interesting mention of the cost of training data in the recently leaked DeepSeek investor meeting, where their CEO referred to the cost of human-generated training data in China (i.e. using Chinese labor) as being the same as that in the US, which seems surprising. He also mentioned the time such data takes to be created. No doubt the Chinese will catch up in this area - this is just time and money, not Dutch technology (ASML) that the US is blocking them from buying.
- dang 3mo agoFrom a superficial look I think the submitted title ("Moonshot built on 20k Nvidia chip cluster from Alibaba") might have been misleading. Article says "That hardware forms a substantial chunk of the overall computing capacity Moonshot uses for its Kimi models", implying that other hardware was also used. (We've changed the title to what the article says now.)
- vrganj 3mo ago> Struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs). This feels like a very American way of designing things - just throw more horse power at it, bigger is better! The rest of the world is usually a bit more resource constrained and efficient at using those resources. See also Mustangs vs German sports cars, giant American fridges, giant American suburban McMansions vs livable cities etc etc.
- HarHarVeryFunny 3mo agoYou've just got fridge jealousy! ;-)
- locknitpicker 3mo ago> This feels like a very American way of designing things - just throw more horse power at it, bigger is better! I don't think so. Bruteforcing problems is a well established strategy in any field that involves computation of any form. Once you get something working, you can get results right now if you throw resources at it. In the meantime, any improvement in efficiency can easily be back ported to the same computational resources you're using.
- conover 3mo agoThis is the bitter lesson: https://en.wikipedia.org/wiki/Bitter_lesson https://en.wikipedia.org/wiki/Bitter_lesson