9 ms·
DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American la
by kamranjon 4mo ago
DeepSeek continues to not only push the boundaries but also publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Chinese labs are doing the most interesting work in AI right now.
- herodoturtle 4mo agoPublishing by necessity I wonder? American labs on the cutting edge pioneering the way forward, so Deepseek open sourcing what they’ve got is to help even the playing field. Hopefully the experts here can offer insight. The above is just my hunch and I’m not a specialist in this field.
- jonplackett 4mo agoWouldn’t that just help the American labs anyway though? Or do they assume they’ve actually already figured this stuff out and kept it secret?
- 7speter 4mo agoFrom what I gather, the Chinese are behind, but a lot of their research amounts to scrappy, clever discoveries in how to use more novel technologies (for Qwen and Deepseek, its mixture of expert models, that can do inference using a portion of the model at a time). The chinese also distill information from American models, so there’s that. The American companies, from my impression don’t involve themselves with such lowly “hacks” because they have so much money to just push forward with doing everything on big heavy models that run on the most cutting edge nvidia chips that they can, the moment, kinda sorta get on demand (I say that in some degree of jest).
- idiotsecant 4mo agoThe American companies would love to develop these 'hacks' because it would make them more money, something they are in existential need of right now. They don't develop them because they don't collaborate publicly anymore. Where would the whole industry be if Google never allowed publishing the transformers paper? It's not a coincidence that the American AI industry grew fastest in capability when it was the most open.
- 7speter 4mo agoJust a crazy catch 22, it seems
- tiahura 4mo agoWhy would they collaborate? Why not defect and just keep theirs private and implement the open ones?
- mistercheph 4mo agothis is not an effective long term strategy in a collaborative environment that is advancing for the same reason that having a private secret fork of the linux kernel with a few proprietary improvements is not an effective strategy. integrating your own work with the latest public advances takes resources. For one or two small changes this is manageable, but the further you diverge from the public, the cost of maintenance rises exponentially if you want to continue to integrate public advances. when you publish your meaningful advance, you offload the maintenance burden onto everyone else (and they only have to pay a linear cost rather than an exponential one) as it's integrated by default in new work. In most cases, the (exponential) maintenance cost of integrating public advances with secret ones exceeds the value of the public advances, so most that undertake this strategy of advancing the open frontier in secret don't attempt to integrate continually, but instead try to make a breakaway sprint in isolation to grab a few sticky customers before the unstoppable wave of the public frontier catches up. This is a pattern commonly seen in university research departments when researchers switch into product development mode, most of these projects are a sprint to advance away from the public frontier once a good idea is found and they do good work and find a few customers for a little while. But if you check back in a few years you won't find an advanced research department but a zombie IP company that brings in a steady income via IP enforcement and a small number of customers for whom switching is too expensive.
- parineum 4mo ago> They don't develop them because they don't collaborate publicly anymore. How do you know they aren't doing this stuff? Something has to account for them leading the industry.
- vintermann 4mo agoIt used to be the case that NSA hired the majority of all math graduates in the US, and were assumed to be years ahead in cryptography. Yet in the 90s, it became clear that they no longer were that - among other things, the cipher of the notorious Clipper chip was broken, and we can rule out that it was made weak on purpose because the whole point of Clipper was that they had a backdoor. So, despite hiring the cream of the crop of math graduates, who could read the papers of free academia, but whose own result the free world could not access - they fell behind. I have a theory explaining why. I think it's because science is an interactive process. NSA cryptographers could read papers, but they couldn't talk openly with the authors of those papers, because of secrecy demands - even asking question might indicate what they were working on. You can easily imagine them spending months on something they could have avoided by going to the original authors and getting told "Oh, we tried that for a long time, it doesn't work". Whether that theory is right or not, cryptography is a concrete example of a domain where public research with fewer resources beat private research with a lot more resources.
- idiotsecant 4mo agoEveryone in this thread is getting distracted by nationalism, but you hit the nail on the head. In this case for whatever reason the Chinese AI industry is collaborative and the American AI industry is not. This will result in the Chinese companies making progress faster. Full stop. This isn't a judgement on the merits of either system, only an observation of likely results.
- tiahura 4mo agoHasn't that been the mantra of open source for 40 years. Armies of companies, trillions of valuation, or even just Wayland, suggest that isn't always the case.
- eikenberry 4mo agoSo free software can only be considered a successful strategy if every single project succeeds?
- 4mo ago
- _0ffh 4mo agoI'm afraid I'm even balking at the word "pioneering" in context with US frontier labs. They are probably doing a few new things, right, but they are not blazing any trails for others to follow along, the Chinese are.
- d0gsg0w00f 4mo agoOr if the US labs are innovating, they're not talking about specifics.
- _0ffh 4mo agoOh, I assume they're innovating - it's what I meant with "doing new things". But the word pioneer comes from French pionnier, literally “foot soldier”, a soldier who goes ahead to prepare the way. If you don't publish you may be advancing, but you're not preparing anyone's way.
- epolanski 4mo agoChinese papers and techniques have been very influential and copied by US labs. Multi-head Latent Attention (MLA), Multi-Token prediction, MoE architecture are some of the most famous examples.
- HarHarVeryFunny 4mo agoMoE is from Google (Noam Shazeer) MTP is from Meta Another DeepSeek advance that the west are copying is DeepSeek Sparse Attention (DSA)
- xgk 4mo agoMixture-of-Expert (MoE) was introduced in the 1990s [1, 2], see also [3, 4]. The idea was that MoE scales up model capacity and only introduces small computation overhead. MoEs did not become viable for high-performance applications until sparse routing was integrated with modern deep networks, made possible by large-scale distributed computation. The breakthrough came with the development of sparsely gated networks [5], which showed that it is possible to maintain model accuracy while activating only a small fraction of a large parameter network during both training and inference. [1] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts. (1991) [2] M. I. Jordan, R. A. Jacobs, Hierarchical mixtures of experts and the EM algorithm. (1993) [3] L. Xu, M. Jordan, G. E. Hinton, An alternative model for mixtures of experts. (1994) [4] S. Waterhouse, D. MacKay, A. Robinson, Bayesian methods for mixtures of experts. (1995) [5] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. (2017)
- HarHarVeryFunny 4mo agoYes - I meant as applied to LLMs/Transformers.
- try-working 4mo agoYes, challenger Labs publish out of necessity. It is a marketing strategy. People assuming open source means giving something up, but the reality is that Z.ai has a revenue of some $100M and it would be about $0M if they never open sourced their models.
- skeledrew 4mo ago> Publishing by necessity It's more a cultural thing. Sharing progress is just in their blood.
- idiotsecant 4mo agoThis is overly simplistic to the point of glazing. Plenty of Chinese companies maintain industrial secrets to gain an advantage.
- skeledrew 4mo agoYes there are Chinese companies which maintain industrial secrets. Doesn't change the general cultural tendency that they prefer to share.
- tomalaci 4mo agoProbably because American AI companies are on the hook for quite a lot of investment money. I think they are trying to find the magical moat to justify their valuation. Revealing optimizations similar to these would pretty much reduce their competitive position.
- lwansbrough 4mo agoChinese labs are also still behind, so they’re incentivized to collaborate and have no reason to do it in private. I suspect their tune will change if they ever take the lead..
- arj 4mo agoNot everyone is motivated by greed
- tirant 4mo agoWhat do you think is the underlaying motivation?
- arj 4mo agoYou ask me what I think. So far deepseek has been very consistently trying to advance state of the art research in a transplant and public way by writing papers and publishing working code. They are also not at the mercy of the stock market in the same way many Americans companies are. Before anyone assumes too much, I live in Europe.
- lwansbrough 4mo agoNot a greed thing. It’s a national security thing.
- colordrops 4mo agoSo the marketplace is working.
- rvz 4mo agoExactly. They did not have to open up their research up and this is what happens when smart researchers are forced to squeeze performance gains out of existing hardware. They don't have TPUs or access to the latest Vera Rubin GPUs either to get performance gains for free. All of the optimizations Deepseek have done are in software and it goes down to the PTX assembly level. Compared to Anthropic who are celebrating in fixing a flickering issue in a terminal app which took months to fix.
- saagarjha 4mo agoAll frontier labs are working down to the PTX level (and lower)
- vidarh 4mo ago> Compared to Anthropic who are celebrating in fixing a flickering issue in a terminal app which took months to fix. It's funny, because if you ran Claude Code on a slow terminal, the cause of the flicker was obvious: They kept dumping the entire history of the chat back into the terminal in a number of situations, and relied on the terminal to them end up in the correct state.
- yorwba 4mo agoAnthropic almost certainly also has optimized software down to the assembly level, considering this take-home interview challenge they published: https://github.com/anthropics/original_performance_takehome/blob/main/perf_takehome.py https://github.com/anthropics/original_performance_takehome/... which is all about instruction-level performance optimizations. That they don't prioritize UI fixes just means they consider other things more important.
- epolanski 4mo agoR1 was very influential on US models development.
- deleted 4mo ago[deleted]
- jmyeet 4mo agoChinese companies (and labs) operate in conjunction with the CCP so whatever they're doing, it's because it's Chinese state policy. What became clear when DeepSeek came onto the scene was that China was seeking to commoditize LLMs. They consider it an issue of national security not to be beholden to US tech companies when it comes to AI. And I, for one, fully endorse this policy. Another data point on this is the black market for Claude tokens in China [1]. The chat logs themselves are a commodity to train models. I believe that OpenAI in particular is a bet on a trillion dollar pot of gold that doesn't exist. Google, Microsoft, Amazon and Meta will all be fine. Anthropic is in a far better position than OpenAI (IMHO) but if DeepSeek or some other Chinese open weight model gets as good at coding, they're in real trouble too. [1]: https://news.ycombinator.com/item?id=48667495 https://news.ycombinator.com/item?id=48667495
- tw1984 4mo ago> Another data point on this is the black market for Claude tokens in China [1]. The chat logs themselves are a commodity to train models. anyone with IQ higher than 130 (thus qualified for actual AI R&D) would be questioning something obvious here - if they are already doing such dodgy stuff with the aim to maximize profits, why would those resellers have large amount of logs with actual American model responses to sell to those AI labs in the first place. shouldn't they just post train & customize some leading Chinese open source models to pretend to be Opus or GPT for the vast majority of their users (as classified by some models) who don't know much about expected Opus behaviours & not skilled enough to tell the differences? that is actually the interesting bit not covered in your censored version of the story line, it is also what happens on the ground. your censored version of the story implies that those dodgy resellers using stolen credit cards, pooling accounts with stolen IDs and illegally selling very personal logs would somehow be honest enough to spend extra $ to ensure their victims (aka paying users) can actually use real Opus and GPT. LOL dude, you failed this IQ test miserably.
- deleted 4mo ago[deleted]
- jampekka 4mo ago
- DivingForGold 4mo agoSure, in part by "stealing" from American AI companies with Distillation attacks: https://yipzap.com/anthropic-accuses-alibaba-of-largest-ai-distillation-attack-28-8m-fraudulent-exchanges/ https://yipzap.com/anthropic-accuses-alibaba-of-largest-ai-d...
- pennomi 4mo agoIf your moat is “please don’t copy my outputs”, you don’t have a moat. There is no such thing as a distillation “attack”.
- steinvakt2 4mo agoHow does it differ from pirating music or movies?
- pornel 4mo agoMachine-extruded text is not copyrightable, since there was no human creativity involved in producing it. (and if you argue the US models do produce copyrighted works, then oooops - whose copyright is it huh?)
- steinvakt2 4mo agoI see. Fair point
- bethekidyouwant 4mo agoOw my head.
- deleted 3mo ago[deleted]
- ReptileMan 4mo agoThat when I pay for a model, the copyright of the output belongs to me. This is as work for hire as it gets.
- dakolli 4mo agoIts because our culture worships pieces of paper the government tells us is worth something.
- IAmGraydon 4mo agoMoney is just a physical representation of the ability to get what you want. The problem is not money. It’s the fact that we live in a “me” society.
- mordae 4mo agoNope, people seek it out because government tells them to pay taxes _or else_.
- darkoob12 4mo agoGoogle and Microsoft publish more than enough and American universities are publishing the science beyond DeepSeek's engineering. That fact that you don't know about them means you're not following the science only reading hacker news.
- kamranjon 4mo agoGoogle hasn’t published much in depth ML work since T5 (which was hugely influential at the time) - most Gemma releases are 1-3 page model card pdfs these days with no in depth analysis. Even TurboQuant is shaking out to have basically been a rehash of previous work without proper attribution. I do think Microsoft is doing some interesting things with smaller models but haven’t read much research, interested in any refs you might have to share!
- darkoob12 4mo agoCheck recent iclr acl icml neurips you will see 10-20 papers from Google Research which are not just simple model cards. they are solid reproducible research.
- utopiah 4mo agoIt's almost as if ... they were what OpenAI was when it started. Sad to see but glad someone is doing is.
- OtomotO 4mo agoThe difference between greed and power
- garn810 4mo agoYep. It's about time western world realized Chinese are not the "very bad guys under dictatorship"
- 3abiton 4mo agoHonestly it's just a hierarchy difference between the two countries. In the US, tech/fin/military companies have the upper hand compared to the government (fragmented between 2 parties). Despite the sharades with Anthropic, Tech-fluencers are in control. Compared to china, the government (dictatorship) has more control over Tech companies (take any example from the past 10 years). For them, undermining the US AI supremacy is an objective, and releasing open weight models is the way, and I'm all for it.
- idiotsecant 4mo agoLet's not get crazy here. You can acknowledge that the Chinese AI industry has some structural advantages right now without trying to claim anything else. China is still a brutal autocracy.
- cloudfudge 4mo agoI don't think it's very common to believe the Chinese people are bad guys. It's the government and its control of the people that's the problem. And no, I don't think the US is immune to that sort of problem either.
- gmerc 4mo agoDeepseek is commoditizing the performance gains US labs rely on to make their investors money.
- teekert 4mo agoI'm deep seeking for that open in OpenAI indeed. It’s clear who’s the most anthropocentric in this space.
- resters 4mo agoThank you so much to everyone at DeepSeek who is working on this and who have the courage and generosity to open source this for humanity. We in the United States will never forget! For all the harm Trump does to the US at least he is helping China!
- sigmar 4mo ago>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2] [1] https://arxiv.org/abs/2211.17192 https://arxiv.org/abs/2211.17192 [2] https://github.com/google-gemma/cookbook/blob/main/docs/mtp/mtp.ipynb https://github.com/google-gemma/cookbook/blob/main/docs/mtp/...
- kamranjon 4mo agoThanks for the clarification - Google does publish more than others - and I actually really appreciate the work they are doing with the Gemma models, which are truly competitive open models. I do wish they’d publish more in depth papers on their Gemma models but appreciate that they are open weights.
- DiabloD3 4mo agoThey weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine. Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.
- spijdar 4mo agoGiven the MTP drafter is basically a separate model, keeping it separate makes more sense IMO. It's out of my wheelhouse but it seems like you could adjust the MTP drafter model separately from the main model, too. Ultimately though the real explanation, I think, is Google doesn't care since for their own purposes (in LiteRT-LM), they do bundle them. As far as I know, anyway.
- anaisbetts 4mo agoI mean just like GGUFs aren't technically necessary yet are _way_ more convenient than using Safetensors and configuring the default Jinja prompt by-hand, it makes sense to bundle the draft model too. For all intents and purposes, the only people who will train a draft model are the people who train the original model
- SubiculumCode 4mo agoIf American labs aren't publishing, it doesn't mean they aren't doing even more interesting work.
- californical 4mo agoYou could also come up with a cure for cancer, but if nobody knows what you’ve done then there’s not a whole lot we can say about it
- etdznots 4mo agoSo fascinating, cant wait to never hear about or be affected by this research until it’s discovered elsewhere. I genuinely wonder how it feels to be working your whole life, actual flesh and blood and heart and mind pouring 40 to make something that is a dead-end on the tree of human progress because it’s miserly masters are terrified of sharing knowledge. Days and nights spent playing pretend human pioneer, when you are a lunatic on an island building towers of coconuts.
- pmarreck 4mo agoThey push the boundaries, alright. Of obtaining the results of work without doing the work themselves, which I hate to say it but this is classic Chinese machiavellianist business behavior: https://www.cnbc.com/2026/06/24/anthropic-alibaba-distillation-campaign.html https://www.cnbc.com/2026/06/24/anthropic-alibaba-distillati...
- etdznots 4mo agoYou mean like training off of pirated copyrighted works for example that Anthropic, OpenAI, and Google stole from the internet?
- godwinson__4-8 4mo agoThe idea that America is going to stay ahead of China is I think at this point clearly delusional. It's also just such silly framing. Why should 350 million people stay ahead of 1 billion people on the other side of the world? If an AI lab in China cures cancer or something do Americans lose? So many Americans seem to (at least in theory) be ready to sign up for this ongoing confrontation with China. Does anyone think it isn't America who is poking the bear when it comes to the Thucydides trap? Why not try to get along? It occurs to me the only people more Chinese innovation would hurt are the mega cap class in the United States. Elon Musk certainly doesn't want BYD in the United States. Same story all the way down with these super capitalized AI companies. Most average Americans would probably be better off in a world where the United States and China got along. But its those Americans who will be called upon to suffer most of the burden if that trap ever springs.
- thesmtsolver2 4mo agoBy this population-only logic, you should concede that India will overtake China. Why not talk about how China shut out American companies for decades before complaining about BYD? As an Indian immigrant, the PRC China has engaged in conflict with almost all its neighbors and stated wars in its short history. China is not so benevolent when they get to the #1 spot: https://m.economictimes.com/industry/renewables/china-wto-complain-against-india-ev-manufacturing-scheme-china-wants-india-to-stop-copying-its-playbook/amp_articleshow/124598355.cms https://m.economictimes.com/industry/renewables/china-wto-co...
- godwinson__4-8 4mo agoIts not population only logic, but it does underscore that it is silly to expect the United States to inevitably be ahead. As for the rest of it: https://youtu.be/74DAI2hr9Kk?t=159 https://youtu.be/74DAI2hr9Kk?t=159
- pertymcpert 4mo agoI dont necessarily disagree with you on your other points, but the population argument sort of shows a lack of understanding of how China works. It’s not guaranteed at all that they will ever overtake the US.
- thesmtsolver2 4mo agoThis is so out of touch. Go to Neurips or the top AI conferences to see what is happening.
- nelox 4mo agoDoing work ≠ publishing work
- janalsncm 4mo agoTheir R1 paper was really well-done. But I think it leaves out a few details necessary for stable training. https://cameronrwolfe.substack.com/p/grpo-tricks https://cameronrwolfe.substack.com/p/grpo-tricks