14 ms·
The path to open-sourcing the DeepSeek inference engine
- rfoo 2y agotl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"
- Havoc 2y agoUnmaintainable seems unduly harsh. There is a big gap between maintainable internally and ready for public consumption
- rfoo 2y ago> Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago If you are in the same boat you'll see how much changed in vLLM compared to one year ago. Also, this meant that they haven't rebased for over a year, I don't believe that's because they don't want, it's because they effectively can't. Yeah, surely they can maintain it as-is. But it will be increasingly hard to port over anything community has.
- lukeschlather 2y agoI get the impression their setup is very hard to maintain but it's worth every penny. They've done optimizations that wring incredible performance out of the hardware they have, but they also have specific machine configurations and I wouldn't be surprised if they have complicated hacks that get 100% speedups for some stuff but those speedups disappear if you have a slightly different motherboard configuration. Also there's suggestion they've made firmware hacks which are worth it at their scale, but might be very dangerous and difficult to apply especially on a small scale. (And some of their hacks might involve both firmware and cluster-level optimizations, which would be useless or counterproductive independently.) And even if you have somewhat similar hardware, the code might not be that helpful, you might be better off with a sketch of the solution and implementing it yourself. If you've got a large enough cluster it's going to pay for itself anyway.
- maknee 2y agoThey're going to spend time and effort into making their optimizations public. Would you rather have them keep their changes internal?
- oldgun 2y agoNice. We've seen some good engineering work from DeepSeek. Keep it coming.
- vintagedave 2y agoI really empathised with this part: > Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases. I've been there. Probably a few of us have. Their approach of working on splitting out maintainable sublibraries and sharing info directly even if not integrated seems a really nice way of working with the community -- ie, they have obstacles, but they're not letting the obstacles cause them to take the easy route of not contributing at all. And while it might seem better to someone wanting to use their techniques to share only working code, not info on the techniques, at least it's still knowledge sharing. And again I think it'd be easier for them not to do it. So kudos to them.
- rvnx 2y agoThey customized and optimized vLLM for their use case, so much that it became a different product (e.g. Debian vs Ubuntu). The fact they share back some of their improvements is great.
- bonoboTP 2y agoNon-runnable code can be really useful. I often wish it was available for some papers even if I never run it just to check what they actually did, because text and equations are often not specific enough.
- ozgune 2y agoIn March, vLLM picked up some of the improvements in the DeepSeek paper. Through these, vLLM v0.7.3's DeepSeek performance jumped to about 3x+ of what it was before [1]. What's exciting is that there's still so much room for improvement. We benchmark around 5K total tokens/s with the sharegpt dataset and 12K total token/s with random 2000/100, using vLLM and under high concurrency. DeepSeek-V3/R1 Inference System Overview [2] quotes "Each H800 node delivers an average throughput of 73.7k tokens/s input (including cache hits) during prefilling or 14.8k tokens/s output during decoding." Yes, DeepSeek deploys a different inference architecture. But this goes onto show just how much room there is for improvement. Looking forward to more open source! [1] https://developers.redhat.com/articles/2025/03/19/how-we-optimized-vllm-deepseek-r1 https://developers.redhat.com/articles/2025/03/19/how-we-opt... [2] https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20...
- nashashmi 2y agoI feel like this is one way to implement censorship.
- efeamzaov 2y ago[dead]
- holoduke 2y agoI wonder if the large quantity release of opensource AI tools, models etc is a deliberate strategy of China to counter the US dominance. A good thing for the market imho
- avodonosov 2y agoWhat motivates the commercial AI companies to share their research results and know-how? Why did Google published the Transformer architecture instead of keeping it to themselves? I understand that people may want to do good things for humanity, facilitate progress, etc. But if an action goes against commercial interest, how can the company management take it and not get objections from shareholders? Or there is a commercial logic that motivates sharing of information and intellectual property? What logic is that?
- varelse 2y ago[dead]
- lofaszvanitt 2y agoThe more people copy your outdated thing, the better for you, because they always gonna lag behind you.
- bcoughlan 2y agoI would guess it comes down to that the best researchers in the world want their work out in the open
- behnamoh 2y agoIlya doesn't. He's a strong proponent of closed source and censorship.
- esperent 2y agoHe is just one person. He happens to be the most famous scientist working on this field at the moment it became a gold rush, but it's work built on the shoulders of those who came before, whose discoveries are just as important.
- Zambyte 2y agoLet's hope Totally Safe Intelligence doesn't end up having the same relationship with their name as Totally Open AI.
- londons_explore 2y ago"We have something that would be of interest to the open source community, but it needs a lot of tidying to even run outside our company, and we don't have the manpower to properly maintain it when released". Plenty of companies are in this position. Please just open source anyway with a note saying "we won't be maintaining this, but feel free to fork!"
- lolinder 2y agoUnfortunately that's not really feasible in the current state of open source. There are enormous numbers of entitled users out there who become a parasitic drain on any project that is open sourced. Solo maintainers can theoretically just develop a think skin, but companies can actually find that the damage to their public image from not having their FOSS project in tip top shape is greater than the benefits of open sourcing it in the first place.
- golergka 2y agoYou can always just disable issues and all other feedback channels. I judge companies on the state of their open source libraries, but it really depends on how the company positions it. Facebook pushed for react to become the default framework, so they deserve the scrutiny when it doesn’t hold up. However, with a clear disclaimer like that? I think that people would have different expectations.
- lolinder 2y agoYou might, but consumers of FOSS code are extremely unreasonable.
- boredatoms 2y agoJust put a tarball url on a plain website then. No community interaction required to share code
- 3abiton 2y ago> There are enormous numbers of entitled users out there who become a parasitic drain on any project that is open sourced. Been there with AOSP, but that won't be changing anytime soon. I highly doubt noobs will learn the open source etiquette unfortunately.
- jeffrallen 2y agoWhat if it turns out Deep seek is actually the first GenAI, and this is the way forward is has chosen: open source itself? Kind of like how biological information is always trying to find new places to reproduce itself. Viruses and fungi do not come with Toss and EULAs. :)
- animal531 2y agoI spent the last two or so months using it as an assistant for code and my conclusion is that it is terrible compared to even the free model of ChatGPT. The incidence of bugs, it not understanding what you're asking or just generating code that is straight up wrong is much worse. Even with guidance it will often be unable to fix issues, leaving you to do all the manual legwork to get things working. Usually you're better off just having done everything yourself from the start. During those two months they really improved GPT as well, its generation speed is now much much faster, and the quality of its output has become a lot better.
- buyucu 2y agoDeepseek is everything OpenAI claims to be.
- gizmodo59 2y agoAs much as I want to geek out and run the things locally, if I have the money I just want to use a SaaS. I want to spend time creating new applications and not toy around with setup, infrastructure etc. I’d gladly pay ChatGPT even more if they keep up with features and they seem to have done that quite well since deepseek (new models often, best image gen hands down, very fast inference compared to 6 months back and even small things like memory). I sometimes feel guilty though. With all this power, I’m just bounded by lack of ideas and execution.
- wseqyrku 2y agoDeepSeek is the only company that dares to say 'towards AGI' next to OpenAI.