4 ms·
DwarfStar4 is a small LLM inference runtime that can run DeepSeek 4. The blog post implies that it currently requires 96GB of VRAM. For others who are lacking
by gcr 5mo ago
DwarfStar4 is a small LLM inference runtime that can run DeepSeek 4. The blog post implies that it currently requires 96GB of VRAM.
For others who are lacking context :-)
- foresto 5mo agoThanks. Outside of LLM circles, DS4 is usually a video game controller.
- artyom 5mo agoWell, I was sitting here expecting the Redis creator have an opinion on still-unannounced Dark Souls 4.
- low_tech_love 5mo agoHaha the same here!!
- jofzar 5mo agoI am actually kind of disappointed it wasn't a deep dive on the dual shock 4
- oezi 5mo agoOr a car from Citroen
- pavlov 5mo agoTechnically DS is an independent sibling of Citroën within Stellantis, a sprawling car conglomerate that owns a dog’s dinner of car brands in Europe and USA.
- orthoxerox 5mo agoIt's still the Lexus to Citroen's Toyota.
- drcongo 5mo agoPavlov's dog's dinner?
- Hamuko 5mo agoIf we want to get really technical, “DS4” is a model from Citroën and they later spun out the DS lineup into its separate brand, with the “Citroën DS4” becoming “DS 4”, “DS” being the make and “4” being the model.
- pavlov 5mo agoAnd even more pedantically, DS has recently adopted a new naming scheme where the former DS 4 is now written as DS N°4, pronounced "number 4"... Their stated inspiration for this SEO bomb is Chanel perfumes.
- insensible 5mo agoTrekkies are experiencing a major regression from Deep Space Nine.
- DeathArrow 5mo ago>The blog post implies that it currently requires 96GB of VRAM. From the Github page it seems it only supports Apple and DGX Spark. I have 128 GB of RAM and a 3090 but it probably won't work.
- thomasm6m6 5mo agoFYI, llama.cpp (which antirez/ds4 is inspired by) supports system ram. E.g. [1] is a good guide for running a similar-sized model with 128gb ram and a 3090-sized GPU. [1] https://unsloth.ai/docs/models/tutorials/minimax-m27 https://unsloth.ai/docs/models/tutorials/minimax-m27 (Unsloth's deepseek-v4 support is still WIP)
- DeathArrow 5mo agoThanks, I can run Qwen 3.6 27B with vllm, but I was curious about antirez tool.
- embedding-shape 5mo agoHave you had it getting stuck in endless loops maybe ~10-20% of the invocations? Seems it happens for both the responses and chatcompletion APIs, and no matter what inference parameters I try it happens at least for 1/10 of the requests, I've tried every compatible vLLM version + currently using it from git (#main) yet the issue persists. Seems to happen with various quantizations too, even the NVFP4 versions and any others, so seems like a deeper issue to me, or hardware incompatible perhaps.
- modmans2nd 5mo agoThere’s a fixed version out there with corrected templates.
- manmal 5mo agoIt wouldn’t be useful with your setup, probably 3-4 token per second.
- zozbot234 5mo ago> The blog post implies that it currently requires 96GB of VRAM. Has anyone tested what happens if you try and run this on lower-RAM Macs? It might work and just be a bit slower as it falls back on fetching model layers from storage.
- conradkay 5mo agoIt'd be way slower since you'd be doing that work every token
- zozbot234 5mo agoTrue (with 64GB RAM it'd have to fetch 20% of its active experts from disk already, about 650MB/tok at 2-bit quant - and that percentage rises quickly as you lower RAM further); my question is just a more practical one about whether it runs at all, how bad the slowdown is, and to what extent you might be able to get some of that decode throughput back by running multiple (slower) agent sessions in parallel under a single Dwarf Star 4 server.
- computably 5mo agoStorage is multiple orders of magnitude slower than RAM. Pretty sure it'd be more like 10s/tok than anything reasonable.
- zozbot234 5mo agoActive params for this model is 13B which takes about 6.5GB at full native quantization, or perhaps 3.25GB at the 2bit quant that's being provided here, that should take significantly less than 10s to fetch on Mac storage, especially given that some fraction of the model weights would be cached in RAM. Sounds like something worth testing out if it can be made to work out of the box with DS4.
- rpigab 5mo agoI knew Death Stranding 3 wasn't out yet!
- smcleod 5mo agoThat's the flash version not the full model and only at Q2-3~ so while impressive it's still quite different from the full model.
- rurban 5mo agoNot really. I'm building now another fast C compiler with DeepSeek 4 Flash, and rarely have to step outside to use Pro or Sonnet, gpt or kimi-2.6. Flash is very capable of almost everything.
- gekoxyz 5mo agowhich harness are you using? pi? opencode?
- rurban 5mo agoThat's not a harness. That's an agent cli. A harness is something completely different. Wish people could use proper terminology. A test harness is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It automates the execution of test suites, providing the necessary stubs, drivers, and runtime environments so developers can isolate and verify specific code components. I use opencode (lockedcode is still vaporware), claude, kimi and codex. And most models. Just no Google models so far, I don't trust them.
- Sinidir 5mo agoHarness: a piece of equipment with straps and belts, used to control or hold in place a person, animal, or object. So yes the generel meaning applies to test setup and running and also to the agent cli which is the harness for the model.
- rurban 5mo agoNo, an agent cli is no harness. You have to provide a harness for an agent by yourself, otherwise it will run free. Which is called vibe coding. Free as you wish, without any harness.
- Wowfunhappy 5mo agoThanks. How is DwarfStar4 different from llama.cpp?
- covoeus 5mo agollama.cpp is general purpose in the sense that it supports many different model architectures. ds4 is laser focused on deepseek v4 flash, thus having a leaner codebase