3 ms·
In case people want to run it, it's DeepSeek-V4-Flash-284B-A13B. So it should just barely run on a single B300, and it's small enough that it'll barely run on a
by spwa4 3mo ago
In case people want to run it, it's DeepSeek-V4-Flash-284B-A13B. So it should just barely run on a single B300, and it's small enough that it'll barely run on an M5 Max too.
- wolttam 3mo agoIt runs really well on 2 DGX Sparks - 60t/s
- Tepix 3mo agoYes, the Dual DGX Spark looks like the sweet spot for this model for now. Good preprocessing speed. Lots of context. Fast enough for 1-5 devs perhaps. Around 8200€ as of today (used to be 6000€). Dual Strix Halo is much slower and current Macs with 256GB are both slower and more expensive (Mac Studio M3 Ultra 256GB around 12000€). To get something faster than the two Sparks you'd need to spend more than $22000 for a server with 2x RTX Pro 6000 at $10000 each. Beyond that you could get 2x AMD MI350P.
- arjie 3mo agoNot yet, right? That's the old DS V4 preview release. We're still waiting for the weights to come out.
- benjiro29 3mo agoProbably the same. When the same base model is trained, the weight do not tend to change a lot. GLM 5.0 > 5.1 > 5.2 are the same base model, that just kept being trained. Weights hardly change as a result. Think in the like few percentage points size difference.
- deleted 3mo ago[deleted]
- lukan 3mo ago"it'll barely run on an M5 Max " The max version I could order now with 128 GB? If so, the price for local inference would be 12 000 € vs 500 000 € for a B300.
- NitpickLawyer 3mo agoThere's also the 2x spark way, which should be ~8k eur? Someone down the thread reported ~60tps for 2x sparks. That's totally usable for local inference. You can also do 2x 6kPRO in a workstation, for ~20k.
- matrik 3mo agoFor the same performance, one could even go about 50% cheaper with 16 channel ddr4 + a rtx3090 for prompt processing. But still, even for mid level projects API is orders of magnitude cheaper, since you don't need to set it up and maintain it.
- gpugreg 3mo agoThe memory bandwidth of the 2x RTX Pro 6000 Blackwell setup will be 10x higher, which should have an equivalent effect on the generated tokens per second.
- spwa4 3mo agoCurrently the 3bit (and 2 bit) quant on DGX spark (on one of them) and the M5 Max should just start. Right now. (I'm hoping to get an M5 Max delivered on monday, let's see if it happens this time. It's 2+ months since I ordered now) The 4 bit quant technically fits (there's a 127 GB version) but ... obviously that's not going to work. It is so close though, surely someone will a way to do it.
- reverius42 3mo agoI'm running a useful quantization of the previous version of Deepseek-V4-Flash -- quite well but with so much fan noise -- on a MacBook Pro M5 Max with 128 GB.
- 3mo ago