4 ms·

256gbs memory bandwidth is about 1/4 that of a 3090. It would be a better buy with half the memory at 4x the speed.

wolttam 3mo ago
Are you sure about that? High memory speed is great for dense models, or when serving at high concurrency. However for local single-user setups, it's often better to have access to more capable/bigger MoE models at reasonable speeds and lower concurrences, which is enabled by these platforms.
- roadside_picnic 3mo ago
  If you're using a MoE model, then why do you care about the larger RAM offered by these devices? That's the main problem with low bandwidth devices: they limit the effective ram you can make use. I do (and have historically done) quite a work with both local LLMs and local diffusion models. I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s While the M3 Max MBP can serve up MoE reasonably fast (~60 token/sec)the RTX 4090 is an entirely different experience (~170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX. The raison d'etre for the DGX spark is not practical home inference, but rather offering the same fundamental architecture as data center cards for a affordable CUDA prototyping. If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. The DGX spark solves this by having the same fundamental setup as what those cards have. That makes these non-NVIDIA DGX-like devices confusing to me. The entire benefit of the DGX series is the NVIDIA architecture itself. Anyone interested in home LLMs should decide whether a Mac or a dedicated GPU is the more sensible path based on their budget and other computer use. Each has their own benefits.
  - wolttam 3mo ago
    I run DSv4 Flash at home on 2 DGX Sparks and am pretty sure there is no more cost effective way for me to do so. I'm not interested in running smaller models.
    - roadside_picnic 3mo ago
      What's the tokens/sec you're getting on that setup (genuinely curious because it's a setup I haven't actually run myself)?
      - wolttam 3mo ago
        Using regular DeepSeek-V4-Flash, I see 2000tok/s prompt processing and from 40 to 50 tok/s generation. Performance doesn't drop much at long contexts, DSv4 is really nice for that. With DeepSeek-V4-Flash-DSpark (Deepseek's new speculative decoding scheme), which is still barely supported anywhere, we're seeing a more steady 45-55 tok/s with bursts into the 60s.
        roadside_picnic 3mo ago
        > from 40 to 50 tok/s generation That's actually much better than I would have thought! Thanks for the answer and it does make this approach make more sense as a budget solution to running larger models locally.
        verdverm 3mo ago
        What quant size are you using? Just picked up my second spark and a cable and wanted to try this one out for fun. I generally run a bunch of models for different purposes (easy / bulk tasks) and use APIs for harder tasks.
        wolttam 3mo ago
        It's the native DeepSeek V4 Flash quant which is released in MXFP4. You have enough RAM left over with 2 sparks for ~2M tokens of working KV cache.
cyanydeez 3mo ago
what matters is how much memory it has; with the new MTP models, Qwen3.6 with 35B MOE, it's pumping out tokens up to ~80k context with little slow down. It's great to get lots of tokens, but being able to handle and extent context is why it'll continue to be a great machine compared to any of the small graphics cards.
muyuu 3mo ago
it depends it allows you to run smaller models much better imo 3090s make the most sense if you can buy at least 2x ideally 4x but of course we're talking about a completely different budget at that point
StableAlkyne 3mo ago
Any performance gains caused by the internal bandwidth of the card will evaporate once you spill into system RAM, because now your bottleneck is probably a slow PCI lane. And if your jobs do fit onto a 24GB card, then you are not the target user for the "AI mini PC" niche that these guys are trying to carve out

Are you sure about that? High memory speed is great for dense models, or when serving at high concurrency. However for local single-user setups, it's often better to have access to more capable/bigger MoE models at reasonable speeds and lower concurrences, which is enabled by these platforms.

roadside_picnic 3mo ago
If you're using a MoE model, then why do you care about the larger RAM offered by these devices? That's the main problem with low bandwidth devices: they limit the effective ram you can make use. I do (and have historically done) quite a work with both local LLMs and local diffusion models. I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s While the M3 Max MBP can serve up MoE reasonably fast (~60 token/sec)the RTX 4090 is an entirely different experience (~170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX. The raison d'etre for the DGX spark is not practical home inference, but rather offering the same fundamental architecture as data center cards for a affordable CUDA prototyping. If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. The DGX spark solves this by having the same fundamental setup as what those cards have. That makes these non-NVIDIA DGX-like devices confusing to me. The entire benefit of the DGX series is the NVIDIA architecture itself. Anyone interested in home LLMs should decide whether a Mac or a dedicated GPU is the more sensible path based on their budget and other computer use. Each has their own benefits.
- wolttam 3mo ago
  I run DSv4 Flash at home on 2 DGX Sparks and am pretty sure there is no more cost effective way for me to do so. I'm not interested in running smaller models.
  - roadside_picnic 3mo ago
    What's the tokens/sec you're getting on that setup (genuinely curious because it's a setup I haven't actually run myself)?
    - wolttam 3mo ago
      Using regular DeepSeek-V4-Flash, I see 2000tok/s prompt processing and from 40 to 50 tok/s generation. Performance doesn't drop much at long contexts, DSv4 is really nice for that. With DeepSeek-V4-Flash-DSpark (Deepseek's new speculative decoding scheme), which is still barely supported anywhere, we're seeing a more steady 45-55 tok/s with bursts into the 60s.
      - roadside_picnic 3mo ago
        > from 40 to 50 tok/s generation That's actually much better than I would have thought! Thanks for the answer and it does make this approach make more sense as a budget solution to running larger models locally.
      - verdverm 3mo ago
        What quant size are you using? Just picked up my second spark and a cable and wanted to try this one out for fun. I generally run a bunch of models for different purposes (easy / bulk tasks) and use APIs for harder tasks.
        wolttam 3mo ago
        It's the native DeepSeek V4 Flash quant which is released in MXFP4. You have enough RAM left over with 2 sparks for ~2M tokens of working KV cache.

cyanydeez 3mo ago

what matters is how much memory it has; with the new MTP models, Qwen3.6 with 35B MOE, it's pumping out tokens up to ~80k context with little slow down. It's great to get lots of tokens, but being able to handle and extent context is why it'll continue to be a great machine compared to any of the small graphics cards.

muyuu 3mo ago

it depends it allows you to run smaller models much better imo 3090s make the most sense if you can buy at least 2x ideally 4x but of course we're talking about a completely different budget at that point

StableAlkyne 3mo ago

Any performance gains caused by the internal bandwidth of the card will evaporate once you spill into system RAM, because now your bottleneck is probably a slow PCI lane. And if your jobs do fit onto a 24GB card, then you are not the target user for the "AI mini PC" niche that these guys are trying to carve out