4 ms·

What I think would be perfect is a model that could run on a single DGX spark and be competitive with DSV4 Flash 731. Flash is already a game changer. Hopefully

What I think would be perfect is a model that could run on a single DGX spark and be competitive with DSV4 Flash 731. Flash is already a game changer. Hopefully meta plans on this, like the old 70b. V4 flash is smart enough for any use but slightly too big. 27b-30b isn’t intelligent enough.

127 2mo ago
DSV4 Flash 0731 already runs on RTX 4090 24GB + 128GB system RAM at a usable tok/s and quantization.
- Gecko4072 2mo ago
  You personally? Just curious. Context window is also a factor and ram isn’t really cheap. Sparks are assembled units which I like.
  - dannyw 2mo ago
    For the same price as a DGX Spark here (A$8499) I can buy roughly 544GB of DDR5-5200MHz from retail; which on a quad channel platform would deliver ~160gb/s real world; and ~320gb/s with octa channels (Xeon, Threadripper Pro). If you can afford it or somehow find a used unit, you can go Epyc for 12 channels. 8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world. With some optimisation and maths, it’s entirely plausible to ach You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still. Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet. Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.
    - danielEM 2mo ago
      Been investigating these multichannel AMD based platforms last year and seem like none of them can in real scenarios utilize anywhere close to their theoretical bandwidth.
  - kybernetikos 2mo ago
    I've run it with a large context window on 256GB ram + 4090. It wasn't super fast, but it was manageable and it completed the tasks I gave it well.
    - xiconfjs 2mo ago
      How many tps and size of ctx?
cmrdporcupine 2mo ago
This model I think will be too slow for that on Spark, even at 4 bit quant. It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size: NVFP4 Q/K/V/O and MLP projections: ~13 GB/token BF16 attention gates: ~3 GB/token BF16 LM head: ~2.5 GB/token Total: ~18.9 GB/token At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.
- rao-v 2mo ago
  Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.
  - cmrdporcupine 2mo ago
    You're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based) That's pretty respectable. Still working on optimizing and cleaning up before I push it.
    - cmrdporcupine 2mo ago
      https://github.com/rdaum/eider/ https://github.com/rdaum/eider/ fwiw

127 2mo ago

DSV4 Flash 0731 already runs on RTX 4090 24GB + 128GB system RAM at a usable tok/s and quantization.

Gecko4072 2mo ago
You personally? Just curious. Context window is also a factor and ram isn’t really cheap. Sparks are assembled units which I like.
- dannyw 2mo ago
  For the same price as a DGX Spark here (A$8499) I can buy roughly 544GB of DDR5-5200MHz from retail; which on a quad channel platform would deliver ~160gb/s real world; and ~320gb/s with octa channels (Xeon, Threadripper Pro). If you can afford it or somehow find a used unit, you can go Epyc for 12 channels. 8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world. With some optimisation and maths, it’s entirely plausible to ach You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still. Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet. Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.
  - danielEM 2mo ago
    Been investigating these multichannel AMD based platforms last year and seem like none of them can in real scenarios utilize anywhere close to their theoretical bandwidth.
- kybernetikos 2mo ago
  I've run it with a large context window on 256GB ram + 4090. It wasn't super fast, but it was manageable and it completed the tasks I gave it well.
  - xiconfjs 2mo ago
    How many tps and size of ctx?

cmrdporcupine 2mo ago

This model I think will be too slow for that on Spark, even at 4 bit quant. It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size: NVFP4 Q/K/V/O and MLP projections: ~13 GB/token BF16 attention gates: ~3 GB/token BF16 LM head: ~2.5 GB/token Total: ~18.9 GB/token At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.

rao-v 2mo ago
Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.
- cmrdporcupine 2mo ago
  You're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based) That's pretty respectable. Still working on optimizing and cleaning up before I push it.
  - cmrdporcupine 2mo ago
    https://github.com/rdaum/eider/ https://github.com/rdaum/eider/ fwiw