4 ms·

Why? 4 x DGX sparks should be enough. That's way less than $80k.

• throwawayffffas 3mo ago
  From a quick google search a DGX spark seems to decode Llama 3.1 70B (FP8) at 2 tokens per second. I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts. For real work anything below 60 tokens per second is essentially unusable. That's not taking into account the prompt filling, Llama 3.1. 70b on DGX spark runs at about 800 tps running at that speed prompt filling a 512k context takes like 11 minutes.
  • naasking 3mo ago
    > I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts. To fit that model you'd need multiple machines which lets you run tensor parallelism for additional speedups, roughly X speedup for X machines. Also you can't naively compare tps across models, and it would certainly run a nvfp4 quant or similar.
• villish 3mo ago
  It would run so slow it would be functionally unusable or at such a low quant that it wouldn't be useful for serious work.
  • naasking 3mo ago
    nvfp4 is pretty good, and with 4 machines you get a roughly 4x speedup using tensor parallelism.