4 ms·
Supposedly people are getting ~40 tps decode at Q8 on 2× DGX Spark (higher for Q4) which is what I assume they're suggesting is just under $10K USD. Prefill is
by Alpha3031 1mo ago
Supposedly people are getting ~40 tps decode at Q8 on 2× DGX Spark (higher for Q4) which is what I assume they're suggesting is just under $10K USD. Prefill is just above 1.5k so TTFT is maybe 2 to 4 minutes? I don't have two DGX Sparks myself so I can't confirm and not 100% sure if that number is with or without speculative decoding already in use (if not probably around 60 to 80 if enabled?).