4 ms·
> This project evaluates local language models running on a single NVIDIA DGX Spark. "Did much better" is a bit misleading w/o that context and 1 hour time lim
by petu 1mo ago
> This project evaluates local language models running on a single NVIDIA DGX Spark.
"Did much better" is a bit misleading w/o that context and 1 hour time limit -- your benchmark design heavily favors V4 Flash. From results on your page V4 Flash processed 1-1.5M tokens an hour, while Q3.8 27B was failed before even reaching 200K tokens.
By the way, how are you running V4 Flash on single Spark? Was it quantized?
- __alexander 1mo agoA single model was loaded at a time and each model was given a 90 minute timeout. This is how I’m running it (actually an older version because they deleted the repo and replaced it) https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark