4 ms·
I'm genuinely disappointed with my Spark. I don't know how anyone can claim it performs decently with LLMs or diffusion models. Back when I worked in VFX in the
by trentor 3mo ago
I'm genuinely disappointed with my Spark. I don't know how anyone can claim it performs decently with LLMs or diffusion models. Back when I worked in VFX in the early 2000s, we had a saying: "Render time is coffee time" and if you try to run this thing with a usable context size, you'll be drinking a lot of coffee. Most of the optimizations it relies on for inference simply aren't available for training, so it crawls like a snail on almost every model. An RTX 6000 Blackwell would have been the better investment for an AI enthusiasts and for general computing there are cheaper offerings.
- mapontosevenths 3mo agoIf you bought it for inference you made a mistake. They aren't good at that. Use it to train models and experiment with ML. It's much better at that. If you just want local inference buy a Mac. If you bought early on, like I did, the Spark is probably worth double what you payed now. I think I paid $3,000 retail for mine and the last time I looked they were fetching close to $6k on ebay. I'm not sure if that's still the case, but you can buy a very nice Mac with $6k.
- trentor 3mo agoMhm... maybe read my full comment?
- mapontosevenths 3mo agoAre you saying that it's slow for training? Sorry, your comment is confusingly worded to me. I've not had any issues in that regard, but I'm working with LLM's not training diffusion models. Are you following one of the Nvidia provided recipes or inventing something on your own? The last time I looked into it they benchmarked very well, but we both know that doesn't always mean much.
- trentor 3mo agoYes it is slow for training/fine-tuning decently sized LLMs and unusable for diffusion models. What are you training were you get usable speeds?
- dllu 3mo agoThe DGX Spark hits a sweet spot for me where it can simultaneously work for general computing and run local LLM inference fast enough for some hobbyist dabbling. The RTX 6000 Pro Blackwell is more than twice the price (it has increased quite a bit recently from $8000 to $11600), not to mention the "rest of the PC" needed to get it working, so it's not really a fair comparison. Compared to other 128 GB unified memory devices like the Mac Studio and the Strix Halo, the DGX Spark fairly priced in my opinion.
- deleted 3mo ago[deleted]
- icedchai 3mo agoI have a AMD Strix Halo box I use for similar dabbling. It definitely wasn't an "out of the box" experience, fiddling around with kernel versions and ROCm installs. These days I mostly wind up using the Vulkan build of llama.cpp for inference.
- Foobar8568 3mo agoI have access to both a RTX 5090 PC with 64gb of ram and a Spark 128gb, the performance of the Spark has been highly disappointing. I prefer 1000x the RTX one, even with 64gb of ram.