3 ms·
Memory is the bottleneck. It limits the size of the models you can run and what you pay for. Spark: 128 GB LPDDR5x, unified system memory 5090 : 32 GB GDD
by nabla9 1y ago
Memory is the bottleneck. It limits the size of the models you can run and what you pay for.
Spark: 128 GB LPDDR5x, unified system memory
5090 : 32 GB GDDR7,
Model sizes (parameter size)
Spark: 200B
5090 : 12B (raw)
- deleted 1y ago[deleted]
- artemisart 1y agoThat's very true and what's segmenting the market, but I don't understand why you're saying the 5090 supports only 12B model when it can go up to 50-60B (= a bit less than 64B to leave room for inference) as it supports FP4 as well.
- nabla9 1y agoIts for comparison using raw, non optimized models. Both can do much better when you optimize for inference. Information is in the ratio of these numbers. They stay the same.
- artemisart 1y agoOk then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.
- ilirium 1y ago@nabla9 have tried to tell you that for DGX Spark, you can also use optimized models; therefore, this means that Spark can also be used for inference with bigger models, such as those exceeding 200B. Please compare the same things: carrots VS carrots, not apples VS eggs.
- artemisart 1y agoI don't understand what's not optimized on 5090. If we're comparing with Apple chips or AMD Strix Halo yes you will have very different hardware + software support, no FP4 etc. but here everything is CUDA, Blackwell vs Blackwell, same FP4 structured sparsity, so I don't get how it would be honest to compare a quantized FP4 model on Spark with an unoptimized FP16 model on a 5090 ?
- NewsaHackO 1y agoTo me, what I think they are saying is that the Spark can use a FP16 unoptimized model with 200B parameters. However I don't really know.
- reissbaker 1y agoYou can't. The Spark has 128GB VRAM; the highest you can go in FP16 is 64B — and that's with no space for context. 200B is probably a rough estimate of Q4 + some space for context. The Spark has 4x the VRAM of a 5090. That's all you need to know from a "how big can it go" perspective.
- canucker2016 1y agofrom the NVidia DGX Spark datasheet: With 128 GB of unified system memory, developers can experiment, fine-tune, or inference models of up to 200B parameters. Plus, NVIDIA ConnectX™ networking can connect two NVIDIA DGX Spark supercomputers to enable inference on models up to 405B parameters.
- reissbaker 1y agoThe datasheet isn't telling you the quantization (intentionally). Model weights at FP16 are roughly 2GB per billion params. A 200B model at FP16 would take 400GB just to load the weights; a single DGX Spark has 128GB. Even two networked together couldn't do it at FP16. You can do it, if you quantize to FP4 — and Nvidia's special variant of FP4, NVFP4, isn't too bad (and it's optimized on Blackwell). Some models are even trained at FP4 these days, like the gpt-oss models. But gigabytes are gigabytes, and you can't squeeze 400GB of FP16 weights into only 128GB (or 256GB) of space. The datasheet is telling you the truth: you can fit a 200B model. But it's not saying you can do that at FP16 — because you can't. You can only do it at FP4.