6 ms·
FP4-sparse (TFLOPS) | Price | $/TF4s 5090: 3352 | 1999 | 0.60 Thor: 2070 | 3499 | 1.69 Spark: 1000 | 3999 | 4.00 ____________ FP8-dense (TFLOPS) | Pri
by hereme888 1y ago
FP4-sparse (TFLOPS) | Price | $/TF4s
5090: 3352 | 1999 | 0.60
Thor: 2070 | 3499 | 1.69
Spark: 1000 | 3999 | 4.00
____________
FP8-dense (TFLOPS) | Price | $/TF8d (4090s have no FP4)
4090 : 661 | 1599 | 2.42
4090 Laptop: 343 | vary | -
____________
Geekbench 6 (compute score) | Price | $/100k
4090: 317800 | 1599 | 503
5090: 387800 | 1999 | 516
M4 Max: 180700 | 1999 | 1106
M3 Ultra: 259700 | 3999 | 1540
____________
Apple NPU TOPS (not GPU-comparable)
M4 Max: 38
M3 Ultra: 36
- aurareturn 1y agoIt's not good value when you put it like that. It doesn't have a lot of compute and bandwidth. What it has is the ability to run DGX software for CUDA devs I guess. Not a great inference machine either.
- jacquesm 1y agoIt's great at one thing: memory. And that's interesting because memory is a commodity, but they still make bank on just being able to access it.
- aurareturn 1y agoMemory is a commodity but access to high bandwidth memory is expensive whether it's HBM, or LPDDR/DDR connected to many memory channels.
- throwaway48476 1y agoIts not priced in a linear way wrt to bom cost.
- aurareturn 1y agoThat's because memory channels cost money. Memory controllers are more complex. Lastly, chips that can make use of high bandwidth VRAM are both of the above. Memory chips are a commodity, that I agree. Though HBM is trending towards not being a commodity.
- throwaway48476 1y agoMemory controllers are die area. mm2 die space is linear bom cost. DRAM+mm2 bom will have a different slope to just DRAM bom but still basically linear. Nonlinear pricing is pure market segmentation.
- aurareturn 1y agoMemory controllers are die area. mm2 die space is linear bom cost. Ignoring the design and platform support that comes with higher bandwidth memory controllers.
- throwaway48476 1y agoAnd this is not also linear or fixed cost, why? I don't think you know how industry pricing works. Wafers have a price, double mm2, double the price of chip in bom.
- aurareturn 1y agoYes. Bigger die size. More complex memory controller designs. More memory lanes. More software support for higher memory lanes. All adds up. I'm explaining why despite memory being a commodity, high memory bandwidth VRAM cost is not cheap.
- throwaway48476 1y agoMemory controllers are copy paste. It's not more complex. It's more expensive in a linear way wrt bom.
- dheera 1y agoThe RTX Pro 6000 Blackwell gets you 96GB of RAM, a LOT more compute, and costs ~$7K which is not that much more than the $3K-$4K you'd pay for a DGX Spark. I think the RTX Pro is probably the best deal right now if you're looking for a GPU dev desktop and don't care about physical size or power consumption.
- jacquesm 1y agoWhat I ended up doing is buy 3090's in bulk from ex mining rigs, stick new fans in them and run 14 of them connected to an old Supermicro chassis using very nice PCI express splitters. I figured out just what the sweet spot was in terms of PCI bandwidth and maxed it out on RAM (512G). The end result was pretty usable and allowed me to run some of my own benchmarks without sharing any data with the usual suspects. The bigger problem was power delivery, the box had four pretty beefy power supplies running on two different phases. It's retired now because we're much too busy with other stuff but that was a lot of fun and gave me a much better angle on all these new developments than I would have gotten otherwise. Total cost of the system was less than one RTX Pro 6000 and it had a lot more VRAM. The whole thing looked terrible though :)
- Y_Y 1y agoYou are doing god's work. In fact you're also doing the work Nvidia should have done when they put together their (imho) ridiculously imprecise spec sheet.
- conradev 1y agowhere does an RTX Pro 6000 Blackwell fall in this? I feel like that’s the next step up in performance (and about the same price as two Sparks)
- qingcharles 1y agoI thought the 6000 was slightly lower throughput than 5090, but obviously has a shitload more RAM.
- skhameneh 1y agoIt's more throughput, but way less value and there's still no NVLink on the 6000. Something like ~4x the price, ~20% more performance, 3x the VRAM. There's two models that go by 6000, the RTX Pro 6000 (Blackwell) is the one that's currently relevant.
- QQ00 1y agothe RTX Pro 6000 (Blackwell) does not have NVlink? if so, what the fuck Mr.leather jacket.
- kouteiheika 1y agoOf course it doesn't; artificial segmentation because they really want you to buy their even more expensive datacenter GPUs for AI training.
- touisteur 1y agoL40/L40S didn't have it either, which was announced late and felt like a gut-punch for the rare actual non-AI use-case at the time. A40 nvlink was limited though, one-to-one (bridge), never saw it daisy-chained or nvswitched (might have just not seen them myself, they may have existed).
- scosman 1y agoHow does the process management comparison work for GPU vs full systems?
- canucker2016 1y ago5090: 32GB RAM (newegg & amazon lowest price seems to be +300 more) 4090: 24GB RAM Thor & Spark: 128GB RAM (probably at least 96GB usable by the GPU if they behave similar to the AMD Strix Halo APU)
- oliwary 1y agoTrue... It would be very interesting to make a comparison of various open models based on token generation speed on these platforms. Presumably starting st some size the larger accessible RAM wins out over raw speed but low VRAM? Although I suppose things like MoE and FP would also matter.
- nodesocket 1y agoOnce the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?
- wmf 1y agoUltras are pretty expensive.
- nodesocket 1y agoI mean the spark is $3,999 and current M3 Max 28-Core CPU 60-Core GPU is the same price. I would expect the refreshed studio will stay around the same price.
- KingOfCoders 1y agoIn Germany the 96gb version is 5000 EUR and the 256gb version is 7000 EUR (no 128gb available as far as I can see).
- spwa4 1y agoAt that point it's far superior to fly to the US, buy it, and fly back. Hell, have a nice week in a hotel and bring two.
- jweir 1y agoFly to a state with no sales tax. Portland, Oregon serves this purpose for high end shoppers that come from out of state and out of the country. Folks fly in to buy their Rolex, Gucci, etc, with no tax.
- KingOfCoders 1y agoDoesn't work with Germany, you'll have to pay VAT at the airport.
- 1y ago
- bjackman 1y agoNote you cannot actually get a 5090 for $1999 that's just the RRP. I believe they actually cost $4k
- IshKebab 1y agoI just googled it and the first result was one in stock for £2200. That's including tax. I assume $1999 is excluding tax. Without tax and converted to dollars it's $2470. From other less reliable sources like eBay they are more like £1800.
- bjackman 1y agoHuh, you are right. I googled it yesterday too but I guess I had confirmation bias and happened to stumble across an old price and go "yep still 4k". Well, I'm glad to be wrong on his!
- alkonaut 1y agoSo long as that's true, it's also likely you'll see the same markup for the spark, so they should compare similarly.
- hnuser123456 1y agoThe prices came down to near MSRP in the last month or so.
- boulos 1y agoAs long as you're going to add FP8 dense, you could do the same for the parts mentioned in the FP4 section. Divide by two from dense => sparse, and another two for FP4 => FP8. That gives you 250 tops of fp8 for Spark.
- nabla9 1y agoMemory is the bottleneck. It limits the size of the models you can run and what you pay for. Spark: 128 GB LPDDR5x, unified system memory 5090 : 32 GB GDDR7, Model sizes (parameter size) Spark: 200B 5090 : 12B (raw)
- deleted 1y ago[deleted]
- artemisart 1y agoThat's very true and what's segmenting the market, but I don't understand why you're saying the 5090 supports only 12B model when it can go up to 50-60B (= a bit less than 64B to leave room for inference) as it supports FP4 as well.
- nabla9 1y agoIts for comparison using raw, non optimized models. Both can do much better when you optimize for inference. Information is in the ratio of these numbers. They stay the same.
- artemisart 1y agoOk then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.
- ilirium 1y ago@nabla9 have tried to tell you that for DGX Spark, you can also use optimized models; therefore, this means that Spark can also be used for inference with bigger models, such as those exceeding 200B. Please compare the same things: carrots VS carrots, not apples VS eggs.
- artemisart 1y agoI don't understand what's not optimized on 5090. If we're comparing with Apple chips or AMD Strix Halo yes you will have very different hardware + software support, no FP4 etc. but here everything is CUDA, Blackwell vs Blackwell, same FP4 structured sparsity, so I don't get how it would be honest to compare a quantized FP4 model on Spark with an unoptimized FP16 model on a 5090 ?
- eurekin 1y agoSo how many generation T/s we can expect for a dense model? I assume we can go up to 120B using fp8?
- deleted 1y ago[deleted]