4 ms·
Nvidia's latest AI PC boxes sound great – for data scientists with $3k to spare
- johnea 2y ago$3K, for what? The "NVIDIA Inside" sticker? Any org with less than several hundred thousand dollars to spare isn't anyone that NVIDIA cares about at all...
- nabla9 2y agoNvidia should join with Valve and make it a high-end gaming console. It would be nice dual use system.
- protimewaster 2y agoI would support this if it meant that Nvidia would improve their Linux drivers. As it stands, though, a device running SteamOS on Nvidia hardware sounds painful.
- hedora 2y agoIt'll take a long time for them to rebuild their reputation on that front. I threw away a pile of perfectly serviceable NVIDIA desktop boards a while back because they were "supported", but they couldn't do stuff like run modern compositors or even XScreenSaver. These were high-end boards that worked fine in Linux when I bought them (as long as you were willing to periodically patch + recompile the kernel from a text console). Anyway, once they have feature parity with Windows + current CUDA with a 100% open source stack going back a decade, then I'll consider them again. Until then, I'll happily buy an AMD that's 90% as fast for the same price.
- tedivm 2y agoThere are already a ton of high-end gaming consoles you can buy today with nvidia chips. In fact, this is exactly what I did for my own AI machine (dual 3090s).
- hedora 2y agoA console would be a device like an XBox, PS5 or Switch. Of those, only the Switch (current + next gen) are NVIDIA devices. The Steam Deck and the other current generation consoles are all AMD.
- ekianjo 2y agoit would be nice if Valve's Steam Big Picture mode actually worked with Nvidia. It's still not the case.
- nickpsecurity 2y agoTheir last console probably inspired them to not try again for a while.
- gpderetta 2y agoThe steam deck? I would say it was fairly successful.
- nickpsecurity 2y agoI got my products mixed up in my head. I retract that. Thanks.
- whalesalad 2y agoHas anyone benchmarked the Spark against a comparably priced Mac?
- dagw 2y agoThe comparably priced Mac comes with half the RAM, so if that's a constraint, it's already lost. For applications that aren't memory constrained it would be very interesting to see some benchmarks. Although I suspect it's going to come down to the quality of the relevant CUDA vs Metal libraries as much as anything else.
- dcrazy 2y agoHow are you arriving at double the price? 16-CPU-core Mac Studio with storage upgrade to 1TB and RAM upgrade to 128GB comes out to $3,700. Add $700 if you want to upgrade to the 28-core CPU (there’s nothing directly comparable to the DGX Spark’s 20-core CPU). None of this speaks to the actual GPU performance. The only spec the DGX Spark webpage mentions is “1,000 FP4 TOPS”.
- nickthegreek 2y agoId love to see a head to head on the DGX, Mac Studio and the framework models to see where the strength and weaknesses are for each.
- mirawelner 2y agothis reminds me of the (now discontinued) lambda tensorbook that I bought a few years back. AI computers are interesting because they are extremely similar to really good gaming computers but the marketing (and some of the hardware) is different. I don't fully understood why they are separate from gaming computers other than marketing, but I am very happy with my tensorbook. I got it when I was very young and impulsive I don't know if I would get it again, although that fact alone speaks to the computer's longevity.
- Tsiklon 2y agoLambda collaborated with Razer on their last tensorbook in 2022. It was literally a then current Razer Blade running Ubuntu
- nickpsecurity 2y agoGaming computers usually run Windows and games. AI computers often run Linux and AI software. The differences require different configurations of both hardware and software. So, they make products preloaded with configurations for each market segment. The customers pay for all the headaches this saves them of custom setups.
- Tenoke 2y ago$3k is already close to the price of just a 5090, so if you can have a reasonably fast but way more (equivalent to) VRAM (128gb) machine sounds great for a hobbyist assuming you can easily run and train llama, Stable Diffusion and similar vram-hungry projects.
- aurareturn 2y agoNvidia completely skipped a middle tier. DGX Spark is very underpowered for $3,000. DGX Station is likely $150k - $200k. There is no middle tier at all. A middle tier could be something like an RTX 5090-like chip connected to 128GB - 256GB of GDDR7 RAM for $15k - $30k.
- yakz 2y agoWell, there is the RTX PRO 6000. Maybe you could double those up, no NVLink though.
- aurareturn 2y agoMemory bandwidth is quite low for the price. $3,000 for 128GB @ 273GB/s. You can buy a Mac Studio with an M4 Max for $3,500. 128GB unified memory @ 546 GB/s. You're also getting a much faster CPU and more usable daily computer. I suppose if you're a CUDA developer, this thing is probably better though I doubt you'd be training anything worthwhile on a computer this weak. Nvidia advertises the DGX Spark as a machine that mimics very large DGX clusters so the environment is the same. But in terms of hardware specs, it's very disappointing for $3,000. DGX Station is another beast. It's Blackwell Ultra with 288GB HBM3e and 496GB LPDDR5X. I'm guessing $150k - $200k.
- ekianjo 2y agoMac Studio is virtually useless if you work with long contexts. You will end up with having to wait minutes before the first token comes out.
- strongpigeon 2y agoAs someone who doesn’t know that much about AI performance, why is that?
- cma 2y agoNext token is mostly bandwidth bound, prefill/ingest can process tokens in parallel and starts becoming more compute heavy. Next token(s) with speculative decode/draft model also becomes compute heavy since it processes several in parallel and only rolls back on mispredict.
- dumbmrblah 2y agoTime point 1 is processing all the tokens from the original prompt. Time point 2 is replying.
- fancyfredbot 2y agoThe M series CPUs have very good memory bandwidth and capacity which lets them load in the billions of weights of a large LLM quickly. Because the bottleneck to producing a single token is typically the time taken to get the weights into the FPU macs perform very well at producing additional tokens. Producing the first token means processing the entire prompt first. With the prompt you don't need to process one token before moving on to the next because they are all given to you at once. That means loading the weights into the FPU onlu once for the entire prompt, rather than once for every token. That means the bottleneck isn't the time to get the weights to the FPU, it's the time taken to process the tokens. Macs have comparatively low compute performance (M4 Max runs at about 1/4 the FP16 speed of the small nvidia box in this article, which itself is roughly 1/4 the speed of a 5090 GPU).
- fancyfredbot 2y agoSome discussion of this box here: https://news.ycombinator.com/item?id=43425935 https://news.ycombinator.com/item?id=43425935 TLDR, AMD machines based on HX395 offer comparable memory bandwidth and size at lower cost ($2000) but lack the high speed networking and compute power (60TF FP16 on AMD and 120TF on NVIDIA) Apple M4 Max costs 23% more with equivalent memory, has twice the memory bandwidth but again no fast networking and significantly less compute (30TF FP16 on M4 Max). For inference on large language models the extra memory bandwidth probably means M4 Max is the fastest option unless you use large batches or long contexts. For training large models needing more than 128GB RAM two of these Nvidia boxes is probably fastest.
- a2128 2y agoAnother thing lacking on AMD machines is the software support. Not just that a lot of stuff is CUDA-only still, but AMD themselves struggle to support their own hardware for ROCm. According to their docs ROCm for HX395 is supported on Windows[0] but not Linux[1]. Though further reading indicates the Windows version is also just a subset of Linux ROCm that seems to not support PyTorch or anything. I really don't know what they're doing over there but I hate it. [0] https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-windows/en/lat... [1] https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/lates...
- fancyfredbot 2y agoIt's a huge mess. I think people are running LLMs on HX395 using the Vulkan backend in llama.cpp which will work on Linux. No RocM or pytorch though.
- daft_pink 2y agoAm I the only one who became curious only to become more interested in the larger DGX station workstation model they are offering with an unaccounced price tag at the time?
- KeplerBoy 2y agoNo, the station is simply unobtainable for mortals.
- nashashmi 2y agoWhere is that tech promise of SSDs as fast as RAM? Seems like a good time for it to go to production.
- bee_rider 2y ago3D Xpoint aka Optane? It never really caught on and has since been discontinued…
- EarthLaunch 2y agoStill one of my fastest drives, and extremely durable. It's too bad.
- deleted 2y ago[deleted]
- rubatuga 2y agoGood for training, definitely a bad idea for inference. But if you are spending that much money, why not just buy the equivalent of GPUs? You could buy 10 12GB 3060s for that price.
- evilduck 2y agoPowering ten 3060's and having a computer that can accept ten GPUs becomes a non-negligible hurdle to overcome.
- bee_rider 2y agoFor LLM developers, is there really no advantage to having a big block of unified memory, rather than a bunch of devices with a small amount of memory each?
- bick_nyers 2y agoMoE inference wouldn't be terrible. That being said, there's not a good MoE model in the 70-160B range as far as I'm aware.
- traspler 2y agoWhy do all articles say "with 784 GB of unified memory" for the Station when on the official spec page it's not listed as unified but as "GPU Memory: Up to 288GB HBM3e | 8 TB/s" and "CPU Memory: Up to 496GB LPDDR5X | Up to 396 GB/s"?
- henning 2y agoAs with buying consumer GPUs from Nvidia now, these prices mean nothing, because it will be impossible to buy one of these at MSRP. The actual price will be 2-3x this from some scalper who has a giant pile of them while people who want to do actual work with them pay through the nose. That's assuming the connectors don't catch on fire, no cores are missing the way ROPs have been missing with 5090s, etc.
- kristjansson 2y agoThey’ve offered reservations that, when selected, give the buyer a 4 day option to purchase. The reservations are accessible enough I was able to grab one without much effort So would seem like they’re managing the allocations pretty carefully, or at least trying harder than they have before.
- dark-star 2y agoToo bad it won't be available at scale. We had a customer who asked us if he could get 200 of those boxes for his AI scientists. But they're apparently only sold in single quantities on the NVidia store or something, not through the official distribution channels
- haunter 2y agoIs it possible to install the DGX stack on other distros besides Ubuntu? Nvidia say it's possible but does anyone actually did it? >You also have the option to install the NVIDIA DGX Software Stack on a regular Ubuntu 22.04 while still benefiting from the advanced DGX features. This installation method supports more flexibility, such as custom partition schemes. https://docs.nvidia.com/dgx/dgx-os-6-user-guide/introduction.html https://docs.nvidia.com/dgx/dgx-os-6-user-guide/introduction...