18 ms·
Nvidia releases new AI chip with 480GB CPU RAM, 96GB GPU RAM
- sylware 3y ago... and still so small and slow compared to a human brain.
- mschuster91 3y agoGive development a few years and it'll be faster than a human.
- jacquesm 3y agoDepending on the task it may well already be faster than a human at equivalent performance.
- MrBuddyCasino 3y agoNeurons are very complex and not just on/off switches. We can't even fully simulate the ~300 neurons of c. elegans, not even close.
- TeMPOraL 3y agoA lot of that complexity comes from them being living cells, optimized for and functioning in a different environment than our silicon-based machines. We don't need to model it all. (Though we do need to pay attention to evolution cheating by overfitting relative to what we'd consider a clean design. Some of the complexity may be doing double duty.)
- MrBuddyCasino 3y ago> Some of the complexity may be doing double duty. Since we have not succeeded in imitating even the most primitive brains, even though computationally we should have enough juice by now, it would seem that complexity can't be discarded at all, no?
- TeMPOraL 3y ago> Since we have not succeeded in imitating even the most primitive brains, even though computationally we should have enough juice by now looks at the browser tab with GPT-4 in it looks back again here ... we didn't?
- AstralStorm 3y agoWe didn't. Try to get ChatGPT through a maze a C. Elegans can solve in minutes. (That slow mostly because pond scum moves slowly.) Seriously, get it to successfully play through a text adventure maze game.
- mschuster91 3y agoThe problem with ChatGPT is it lacks the context depth for more complex tasks - but that is a computation resource limit, not a technical one.
- TeMPOraL 3y ago> Seriously, get it to successfully play through a text adventure maze game. I did exactly that the other day in response to a different objection on a HN thread. Or at least similar enough. https://cloud.typingmind.com/share/c0a68cb2-5f59-4e83-b383-bcd7aad497f2 https://cloud.typingmind.com/share/c0a68cb2-5f59-4e83-b383-b... Now, the goal there wasn't to get it to solve a maze, but rather to see how it can come up with a plan of action and adjust it on the fly. But I see no reason a variant of that wouldn't work with a traditional maze game - provided you remember this is a stateless model without volatile memory, so it needs to be fed its memory with every request.
- MrBuddyCasino 3y agoNo we haven't. ChatGPT is basically a sophisticated Markov chain. It is very good at pattern matching, but it has no understanding of anything, or its own will. People who think is even close to AGI are deluded, fooled by an elaborate Mechanical Turk. This is also the reason why its output sounds convincing, but is very often factually wrong.
- mrguyorama 3y agoSlime molds and single celled creatures can learn things despite having ZERO neurons. Neurons are built on top of an already incredibly complex machine evaluating hundreds of thousands of chemical and physical interactions per second that ALL effect how the cell works. We aren't likely ever going to reduce that to a model as simple as the one used in machine learning, because it probably isn't that simple period. Neurons are not "just" electrical signalling devices. They are complicated processors and systems in their own right.
- KeplerBoy 3y agoI don't know about you guys but computers have always been faster at math than my brain.
- hardware2win 3y agoHow about learning e.g car driving?
- firecall 3y agoIt's just a matter of knowing how to teach it...
- ed25519FUUU 3y agoWhy not have it teach itself if it’s so smart!
- echelon 3y agoIt took humans 300,000 years to learn to drive the car. Much longer than that, if you consider our lineage.
- hardware2win 3y ago>It took humans 300,000 years to learn to drive the car. False, to invent car. People manage to learn to operate cars in less than 50 hours
- detrites 3y agoYou're comparing training a model from scratch in ML, to the equivalent of model fine-tuning in humans. It's unfair and incorrect. Eg, no human can drive without first learning to operate their limbs, recognise shapes etc. The GP is pointing out that training in fine muscle motor skills, self-awareness and ability to project self-awareness to other objects under ones control etc, all took many thousands of years to develop. AI is faster. However, it's again unfair, as AI only knows what it knows from us, so in that sense any comparison is built on shaky ground. But for the purposes of comparing a stock human brain as hardware, versus a current high-end GPU specifically in terms of ingesting information and then perform tasks, the GPU beats the human brain "hands-down" in any category. The only categories it doesn't are simply ones that no one has trained it to yet - so the argument on a pure hardware capability basis stands.
- dauertewigkeit 3y agoI would like to know how fast this is compared to a model shared on 3 A100s with 32 GBs of VRAM each. What I am also interested in, is why model sharding has to be done manually. It seems like, one should be able to write a framework that will take your forward step and distribute the amount of layers on the available GPUs, automatically. But I haven't come across such a framework yet.
- ttul 3y agoJax has auto-sharding: https://jax.readthedocs.io/en/latest/notebooks/Distributed_arrays_and_automatic_parallelization.html https://jax.readthedocs.io/en/latest/notebooks/Distributed_a...
- simon_acca 3y ago> take your forward step and distribute the amount of layers on the available GPUs, automatically This is part of the value proposition of Mojo, Chris Lattner’s latest project. The compiler infrastructure is still in its infancy, but looks promising: https://www.modular.com/mojo https://www.modular.com/mojo
- josephg 3y agoProbably much faster. PCIe is the limiting factor in a 3x A100 setup. Having all that memory directly accessible by the GPU will make a massive difference. Even the massive 480 gigs of cpu memory in this architecture can be accessed many times faster than pcie allows.
- touisteur 3y agoHopefully if you have 3x A100 you have bridged them with NVLink and that's your bottleneck. If you want more bandwith you'll need a NVSwitch. Or a H100 GPU (Lovelace didn't get NVLink, not even the A40-little-NVLink-port - or PCIe5 for that matter) with next gen NVLink.
- yeahwhatever10 3y agoDeepSpeed?
- tmikaeld 3y agoHm, that's a heft price, 200K per card. Mac Studio M2 ULTRA has 192Gb of RAM, potentially 188Gb available for GPU, for 5K. Wouldn't apple be able to compete with that if they scaled it up?
- ChuckNorris89 3y agoGo ahead and ask datacenters why they all use overpriced Nvidia chips for AI training instead of shoving cheaper Mac Studios in there. Their answer might blow your mind. Spoiler alert: CUDA ecosystem, Linux suport, and most importantly for data centers, Mellanox high speed interfacing with virtually infinite scalability and great virtualization support so they can rent out slices of their HW to customers in exchange for money.
- edf13 3y agoWhat is the answer?
- throwaway485 3y agoI am speculating the answer is that "Nvidia just works", where Apple may be more niche & hassle to get working with their preferred frameworks/stacks/tools.
- less_less 3y agoMaybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops). By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU. So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.
- washadjeffmad 3y agoWhat's the form factor, just two fused SXM5s?
- schlupfknoten 3y agoWhat would something like that cost?
- laweijfmvo 3y ago$200,000
- drtgh 3y agoI was going to talk about the power consumption ranging from 450W to 1000W (expensive), but I guess with such price level it will not mind at all.
- seydor 3y agoLaptops should have this in a year
- Hamuko 3y ago>Module thermal design power (TDP): Programmable from 450W to 1000W (CPU + GPU + memory) I don't think so Tim.
- ed25519FUUU 3y agoNothing like programming with a 1 kW power plant on your crotch.
- krylon 3y agoThey could sell it as a contraceptive for men. (SCNR)
- z29LiTp5qUC30n 3y agoYeah, that was like trying to program when my ex-wife wanted attention. Just wish I could have typed half as well as Hugh Jackman in Swordfish; then I wouldn't feel so sorry for all of the bugs I introduced into production because of her.
- seydor 3y agoUnfoldable solar panel. Apple said yesterday that sitting in the sun is good for your eyes
- doubleorseven 3y agoYes, but will it blend?
- pixl97 3y agoOnly if you want your lap to catch on fire.
- NickHoff 3y agoMy question here is about underlying fab capacity. This chip is made on TSMC 4N, along with the H100 and 40xx series consumer GPUs. I assume Nvidia has purchased their entire production capacity. I also assume that Nvidia is using that capacity to produce the products with the highest margins, which probably means the H100 and this new GH200. So when they release this new chip, does it mean effectively fewer H100s and 4090s? Or is that not how fabrication capacity works? I'm asking because whenever I look at ML training in the cloud, I never see any availability - either for this architecture or the A100s. AWS and GCP have quotas set to 0, lambda labs is usually sold out, paperspace has no capacity, etc. What we need isn't faster or bigger GPUs, it's _more_ GPUs.
- huijzer 3y agoIt sounds to me like the GH200 achieves more FLOPS per transistor. So, compute demand will be quicker satisfied via the GH200 than via "smaller" chips such as the H100. Having said that, I don’t think we’re anywhere near some kind of equilibrium for AI compute. If chip supply would magically double tomorrow, then the large companies would buy it for their datacenters and have 100% utilization in a few weeks. They all want to train larger models and scale inference to more users.
- rcme 3y agoIn addition to training larger models, I'm sure there are many use cases that AI could serve that are currently cost prohibitive due to the cost of running inference.
- Tepix 3y agoI see A100 80GB cloud capacity available on both runpod.io and vast.ai currently.
- fxtentacle 3y agoI believe availability is low because the GPUs are too expensive so those that need to scale up use the older and much more affordable models.
- 3y ago
- samwillis 3y agoThis may be a naive question, in "crypto" we saw a shift from GPUs to ASICs as it was more efficient to design and run chips specifically for hashing. Will we see the same in ML, will there be a shift to ASICs for training and inference of models? Apple already have the "neural" cores, is that more or less what they are? Could there be a theoretical LLM chip for inference that is significantly cheaper to run?
- nynx 3y agoThese are “GPUs” in a pretty stretched sense. They have a lot of custom logic specifically for doing tensor operations.
- cubefox 3y agoWhich isn't very informative when it isn't clear how close they are to TPUs, which are ASICs.
- PUSH_AX 3y agoI think this is already a thing. https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit
- BobbyJo 3y agoThat shift won't happen until research slows down. Nobody wants to invest significant amounts of money in hardware that will be obsolete in a year.
- voxadam 3y ago> Nobody wants to invest significant amounts of money in hardware that will be obsolete in a year. When talking about the current ML industry it's more like nobody wants to invest significant amounts of money in hardware that will be obsolete before it's even taped out.
- kramerger 3y ago
- bilsbie 3y agoWhy does it have cpu ram?
- Synaesthesia 3y agoBecause it's still got separate Ram for CPU and GPU which is how most PC's wor. It's not an integrated circuit which uses shared memory.
- bilsbie 3y agoIsn’t this a gpu card though? Should the cpu and system ram be separate?
- throwaway290 3y ago> Global hyperscalers and supercomputing centers in Europe and the U.S. are among several customers that will have access to GH200-powered systems. So, not for everyone.
- keyme 3y agoIf you had a machine that prints money, you wouldn't sell it, would you? You would run it yourself. It's the same case as with high-end electronic components since 10-15 years ago. No one can/could produce their own smartphone because the high end components are sold only to the largest incumbents. A medium-size startup "has the money" to buy these, even in volume. But qualcomm won't sell. Same fate expects us here in general purpose computing. Hard to believe, but so was what happened to the electronics supply chain, at the time.
- fennecfoxy 3y agoWell there are restrictions on certain countries buying systems this powerful with the idea that they might be used for bad things. Granted, we can use them for bad things, too, but nobody wants to get stabbed with their own knife.
- Aardwolf 3y agoWhat kind of CPU is the CPU part? The link doesn't tell. Is it like ARM or RISC V and can it run a general purpose OS like Linux? Do you plugin in DDR memory somewhere for the 480GB, or is this already on the board? EDIT: found answer to my own question in the datasheet: "The NVIDIA Grace CPU combines 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory with ECC."
- tromp 3y agoQuoting from https://www.nvidia.com/en-us/data-center/grace-cpu-superchip/ https://www.nvidia.com/en-us/data-center/grace-cpu-superchip... > The NVIDIA Grace CPU Superchip uses the NVIDIA® NVLink®-C2C technology to deliver 144 Arm® Neoverse V2 cores and 1 terabyte per second (TB/s) of memory bandwidth. > High-performance CPU for HPC and cloud computing Superchip design with up to 144 Arm Neoverse V2 CPU cores with Scalable Vector Extensions (SVE2) > World’s first LPDDR5X with error-correcting code (ECC) memory, 1TB/s total bandwidth > 900 gigabyte per second (GB/s) coherent interface, 7X faster than PCIe Gen 5 > NVIDIA Scalable Coherency Fabric with 3.2TB/s of aggregate bisectional bandwidth > 2X the packaging density of DIMM-based solutions > 2X the performance per watt of today’s leading CPU
- jabl 3y agoThis is about the "Grace Hopper Superchip", which has one of the Grace CPU's replaced with a GPU. Thus, 72 Neoverse V2 cores. https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...
- greggsy 3y ago72 Neoverse V2 Armv9 cores. Not sure how one interfaces with it, but it presumably runs an approved Linux distro, with a web server at best.
- whatisyour 3y agoit's a normal chip like your x64 chip.you install ARM variant of your linux distribution on it and run it natively. source: I have one
- bingdig 3y ago> Grace™ Hopper™ Can anyone with more legal knowledge share how they trademarked the name of Grace Hopper?
- adolph 3y agoIt looks like two separate trademarks. https://en.wikipedia.org/wiki/Salami_slicing_tactics https://en.wikipedia.org/wiki/Salami_slicing_tactics
- jabl 3y agoI think it's more due to this device combines a "Grace" CPU and a "Hopper" GPU, thus creating a "Grace Hopper" superchip.
- mk_stjames 3y agoSomething to do with Grace Hopper being an actual person (although she is deceased) and thus not being to trademark the entire name?
- jabl 3y agoHopper is the name of the GPU architecture, and Grace is the name of the CPU. Combining them in a device gets you a "Grace Hopper" superchip. (And yes, I'd guess the codenames where chosen back in the day with an eye towards combining them in the same device.)
- jsheard 3y agoThere's one funny exception to the rule of Nvidia naming their GPU architectures after just the surname of a famous scientist - they always refer to the "Ada Lovelace" architecture using her full name, presumably to avoid association with the other famous Lovelace.
- ConceptJunkie 3y agoI think it would be interesting to compare this with the Apple BHA. Although in this case, Admiral Hopper's family would have to take up the fight.
- ulrikhansen54 3y agoMore powerful chips are great, but NVIDIA really ought to focus some of their best folks on ironing out some of the quirks of using their CUDA software and actually getting stuff to run on their hardware in a simpler manner. Anyone who's ever fiddled with various CUDA device drivers and lining up PyTorch & Python versions will understand the pain.
- omgJustTest 3y agoHardware before software!
- paulryanrogers 3y agoATI/AMD GPUs supposedly have great hardware, hamstrung by less-than-great software. In fact it's the lack of some software features making me hesitate to switch despite major cost savings.
- kapperchino 3y agoIn fact the amd fine wine is just them fixing their drivers from launch
- mrguyorama 3y agoAMD drivers are fine if you only care about gaming. There's the occasional idiocy like the default fan curve for my graphics card refusing to run higher than 70% so that the card will cook itself if you actually use it and hard crash your system or the driver, but eh. The real problem is that ROCm is a fucking joke, pathetic, half assed, pretend project. Nobody with power in AMD seems to care that nobody can learn machine learning on their hardware to push it in other places, or that their GPUs that they have recently spent all this time boasting about their higher VRAM which is literally useless unless you want to play poorly optimized AAA titles ported from the PS5. People say it works but you basically have to be one of the engineers who wrote it to prove that. Good luck getting it to work with Windows, or any hardware that wasn't purpose built for a cluster partner. It's so stupid. Maybe they genuinely intended to make a real CUDA competitor but noticed the ways that nVidia then had to artificially segment their market through dumb decisions (the VRAM) and bios hacks that didn't work and just gave up on that path.
- fulafel 3y agoWhat's a superchip? Is it their marketing term for MCM (multi-chip module)?
- wmf 3y agoMore or less yes.
- kristopolous 3y agoThey also have a turnkey product with 256 of these things. 1 exaflop + 144TB memory https://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh200-ai-supercomputer https://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh20...
- bigyikes 3y agoI watched Jensen’s announcement for this. He calls it the worlds largest GPU. It’s just one, giant compute unit. Unlike super computers, which are highly distributed, Nvidia says this is 140 TERABYTES of UNIFIED MEMORY. My mind still gets blown just thinking about it. My poor desktop GPU has 4 gigabytes of memory. Heck, it only has 2 terabytes of storage!
- markus_zhang 3y agoIs it similar to the mainframe in concept?
- tiberious726 3y agoNo, kinda the opposite: mainframes use a litany of sophisticated and special purpose hardware to achieve their tasks (eg hardware io _channels_), while this is a massively overgrown instance of a single kind of hardware (vector processor).
- bitwize 3y agoThe thing that made the lightbulb go off in my head w.r.t. mainframes was understanding that mainframe I/O channels are computers. The mainframe had several dedicated computers that each specifically handled I/O to a terminal, printer, disk or tape drive, punchcard reader, etc. Made I/O programming a breeze, as you just had to tell the channel to read or write, specifying a block of memory to use as a buffer, and the channel would DMA out the data to be written, or DMA in read-in data. It also explains the reason why despite having middling CPU power, mainframes had a reputation for stinkloads of I/O bandwidth so they could process everyone's credit card transactions, airline bookings, and that: the mainframe's CPU was involved very little in I/O, that was all handled by the channel processors!
- tppiotrowski 3y agoKind of curious what WebGL report [1] would be for one of these devices. Does the extra RAM make maximum texture sizes massive and allow for thousands of textures or is that a software limitation? [1] https://webglreport.com/ https://webglreport.com/
- deleted 3y ago[deleted]
- gitcommitco 3y ago[flagged]
- tiffanyh 3y agoDumb questions ... - Am I wrong in understanding this is a general purpose computer (with massive graphic capabilities)? - And if so, what CPU is it using (an NVIDIA ARM CPU)? - And what OS does it run?
- zucker42 3y agoCorrect, it's called the Grace Hopper superchip because it using Nvidia Grace CPU (which are ARM) and Nvidia Hopper GPU. For OS, it will run some form of Linux. I'm not sure if the particular recommended build has been (or will be) publicly released.
- indymike 3y agoIs there some connection between Nvidia and Admiral Hopper's family that makes it ok to appropriate her identity for their product?
- vinay427 3y agoAs far as I can tell they have frequently used the names of famous scientists, such as Kepler, Fermi, Maxwell, Pascal, Turing, Ampere, and Ada Lovelace. This has existed long before Hopper.
- justinclift 3y ago> appropriate her identity Hmmm, what's the difference between homage and appropriation for things like this?
- indymike 3y agoClaiming a trademark.
- justinclift 3y agoThat's a good point. If they're trying to trademark the name, that sounds like it'd raise some ethical issues. :)
- tmaly 3y agoCan I get one of these for my desktop so I can run the 65B models?