21 ms·
Nvidia Unveils Grace: A High-Performance Arm CPU for Use in Big AI Systems
- remexre 6y ago> Today at GTC 2021 NVIDIA announces its first CPU Wait, Nvidia's been making ARM CPUs for years now; most memorably Project Denver.
- 1MachineElf 6y agoI wonder what percentage of it's supported toolchain components will be proprietary.
- crb002 6y ago+1 ECC RAM
- cma 6y agoReal business-class features we want to know about: Will they auto-detect workloads and cripple performance (like the mining stuff recently)? Only work through special drivers with extra licensing feeds depending on the name of the building it is in (data center vs office)?
- rubatuga 6y agoMarket segmentation is practiced by every chip company that you use. Intel: ECC. AMD: ROCM. Qualcomm: cost as percentage of the phone price.
- cma 6y agoI still think Nvidia takes it further.
- volta83 6y agoEvery company does market segmentation: it makes sense to have customers that want a feature pay more for it. Still, every company does it differently. For example, both NVIDIA and AMD compute GPUs are necessarily more expensive than gamer GPUs because of hardware costs (e.g. HBM). However, NVIDIA gamer GPUs can do CUDA, while AMD gamer GPUs can't do ROCm. The reason is that NVIDIA has 1 architecture for gaming and compute (Ampere), while AMD has two different architectures (RDNA and CDNA).
- valine 6y agoI like the sound of a non-Apple arm chip for workstations. Given my positive experience with the M1 I'd be perfectly happy never using x86 again after this market niche is filled.
- awill 6y agoMe too. But my decades old steam collection isn't looking forward to it. That's one advantage of cloud gaming. It won't matter what your desktop runs on.
- webaholic 6y agoI don't think this will be anywhere near as good as the M1, since they are using the ARM Neoverse cores.
- ac29 6y agoApple throws a lot of transistors at their 4 performance cores in the M1 to get the performance they do - its not clear that approach would realistically scale to a workstation CPU with 16, 32, or more cores (at least not with current fab capabilities).
- gchadwick 6y agoIt'd be interesting to know if NVidia are going for an ARMv9 core, in particular if they'll have a core with an SVE2 implementation. It may be they don't want to detract from focus on the GPUs for vector computation so prefer a CPU without much vector muscle. Also interesting that they're picking up an arm core rather than continuing with their own design. Something to do with the potential takeover (the merged company would only want to support so many micro-architectural lines)?
- klelatti 6y agoThis has got me wondering whether an Nvidia owned Arm could limit SVE2 implementations so as not to compete with Nvidia's GPU. That would certainly be the case for Arm designed cores - not a desirable outcome.
- MikeCapone 6y agoI doubt it, it's not like the market for acceleration is stagnant and saturated and they need to steal some marketshare points from one side to help the other. It's all greenfield and growing so far, they'll win more by having the very best products they can make on both sides.
- mlyle 6y agoYou'd think. But it wouldn't be the first time a new product is hampered to not slightly theoretically cannibalize an existing product family.
- adrian_b 6y agoThey have said clearly that the core is licensed from ARM and one of the Neoverse future models. There was no information whether it will have any good SVE2 implementation. On the contrary they insisted only on the integer performance and on the high-speed memory interface.
- dragontamer 6y ago
- lprd 6y agoSo is ARM the future at this point? After seeing how well Apple's M1 performed against a traditional AMD/Intel CPU, it has me wondering. I used to think that ARM was really only suited for smaller devices.
- Hamuko 6y agoBased on initial testing, AWS EC2 instances with ARM chips performed as well if not better than the Intel instances, but they cost 20% less. The only drawback that I've really encountered thus far was that it complicates the build process.
- moistbar 6y agoDoes ARM have a uniquely complex build process, or is it the mix of architectures that makes it more difficult?
- Hamuko 6y agoMix of architectures and the fact that our normal CI server is still x86-based and really didn't want to do ARM builds.
- sumtechguy 6y agoARM is all over the place with its ISA. x86 has the benefit that most companies made it 'IBM compatible'. There are one off x86 ISAs but they are mostly forgotten at this point. The ARM CPU family itself is fairly consistent (mostly), but included hardware is a very mixed bag. The x86 has on the other hand the history of build it to make it work like IBM. All the way from how things boot up, memory space addresses, must have I/O, etc. ARM on the other hand may or may not have that depending on which ISA you target or are creating. Things like the raspberry PI has changed some of that as many are mimicking the broadcom ISA and specifically that with the raspberry pi one. The x86 arch has also picked up some interesting baggage along the way because of what it is. We can mostly ignore it but it is there. For example you would not build a ARM board these days with an IDE interface but some of those bits still exist in the x86 world. ARM is more of a tool kit to build different purpose built computers (you even see them show up in usb sticks). While x86 is particular ISA that has a long history behind it. So you may see something like 'Amazon builds its own ARM computers'. That means they spun their own boards, built their own toolchains (more likely recompiled existing ones), and probably have their own OS distro to match. Each one of those is a fairly large endeavor to do. When you see something like 'Amazon builds its own x86 boards', they have shaved out the other two parts of that and are focusing on hardware. That they are building their own means they see the value in owning the whole stack. Also if you have your own distro means you usually have to 'own' building the whole thing. So I can go grab an x86 gcc stack from my repo provider. They will need to act as the repo owner and build it themselves and keep up with the patches. Depending on what has been added that can be quite the task all by itself.
- de6u99er 6y agoDon't know if it's just me but this product looks like a beta-product for early adopters.
- rektide 6y agoIt's initially for two huge HPC systems. It'll be interesting to see what kind of availability it ever has to the rest of the world.
- lprd 6y agoSo is ARM the future at this point? After seeing how well Apple's M1 performed against a traditional AMD/Intel CPU, it has me wondering. I used to think that ARM was really only suited for smaller devices.
- fulafel 6y agoThe instruction set doesn't make a significant difference technically, the main things about them are monopolies (patents) tied to ISAs, and sw compatibility.
- rvanlaar 6y agoI'm interested in your thoughts on why this doesn't make a significant difference. From what I've read, the M1 has a lot of tricks up its sleeve that are next to impossible on X86. For example ARM instructions can be decoded in parallel.
- fulafel 6y agoInstruction decoding is more power efficient on arm, but x86 has solved it as a perf bottleneck, with the trace/uop caches and by doing some speculative work in the decoders. (Parallel decoding is also old hat and not a M1 or ARM land invention, it's trivial with RISC style insn format.). What other tricks do you have in mind? More broadly, as to why the ISA doesn't make a big difference: The major differences are at the microarchitecture level since OoO processors have such flexible dataflow machinery in them that you can kind of view the frontend as compiler technology. x86 and ARM are decades-old ISAs that have seen a many many rounds of iteration in form of added instructions and even backwards incompatible reboots at the 64-bit transition points so most hinderances have been fixed. In the olden days ISAs were important because processors were orders of magniture simpler, and instructions were processed as-is very statically (to the point that microarchitectural artifacts like branch delay slots were enshrined in some ISAs). This meant that eg the complexity of individual instructions could a bottleneck to how fast a chip could be clocked. Or in CISC land your ISA might have been so complex that the CPU was a microcoded implementation of the ISA and didn't have any hardwired fast instructions...
- callesgg 6y agoSuper parallell arm chips could that not be a future thing for nvidia or another chip manufacturer. A normal CPU die with thousands of independent Cores.
- astrange 6y agoThat's Xeon Phi (formerly known as Larrabee) but in general this isn't that useful. Or rather, when it is useful it's called a GPU.
- api 6y agoTangent: Apple should bring back the Xserve with their M1 line, or alternately license the M1 core IP to another company to produce a differently-branded server-oriented chip. The performance of that thing is mind blowing and I don't see how this would compete with or harm their desktop and mobile business.
- AnthonyMouse 6y agoThe cheapest available Epyc (7313P) has 16 cores and dual socket systems have up to 128 cores and 256 threads. Server workloads are massively parallel, so a 4+4 core M1 would be embarrassed and Apple wouldn't want to subject themselves to that comparison. But another reason they won't do it is that TSMC has a finite amount of 5nm fab capacity. They can't make more of the chips than they already do.
- api 6y agoI'm thinking of a 64-core M1. It would not be the laptop chip.
- ac29 6y agoA 4+4 core M1 is 16 billion transistors. Some of that is the little cores, GPU, etc, but its not clear to me its practical to get, say 8x larger. That would be 128 billion transistors. As a point of comparison, NVIDIA's RTX 3090 is 28B transistors, and thats a huge, expensive chip.
- bombcar 6y agoHow much of that performance is on-chip memory and how usable/scalable is that? An Xserve that is limited to one CPU and can't have more RAM would pretty mediocre.
- _ph_ 6y agoI am also hoping for a return of the Xserve once Apple makes high-corecount variations of the Apple Silicon for the Mac Pro. This would have several benefits. First of all, it would greatly increase the production count of that variant, it could be too expensive to make such a chip just for the Mac Pro. In any case, it should be cheaper than an equivalent Intel CPU as Apple would not have to pay for Intels profits. And finally, just the power savings for the vast compute centers Apple operates should mean a lot of money saved too.
- deleted 6y ago[deleted]
- rektide 6y agoThere's a lot of interconnects (CCIX, CXL, OpenCAPI, NVLink, GenZ) brewing. Nvidia going big is, hopefully, a move that will prompt some uptake from the other chip makers. 900GBps link, more than main memory: big numbers there. Side note, I miss AMD being actively involved with interconnects. InfinityFabric seems core to everything they are doing, but back in the HyperTransport days it was something known, that folks could build products for, interoperate with. Not many did, but it's still frustrating seeing AMD keeping cards so much closer to the chest.
- deleted 6y ago[deleted]
- rektide 6y agolot of downvotes. anyone want to say any reason why they think this deserves a downvote? very unclear to me. do you all just not have the historical context? what's wrong here? give me some hints why you don't get what i'm saying here.
- pezezin 6y agoThat's something weird I have noticed about HN, sometimes perfectly reasonable comments are downvoted to hell without any reply. At least in the good ol' Slashdot days you will get the reason why you got downvoted, now... nothing.
- Aissen 6y agoGPU-to-CPU interface >900GB/sec NVLink 4. What kind of interconnect is that ? Is that even physically realistic ?
- freeone3000 6y agoDepends on how big you want to make it. If they're willing to go four inches, that'd do it with existing per-pin speeds from NVLink 3.
- rincebrain 6y agoWell, according to [1], NVIDIA lists NVLink 3.0 as being 50 Gb/s per lane per direction, and lists the total maximum bandwidth of NVSwitch for Ampere (using NVLink 3.0) as 900 GB/s each direction, so it doesn't seem completely out of reach. [1] - https://en.wikipedia.org/wiki/NVLink https://en.wikipedia.org/wiki/NVLink
- Aissen 6y agoWith 50Gb/s per lane, that would be 144 lanes to reach 900GB/s. Quite impressive.
- rincebrain 6y agoFascinatingly, NVIDIA's own docs [1] claim GPU<->GPU bandwidth on that device of 600 GB/s (though they claim total aggregate bandwidth of 9.6 TB/s). Which would be what, 96 and 1536 lanes, respectively? That's quite the pinout. [1] - https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/
- deleted 6y ago[deleted]
- robomartin 6y agoWell, PCIe 6 x16 will do 128 GB/s. Of course, the real question is how many transactions per second you get. For the PCIe 6 16 lanes it's about 64 GT/s. Speaking in general terms, data rate and transaction rate don't necessarily match because a transaction might require the transmitter to wait for the receiver to check packet integrity and then issue acknowledgement to the transmitter before a new packet can be sent. Yet another case, again, speaking in general terms, would be the case of having to insert wait states to deal with memory access or other processor architecture issues. Simple example, on the STM32 processor you cannot toggle I/O in software at anywhere close to the CPU clock rate due to architectural constraints (to include the instruction set). On a processor running at 48 MHz you can only do a max toggle rate of about 3 MHz (toggle rate = number of state transitions per second).
- modeless 6y agoI hope they make workstations. I want to see some competition for the eventual Apple Silicon Mac Pro.
- dhruvdh 6y agoThey are licensing ARM cores; which as of now cannot compete with Apple silicon. While there are using some future ARM core, and I've read rumors that future designs might try to emulate what has made Apple cores successful; we cannot say whether Apple designs will stagnate or continue to improve at current rate. There is potential for competition from Qualcomm after their Nuvia acquisition though.
- adgjlsfhk1 6y agoIt seems weird to me to say that arm cores can't compete with apple silicon given that apple doesn't own fabs. They are using arm cores on TSMC silicon (exactly the same as this).
- seabrookmx 6y ago> They are using arm cores on TSMC silicon (exactly the same as this) No the Apple Silicon chips use the arm _instruction set_ but they do not use their core design. Apple designs their core in house, much like Qualcomm does with snapdragon. Both of these companies have an architectural license which allows them to do this.
- tibbydudeza 6y agoQualcomm no longer makes their own cores - they just use ARM reference IP designs since the Kryo. That will probably change with their Nuvia acquisition.
- devmor 6y agoWhat do you mean ARM cores can't compete with Apple silicon? "Apple silicon" are ARM cores.
- GrumpyNl 6y agoI need a new video card and there are no Nvidia to buy, all is bought by miners. Will it go the same with this card?
- redtriumph 6y agoCurrently, there are no plans for consumer-grade CPUs. Even this new CPU class is shipping in 2023.
- CalChris 6y agoGrace, in contrast, is a much safer project for NVIDIA; they’re merely licensing Arm cores rather than building their own ... NVIDIA is buying ARM.
- klelatti 6y agoTrying to buy Arm. Multiple competition investigations permitting.
- TheMagicHorsey 6y agoIs anyone but Apple making big investments in ARM for the desktop? This is another ARM for the datacenter design. If other companies don't make genuine investments in ARM for the desktop there's a real chance that Apple will get a huge an difficult to assail application performance advantage as application developers begin to focus on making Mac apps first, and port to x86 as an afterthought. Something similar happened back in the day when Intel was the de facto king, and everything on other platforms was a handicapped afterthought. I wouldn't want to have my desktops be 15 to 30% slower than Macs running the same software, simply because of emulation or lack of local optimizations. So I'm really looking forward to ARM competition on the desktop.
- nabla9 6y agoFinally news from Nvidia that really moved markets. Nvidia +4.68%, Intel -4.65% AMD -4.47%
- 01100011 6y agoI wonder how permanent this is. As a Nvidian who sells his shares as soon as they vest and who owns some Intel for diversification, I wonder if I should load up on Intel? You really can't compete with their fab availability. Having a great design means nothing unless you can get TSMC to grant you production capacity.
- nabla9 6y agoTSMC takes orders years ahead and builds capacity to match working together with big customers. Those who pay more (price per unit and large volume) get first shot. That's why Apple is always first, followed by Nvidia and AMD, then Qualcomm. There is bottled demand because Intel's failure to deliver was not fully anticipated by anyone.
- DonHopkins 6y agoI love the name "Grace", after Grace Hopper.
- paulmd 6y agoThere's a tendency to use first names to refer to women in professional settings or political power that is somewhat infantilizing and demeaning. I doubt anyone really deliberately sets out to be like "haha yessss today I shall elide this woman's credentials", but this is one of those unconscious gender-bias things that is commonplace in our society and is probably best to try and make a point of avoiding. https://news.cornell.edu/stories/2018/07/when-last-comes-first-gender-bias-names https://news.cornell.edu/stories/2018/07/when-last-comes-fir... https://metro.co.uk/2018/03/04/referring-to-women-by-their-first-names-and-men-by-their-surname-is-a-form-of-gender-discrimination-7325551/ https://metro.co.uk/2018/03/04/referring-to-women-by-their-f... (etc etc) I'd prefer they used "Hopper" instead, in the same way they have chosen to refer to previous architectures by the last names of their namesakes (Maxwell, Pascal, Ampere, Volta, Kepler, Fermi, etc). I'd see that as being more professionally respectful for her contributions. But yes I very much like the idea of naming it after Hopper.
- hderms 6y agoI feel like there's a non-zero chance they named it Grace instead of Hopper so their new architecture doesn't sound like a bug or a frog or something. You could be right, though
- trynumber9 6y agoHopper was already reserved for an Nvidia GPU: https://en.wikipedia.org/wiki/Hopper_(microarchitecture) https://en.wikipedia.org/wiki/Hopper_(microarchitecture)
- paulmd 6y agoYeah, I dunno what is going on with that, I assumed that had changed if they were going to use the name "grace" for another product. I guess I'm not sure if "Hopper" refers to the product as a whole (like Tegra) and early leakers misunderstood that, or whether Hopper is the name of the microarchitecture and "Grace" is the product, or if it's changed from Hopper to Grace because they didn't like the name, or what. Otherwise it's a little awkward to have products named both "grace" and "hopper"...
- ksec 6y agoBased on Future ARM Neoverse, so basically nothing much to see here from CPU perspective, What really stands out, are those ridiculous number from its Memory system and Interconnect. CPU: LPDDR5X with ECC Memory at 500+GB/s Memory Bandwidth. ( Something Apple may dip into. R.I.P for Mac with upgradable Memory ) GPU: HBM2e at 2000 GB/s. Yes, three zeros, this is not a typo. NVLink: 500GB/s This will surely further solidify CUDA dominance. Not entirely sure how Intel's XE with OneAPI and AMD's ROCm is going to compete.
- Dylan16807 6y ago> GPU: HBM2e at 2000 GB/s. Yes, three zeros, this is not a typo. It's a good step forward but your average consumer GPU is already around a quarter to a third of that and a Radeon VII had 1000 GB/s two years ago.
- m_mueller 6y agoI think what you’re missing here is the NVLink part. The fact that you can get a small cluster of these linked up like that for 400k, all wrapped in a box, makes HPC quite a bit more accessible. Even 5 years ago, if you wanted to run a regional sized weather model at reasonable resolution, you needed to have some serious funding (say, nation states or oil / insurance companies). Nowadays you could do it with some angel investment and get one of these Nvidia boxes and just program them like they’re one GPU.
- kllrnohj 6y agoCritically it's CPU to GPU NVLink here, not the "boring" GPU to GPU NVLink that's common on Quadros. 500GB/s bandwidth between CPU & GPU massively changes when & how you can GPU accelerate things, that's a 10X difference over the status quo.
- kimixa 6y agoAlso "cpu->cpu" NVLink is interesting. Though it was my understanding that NVLink is point-to-point, and would require some massive switching system to be able to access any node in the cluster anywhere near that rate without some locality bias (IE nodes on the "first" downstream switch are faster to access and less contention)
- titzer 6y agoGiven that there are essentially no architectural details here other than bandwidth estimates, and the release timeline is in 2023, how exactly does this count as "unveiling"? Headline should read: "NVidia working on new arm chip due in two years", or something else much more bland.
- mrlento234 6y agoNot quite. CSCS supercomputing center in Switzerland have already started receiving the hardware (https://www.cscs.ch/science/computer-science-hpc/2021/cscs-director-shares-details-about-planned-successor-to-piz-daint/ https://www.cscs.ch/science/computer-science-hpc/2021/cscs-d...). Perhaps, we may see some benchmarks. To wider HPC users, it will be only available in 2023 as the article mentioned.
- titzer 6y agoHopefully some architectural details are forthcoming then! But that is not what is in this article.
- IanCutress 6y agoI suspect that's more racks of storage, not racks of compute. Nothing to suggest it's compute.
- seniorivn 6y agoas i understand it's compute, just not cpu compute, those cpu are designed to be good enough for cuda servers
- DetroitThrow 6y agoHey Ian, I love reading your posts on Anandtech, you're a fantastic technical communicator.
- bdc-hpc 6y agoThe Alps system at CSCS will have racks with different processors, to be installed in phases. CSCS has taken delivery of the first racks with AMD EPYC processors, for non-GPU workloads. CSCS will be one of the first customers to get their hands on Grace Hopper, but they will have to wait until 2023.
- deleted 6y ago[deleted]
- legulere 6y agoBig Data, Big AI, what's next? Big Bullshit?
- jhgb 6y agoNah, that's already been here for quite a while.
- alexhutcheson 6y agoThe fact that they are using a Neoverse core licensed from ARM seems to imply that there won’t be another generation for NVidia’s Denver/Carmel microarchitectures. Somewhat of a shame, because those microarchitectures were unorthodox in some ways, and it would have been interesting to see where that line of evolution would have lead. I believe this leaves Apple, ARM, Fujitsu, and Marvell as the only companies currently designing and selling cores that implement the ARM instruction set. That may drop to 3 in the next generation, since it’s not obvious that Marvell’s ThunderX3 cores are really seeing enough traction to be be worth the non-recurring engineering costs of a custom core. Are there any others?
- klelatti 6y agoDesigning but not yet selling Qualcomm / Nuvia?
- alexhutcheson 6y agoYeah will be interesting to see if and when they bring a design to market.
- intvocoder 6y agoThe ThunderX3 team is mostly gone, it's been hollowed out.
- Bluestein 6y agoThe whole combination of AI and the name gives "watched over by machines of loving grace" a whole new twist, eh?
- filereaper 6y agoLooks like NVidia broke up with POWER on IBM and made their own chip. They have interconnects from Mellanox, GPUs and their own CPUs now. I suspect the supercomputing lists will be dominated by NVidia now.
- arcanus 6y agoThat is certainly the trend. AMD is bringing Frontier online later this year, which might be the only counter to this.
- physicsguy 6y agoIBM have basically hollowed out their team, so I'd say it's IBM ditching the market more than anything... our centre would not now consider POWER even though we currently have nodes.
- rexreed 6y agoHonestly the bottom down-voted comment has it right. What AI application is actually driving demand here? What can't be accomplished now (or with reasonable expenditures) that can be accomplished by this one CPU that will be released in 2 yrs? What AI applications will need this 2 yrs from now that don't need it now? I understand the here-and-now AI applications. But this is smelling more like Big AI Hype than Big AI need.
- cracker_jacks 6y ago"640K ought to be enough for anybody."
- wmf 6y agoGPT-4 and GPT-5.
- gwern 6y agoHuang said "We expect to see multi-trillion-parameter models by next year, and 100 trillion+ parameter models by 2023". He probably knows more about what AI applications there are than you do, and spends a large chunk of the keynote discussing many applications.
- temp667 6y agoI know we are going to hear from the Apple haters soon or those that don't like what apple is doing (modular upgradeable systems going away) BUT it seems like Apple is moving in a similar direction as nvidia. Apple is also I think going to soldered on / close in RAM. Nvidia looks to be doing this two CPU / GPU / Ram all close together and it doesn't look like any upgrade options. Some thinking was that Apple was continuing to increase durability / reliability etc with their RAM move. Does anyone know requirements for the LPDDR5X type of ram mentioned here. Does this require soldering things (you obviously get lots more control if you spec chips yourself and solder on)?
- gradschoolfail 6y agoIf the next one is Jean or Ada we know they took it from a google search.
- leric 6y agoA Rosetta for windows could be a final hit to Intel