7 ms·
AMD's CDNA 3 Compute Architecture
- jdewerd 3y ago> AMD diverged their GPU architecture development into separate CDNA and RDNA lines specialized for compute and graphics respectively. Ooooh, is that why the consumer cards don't do compute? Yikes, I was hoping that was just a bit of misguided segmentation but this sounds like a high level architecture problem, like an interstate without an on ramp. Oof.
- CuriousCosmic 3y agoThey added support for high end radeon cards 2 months ago. Rocm is "eventually" coming to RDNA in general but it's a slow process which is more or less in line with how AMD has approached rocm from the start (targetting a very small subset of compute and slowly expanding it with each major version). https://www.tomshardware.com/news/amd-enables-rocm-and-pytorch-on-radeon-rx-7900-xtx https://www.tomshardware.com/news/amd-enables-rocm-and-pytor...
- mnau 3y agoThat's a good step forward. This separation is a massive barrier to entry. Making CUDA work reliably on every product was a masterstroke by NVIDIA.
- deskamess 3y agoI am getting the impression their (AMD) understanding of software/Devx is not as good as nvidia. There seems to be some sort of inertia in getting the software side moving - as if, now that they are behind, they are not willing to make any mistakes, so they hold off on doing anything decisive - instead opting for continuing the status quo mode but perhaps adding a few more resources.
- treprinum 3y agoThe issue is that RDNA and CDNA are different so when an enthusiast makes a fast RDNA code it doesn't mean it would work well on CDNA and vice-versa. Not sure why AMD had to go this route, only high-end pros will write software for CDNA and they won't get any mindshare.
- JonChesterfield 3y agoWhat do you have in mind for code that works noticeably better on one than the other?
- CuriousCosmic 3y agoThat's not accurate at all. Someone writing for the rocm platform will be writing HIP and then HIPCC will compile that down into the actual runtime targeting the given architecture. HIP is pretty platform agnostic so very little device specific optimization tends to go into idiomatic HIP.
- treprinum 3y agoThere are many low-level details in CUDA to get it working fast on a given GPU architecture that compilers don't address at all.
- DarkmSparks 3y agoI was quite surprised to find at least one of the the top500 is based on AMD gpus, it seems to me not bringing that to the consumer market was a conscious choice made some time ago.
- fomine3 3y agoSome HPC applications are specially made for the HPC, so minor API is acceptable for some HPCs. That's not the case for consumer GPGPU use case.
- slavik81 3y agoThere are eleven CDNA 2 systems in the top500. https://www.top500.org/lists/top500/list/2023/11/?page=1 https://www.top500.org/lists/top500/list/2023/11/?page=1
- DarkmSparks 3y agothank you, I hadn't looked at the exact break down, just ran in to one of them a few weeks ago. I found it surprising because I wasn't aware at the time there was any, and it's not like it isn't something I should have known.
- nolok 3y agoThey're far behind Nvidia in gaming, to they try to compete on price to get market share back. Eg the 7900 XTX competes with the 4090 in rastering but is between 30 and 50% cheaper, 7900 XT / 4080, 7800 XT / 4070 are all winning by decent amount in pricing. Frankly this focus has allowed them to catch back in a field they've behind for more than a decade now. The rise of RT and other compute tools in gaming (DLSS and co) is hurting this strategy, but their APU being on PS5 AND Xbox Series keep them in the race, game engines and game developpers need to make it work on AMD tech. With Nvidia pricing and memory choice this generation, it's the first time in a while that bying an AMD card for gaming is the better choice at almost every price point (unless you can afford the 4090).
- cogman10 3y agoLooks like the split was around 2016? Which could make sense given the state of crypto at the time. One problem that hit nvidia more than AMD is consumer cards getting vacuumed up by crypto farms. AMD making a conscious split effectively isolated their compute cards from their gamer cards. That being said, I can't imagine this is something that's been good for adoption of AMD cards for compute tasks. The wonderful thing about CUDA is you don't need a special accelerator card to develop cuda code.
- Zardoz84 3y agoif i remember correctly, AMD GPU cards was being sold very well to crypto miners. The split wasn't because that. They did, because gamming and computing having different hardware optimization requisites, and that allowed to get better and more competitive GPUs and computing cards on the market.
- wmf 3y agoYou can't really isolate crypto mining. If a gaming card is profitable to mine with, miners will buy it and use the gaming drivers.
- segfaultbuserr 3y agoCorrect. Whether or not a card has official GPGPU support is irrelevant for its use in mining, you can use pure Direct3D or OpenGL for mining if you really want it. The decision to not support customer GPGPU by AMD only hampers other useful non-mining applications. Ironically, mining is probably the most optimized GPGPU application a regular end-user can run on AMD GPUs. One reason was that GCN was a computing-first GPU, but also that miners have a strong motivation to go through the troubles of overcoming the lack of computing support by AMD. They would write the entire application in OpenCL 1.0 and optimize it to extract the last bit of performance in spite of its difficulties - the hashing kernels used for mining also tends to be much simpler than other applications, so it's feasible to do so. Meanwhile other people who want a decent programming environment would've picked CUDA without looking back.
- dotnet00 3y ago
- ColonelPhantom 3y ago> Ooooh, is that why the consumer cards don't do compute? No, not really. Sure, the consumer cards are based on a less suitable architecture, so they won't be as good, but it's not like it's not a general purpose GPU architecture anymore. The real problem is that ROCm has made the very weird technical decision of shipping GPU machine code, rather than bytecode. Nvidia has their own bytecode NVPTX, and Intel just uses SPIR-V for oneAPI. This means that ROCm support for a GPU architecture requires shipping binaries of all the libraries (such as rocBLAS, rocDNN, rocsparse, rocFFT) for all architectures. As the cherry on top, on paper one GPU family consists of different architectures! The RX 6700 XT identifies as a different 'architecture' (gfx1031) than the RX 6800 XT (gfx1030). These are 'close enough' that you can tell ROCm to use gfx1030 binaries on gfx1031 and it'll usually just work, but still. I feel like this is a large part of the reason AMD is so bad about supporting older GPUs in ROCm. It'd cost a bit of effort, sure, but I feel like the enormous packages you'd get are a bigger issue. (This is another factor that kills hobbyist ROCm, because anyone with a Pascal+ GPU can do CUDA, but ROCm support is usually limited to newer and higher end cards.)
- slavik81 3y ago> As the cherry on top, on paper one GPU family consists of different architectures! The RX 6700 XT identifies as a different 'architecture' (gfx1031) than the RX 6800 XT (gfx1030). These are 'close enough' that you can tell ROCm to use gfx1030 binaries on gfx1031 and it'll usually just work, but still. As far as I can tell, the gfx1030 and gfx1031 ISAs are identical in all but name. LLVM treats them exactly the same and all tests for the ROCm math libraries will pass when you run gfx1030 code on gfx1031 GPUs using the override. The Debian package for HIP has a patch so that if the user has a gfx1031 GPU and there are no gfx1031 code objects available, it will fall back to using gfx1030 code objects instead. When I proposed that patch upstream, it was rejected in favour of adding a new ISA that explicitly supports all the GPUs in a family. That solution is more complex and is taking longer to implement, but will more thoroughly solve that problem (as the solution Debian is using only works well when there's one ISA in the family that is a subset of all the others).
- JonChesterfield 3y agoThere was a ship-bytecode idea branded HSAIL, associated with opencl finalisers in some way. I'm not sure what killed that - would speculate that the compiler backend was annoyingly slow to run as a JIT. It's somewhat conceptually popular under the spirv branding now. Some of the ROCm libraries are partially written in assembly. Hopefully they all have C reference implementations that can be compiled for the other architectures. My guess is that the AMD strategy of only shipping libraries for some cards + open source means ROCm-ish as built by Linux distros, who are likely to build the libs for all the architectures and not just the special few, can be expected to drive people away from the official releases.
- beebeepka 3y agoI thought trolling wasn't welcome on HN
- vegabook 3y agoAMD has always been a terrible steward of ATI. They're fundamentally a hardware company (as per Lisa Su's CV) which didn't cotton on to the fact that Cuda was the killer. I remember how @Bridgman kept playing a rearguard action on Phoronix, his job being to keep devs on board. Losing battle. I kinda understand it because [80/90]s era hardware people intrinsically think hardware is the top dog in the stack, and that's where all AMD management come from including Su. Kodura understood that Nvidia was killing AMD because the consumer cards could run CUDA. So he pushed through, against Lisa Su, the Radeon VII, which until very recently, and for many years, was the only consumer card that ROCm supported. He was basically fired shortly thereafter, and the RVII, which was a fantastic card, was shutdown sharpish. Then they brought in Wang who crystallised the consumer/pro segmentation. Now they're furiously trying to backpedal, and it's too late. There are multiple attempting competitors, but basically the only one worth talking about is AAPL and Metal. AMD lost the window.
- simfree 3y agoATI on its own did not have a solid future, they could have easily ended up like their other local Canadian counterparts Nortel or Blackberry.
- vegabook 3y ago"Dead or Canadian" yes?
- hylaride 3y agoI live in Toronto and knew people who worked for ATI back in the 1990s-2000s. The management structure was fucked. In those days it was generally regarded that ATI could and did produce better hardware than NVIDIA, but ATI’s drivers were horrible and only evened out near the end of the hardware’s lifetime. But the end of the lifetime was far too late and all the benchmarks would have long been done on the buggy drivers, driving people to NVIDIA. I kid you not, but the software side of ATI’s reviews and incentives were literally QA got rewarded for finding bugs and then the software engineers were then rewarded for fixing them. You can imagine what happened next. The bitch of it is that it was overall just a symptom of the problem that the company as a whole didn’t prioritize software. NVIDIA knew that the drivers and software efficiency of using the hardware was just as if not more important than the hardware (the stories in the 1990s of how John Carmack used various hacks with hardware is a good example of that).
- sva_ 3y agoWell the APU does, there are compute intrinsics and even a small guide by AMD on how to use them: https://gpuopen.com/learn/wmma_on_rdna3/ https://gpuopen.com/learn/wmma_on_rdna3/
- pauljurczak 3y agoHow do you use these intrinsics on an APU? https://llvm.org/docs/AMDGPUUsage.html https://llvm.org/docs/AMDGPUUsage.html ?
- Const-me 3y ago> the consumer cards don't do compute? Typically, software developers only support a single GPGPU API, and that API is nVidia CUDA. Technically, consumer AMD cards are awesome at compute. For example, UE5 renders triangle meshes with compute instead of graphics https://www.youtube.com/watch?v=TMorJX3Nj6U https://www.youtube.com/watch?v=TMorJX3Nj6U Moreover, AMD cards often outperform nVidia equivalents because nVidia prioritized ray tracing and DLSS over compute power and memory bandwidth. The issue is, no tech company is interested in adding D3D or Vulkan backend to AI libraries like PyTorch. nVidia is not doing that because they are happy with the status quo. Intel and AMD are not doing that because both hope to replace CUDA with their proprietary equivalents, instead of an open GPU API.
- boppo1 3y ago>Vulkan backend to AI libraries like PyTorch How many man hours is a project like this?
- Const-me 3y agoCouple times in the past I wanted to port open source ML models from CUDA/Python to a better technology stack. I have ported Whisper https://github.com/Const-me/Whisper/ https://github.com/Const-me/Whisper/ and Mistral https://github.com/Const-me/Cgml/ https://github.com/Const-me/Cgml/ to D3D11. I don’t remember how much time I spent, but given both were unpaid part-time hobby projects, probably under 160 hours / each. These software projects were great to validate the technology choices, but note I only did bare minimum to implement specific ML models. Implementing a complete PyTorch backend gonna involve dramatically more work. I can’t even estimate how much more because I’m not an expert in Python or these Python-based ML libraries.
- versteegen 3y agoWow, a very nice reimplementation. To go on a tangent, I note your custom 'BCML1' 5bit per weight compression codec and your optimised hand-coded AVX2 to encode it... was that really needed? Are the weights encoded on every startup? Why not do it once and save to disk?
- 7speter 3y agoThe big kicker is that cdna is still based on the much maligned
- Pomfers 3y agoAMD consumer cards do compute, but do not have a developed or well supported ecosystem (ROCm is a mess). Misguided segmentation or a high level architecture problem however, it is not. A specialized product will perform its specialty better than a general purpose product. There's minimal demand for a card that's good at both compute and gaming. These people exist, but are few compared to those who only care about one or the other. The benefit of splitting GCN into RDNA and CDNA was immediate. Comparing the Radeon VII (GCN 5) vs RX 5700 XT (RDNA 1), you'll see that they're fairly evenly matched on gaming, trading blows with the Radeon VII slightly ahead when averaged. The RX 5700 XT takes a very heavy loss on compute benchmarks. Both are TSMC 7nm, but the RX 5700 XT has fewer shaders (2560 vs 3840), a smaller die (251 vs 311 mm2), and lower power consumption (225 vs 300 w) which shows how much more efficient it is at gaming. That much lower power consumption and noise, and a couple hundred dollars cheaper made it a much more compelling card for gamers. CDNA cards are apparently missing components needed for gaming, like render output units. Hence, they have no official support for DirectX, OpenGL, or Vulkan. I've never seen anyone get one of these working for gaming. In exchange, their compute performance is so good that a number of companies are buying these cards over Nvidia's, despite the overwhelming CUDA ecosystem. In 2013, a GCN based supercomputer made the top 100. That is the one and only GCN based system to ever make the top 100. Compare to now, where 8 of the top 10 most energy efficient super computers use CDNA accelerators, as does the number 1 fastest supercomputer outright.
- diego_sandoval 3y ago> These people exist, but are few compared to those who only care about one or the other. They're few, but they're very important, because 5 years from now, they will be the ones working for the companies that AMD will want to sell their compute cards to. The reasons you cite may still be more important. I'm not saying AMD made a bad decision. I'm just saying that neglecting the path from hobbyist to professional (or student to professional) is not insignificant.
- Dylan16807 3y agoConsumer cards that are okay at compute do that job almost as well. And RDNA meets that bar... when the software works.
- kaycebasques 3y agoI wasn't familiar with VLIW. Sounds cool! > Very long instruction word (VLIW) refers to instruction set architectures designed to exploit instruction level parallelism (ILP). Whereas conventional central processing units (CPU, processor) mostly allow programs to specify instructions to execute in sequence only, a VLIW processor allows programs to explicitly specify instructions to execute in parallel. This design is intended to allow higher performance without the complexity inherent in some other designs. > The traditional means to improve performance in processors include dividing instructions into substeps so the instructions can be executed partly at the same time (termed pipelining), dispatching individual instructions to be executed independently, in different parts of the processor (superscalar architectures), and even executing instructions in an order different from the program (out-of-order execution).[1] These methods all complicate hardware (larger circuits, higher cost and energy use) because the processor must make all of the decisions internally for these methods to work. https://en.wikipedia.org/wiki/Very_long_instruction_word https://en.wikipedia.org/wiki/Very_long_instruction_word
- mpweiher 3y agoThe most prominent example of VLIW processors was the Itanic, er, Itanium. It, er, didn't work out well. Hence Itanic. Their premise was that the compiler could figure out dependencies sufficiently statically so that they could put multiple sequential and some divergent execution paths in the same instruction. It turned out the compilers couldn't actually do this, so processors figure out dependencies and parallelizable instructions dynamically from a sequential instruction stream. Which is a lot of work, a lot of chip resources and a lot of energy. And only works up to a point, after which you hit diminishing returns. Which is where we appear to be these days.
- kimixa 3y agoVLIW is also still heavily used in DSPs and similar. And arguably the dual issue instructions in both AMD's and Nvidia's latest GPUs is a step back towards that. It feels like a lot of these things are cyclical - the core ideas aren't new and phase in and out of favor as other parts of architecture design changes around and increases or decreases benefits accordingly.
- einpoklum 3y ago> Compute has been outpacing memory for decades. Like CPUs, GPUs have countered this with increasingly sophisticated caching strategies. I'd say it's rather the contrary. Unlike CPUs, GPUs don't attempt to directly counter this. By accepting higher latencies, they have let themselves parallelize much more widely (or wildly) relative to CPUs - and the high number of parallel pseudo-threads provides a "latency hiding" effect. This effect is illustrated for example, in this presentation on optimizing GPU code: https://www.olcf.ornl.gov/wp-content/uploads/2019/12/03-CUDA-Fundamental-Optimization-Part-1.pdf https://www.olcf.ornl.gov/wp-content/uploads/2019/12/03-CUDA... (lame) animation on slide 11 and onwards.
- ColonelPhantom 3y agoGPUs do deal with memory in other ways than parallelism! That's why GPUs tend to offer huge register files (up to 256 architectural registers per thread on RDNA1!) and local memories (up to 64KB LDS per workgroup on RDNA1). This means lots of work can be done purely in registers and LDS, and trips to global memory are way rarer compared to CPUs where global memory contain everything except 16 or so architectural registers. Even then, global memory is an issue. And not just because of latency, but also bandwidth! That's why RDNA2 and Ada both added a ton of last-level cache, not to better hide latency (although it's a welcome addition), but mostly as a 'bandwidth amplifier'.
- einpoklum 3y agoTwo valid points, +1.
- soundarana 3y agoCPUs have been growing their register files too - AVX512 has 32 x 512 bits registers.
- m_mueller 3y agoI think register files, shared memory and even texture memory patterns are basically the GPU's way of having fully programmable caches, as opposed to cache being a grey box like on CPUs. Overall I think for HPC code it works out better. Looking at vectorized and cache-optimized CPU code gives me the shivers... Mostly due to the above I think Intel's years long push to try to have people just run their CPU-optimized OpenMP codes on their Xeon Phis was the downfall of that architecture - it just can't win against a data-parallel-first framework like CUDA, mainly because there the data parallel compute thread can be programmed just as such, rather than trying to get the compiler to behave exactly as you want (and in the end giving up and dropping down to assembly-like SSE instructions).
- feanaro 3y agoA bit off topic, but when did "compute" become a noun? It is extremely grating to my ears.
- synergy20 3y agoit's pretty common these days due to AI with GPU-alike chips.
- EarthLaunch 3y agoI first recall it from the rise of AWS at least. "Amazon Elastic Compute Cloud (EC2)" launched 2006 [0]. Google Trends [1]. 0: https://en.m.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud https://en.m.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud 1: https://trends.google.com/trends/explore?date=all&q=Compute&hl=en https://trends.google.com/trends/explore?date=all&q=Compute&...
- overstay8930 3y agoIt was a thing in Deep Space Nine (1999) so either the lingo was really on point or it's just something that has fallen in and out of favor.
- ip26 3y agoI don’t know the reason, but I suspect you can reason it out.
- greggsy 3y agoI hear, read and use the term on a daily basis, and I would say that it emerged in my workplaces around five years ago, and came into common use around two.
- Geisterde 3y ago5 points for the Luddite (me) that said amd would leverage their knowledge of chiplets and bus fabrics to make a comeback in AI. I wont pretend to be able to read this article, or read in general, just wanted to plant my flag.
- de6u99er 3y agoI don't think AMD will be able to compete with NVidia in the neat future because many scientists developing core libraries in the ML/AI space, are getting free or heavily discounted GPUs from NVidia. If they had to buy GPU's from their own or grant money, and pay the same price as regular consumers, the situation might be a different one. IMHO the way NVidia is infiltrating academics and academia, is highly unethical.
- dagw 3y agoThere is an also a lot of bad blood towards AMD in the space. I know several people who spent a lot of time trying to support both Nvidia and AMD GPUs back in the early days only to have AMD stop supporting APIs and making their code useless. Their CUDA code on the other hand just kept working on generation after generation of new Nvidia cards.
- DeathArrow 3y agoNothing stops AMD to give free cards to developers.
- elAhmo 3y agoIt is important to have in mind that Nvidia started investing resources and time in this 10+ years ago, CUDA was introduced in 2007, where none of the current ML/AI flows even existed. And they continued waiting, betting the company a number of times that the market will "show up" for the products they have built. In the past few years that has happened, as reflected on the stock price. Other players are basically a decade behind them and with all of the current hype and AI/ML workflows becoming mainstream, I think it is next to impossible for anyone to catch up.
- PH95VuimJjqBqy 3y agoThey built it for bit-coin mining, not some super long term plan for ML/AI.
- qeternity 3y ago> Other players are basically a decade behind them I disagree. NVDA aren't in the position they are because they had to work 10 years to get there. They're in that position because it took the market that long to develop, and they were there the whole time. But the main factor is demand. You can't rush that, no matter how early you start. And second mover advantage is a real thing. CUDA is a double edged sword. Everyone has an incentive to break NVDA's dominance, and ultimately their moat will become commoditized especially has dominant architectures like Transformers drive the vast majority of demand. People will simply optimize for that. Can AMD catch up to them quickly on the training / R&D side? Doubtful. Can AMD catch up or even surpass them on the inference side? Absolutely.