15 ms·
The AMD Radeon Instinct MI300A's Giant Memory Subsystem
- amelius 2y agoI'm curious why this space hasn't been patented to death.
- hedora 2y agoIt has been. All sides have a pile of patents. All sides violate all the other sides’ patents. If anyone sues, everyone goes out of business. This is the system working as currently intended. No matter what happens, the lawyers get will rich. If a small company comes in and doesn’t pay the lawyers, it’ll get sued for violating the patents.
- yvdriess 2y agoYep, it's an area denial weapon. You basically cannot do anything worthwhile in this space without violating someone's patents. It's beneficial patent and corporate lawyers, but it's detrimental to innovation. As an engineer you are asked to not look up existing techniques or designs as this will taint you legally.
- kmeisthax 2y ago"Tainting" isn't a thing in patent law. All engineers worldwide are tainted the moment the patent is published; that's why parallel reinvention is not a defense to patent infringement.
- lhl 2y agoBut you pay triple damages if you knowingly vs unknowingly violate a patent (35 U.S.C. § 284). Of course, everything is patented, so, engineers are just told to not read patents.
- amelius 2y ago> If a small company comes in and doesn’t pay the lawyers, it’ll get sued for violating the patents. This assumes the small company isn't just in it for the patents.
- WithinReason 2y agoMutually assured destruction
- btown 2y agoI've often thought that one of the places AMD could distinguish itself from NVIDIA is bringing significantly higher amounts of VRAM (or memory systems that are as performant as what we currently know as VRAM) to the consumer space. A card with a fraction of the FLOPS of cutting-edge graphics cards (and ideally proportionally less power consumption), but with 64-128GB VRAM-equivalent, would be a gamechanger for letting people experiment with large multi-modal models, and seriously incentivize researchers to build the next generation of tensor abstraction libraries for both CUDA and ROCm/HIP. And for gaming, you could break new grounds on high-resolution textures. AMD would be back in the game. Of course, if it's not real VRAM, it needs to be at least somewhat close on the latency and bandwidth front, so let's pop on over and see what's happening in this article... > An Infinity Cache hit has a load-to-use latency of over 140 ns. Even DRAM on the AMD Ryzen 9 7950X3D shows less latency. Missing Infinity Cache of course drives latency up even higher, to a staggering 227 ns. HBM stands for High Bandwidth Memory, not low latency memory, and it shows. Welp. Guess my wish isn't coming true today.
- mpercival531 2y agoThey are. Strix Halo is going after that same space of Apple M4 Pro/Max where it is currently unchallenged. Pairing it with two 64GB LPCAMM2 modules will get you there. Edit: The problem with AMD is less the hardware offerings, but more that their compute software stack historically tends to handwave or be very slow with consumer GPU support — even more so with their APUs. Maybe the advent of MI300A will change the equation, maybe not.
- lhl 2y agoI don't know of any non-soldered memory Strix Halo devices, but both HP and Asus have announced 128GB SKUs (availability unknown). For LLM inference, basically everything works w/ ROCm on RDNA3 now (well, Flash Attention is via Triton and doesn't have support for SWA and some other stuff; also I mostly test on Linux, although I did check that the new WSL2 support works). I've tested some older APUs w/ basic benchmarking as well. Notes here for those interested: https://llm-tracker.info/howto/AMD-GPUs https://llm-tracker.info/howto/AMD-GPUs
- neuroelectron 2y ago>Still, core to core transfers are very rare in practice. I consider core to core latency test results to be just about irrelevant to application performance. I’m only showing test results here to explain the system topology. How exactly are "applications" developed for this? Or is that all proprietary knowledge? TinyBox has resorted to writing their own drivers for 7900 XTX
- latchkey 2y agoROCm is the stack that people write code against to talk to AMD hardware. George wrote some incomplete non-perfomant drivers for a consumer grade product. Certainly not an easy task, but it also isn't something that most people would use. George just makes loud noises to get attention, but few in the HPC industry pay any attention to him.
- tucnak 2y agoNobody cares what HPC industry has to say; until recently, they have happily been jerking off Monte-Carlo simulations on overpriced nation-grade supercomputer NUMA clusters and didn't know what a "GPU" was anyway! Also please stop spreading "consumer grade product" propaganda. I had used AMD Instinct MI50's—supposedly datacenter-grade hardware, and have faced the exact same problems as George. Except in my case there was no call-line at Lisa's. Guess what, the AI industry has spoken: hyper-scalers would buy NVIDIA, or rather design their own silicon. Any thing, any how, but nothing to do with AMD. Also: if your business is doing so great, how come you're constantly in all these Hacker News threads talking and talking and talking but not actually releasing products of any kind, of any bread, that any of the hackers on here could use?
- latchkey 2y ago> but not actually releasing products of any kind, of any bread, that any of the hackers on here could use? Our "product" is open access to a very specific type of HPC compute that previously was locked up and only available to a short list of researchers. Thanks for asking, we just added 1 GPU / 1 minute docker container access through our excellent partners: https://shadeform.ai https://shadeform.ai 1 GPU / 1 VM / 1 minute is coming soon.
- mk_stjames 2y agoSo the 300A is an accelerator coupled with a full 24-core EPYC and 128GB of HBM all on a single chip (or, packaged chiplets, whatever). Why is it I can't buy a single one of these, on a motherboard, in a workstation format case, to use as an insane workstation? Assuming you could program for the accelerator part, there is an entire world of x86-fixed CAD, engineering, and entertainment industry (rendering, etc) where people want a single, desktop machine with 128GB + of fast ram to number crunch. There are Blender artists out there that build dual and quad RTX4090 machines with Threadrippers for $20k+ in components all day, because their render jobs pay for it. There are engineering companies that would not bat an eye at dropping $30k on a workstation if it mean they could spin around 80 gigabyte CATIA models of cars or aircraft loaded in RAM quicker. I know this at least because I sure as hell did with with several HP Z-series machines costing whole-Toyota-Corolla prices over the years... But these combined APU chips are relegated to these server units. In the end is this a driver problem? Just a software problem? A chicken and egg problem where no one is developing the support because there isn't the hardware on the market, and there isn't the hardware on the market because AMD thinks there is no use case? Edit: and note my use cases mentioned don't rely on latency, really, like videogamers need to hit framerates. The cache miss latency mentioned in the article doesn't matter as much for these type of compute applications where the main problems are just loading and unloading the massive amount of data. Things like offline renders and post-processing CFD simulations. Not necessarily a video output framerate.
- latchkey 2y ago(I run a company that buys MI300x.) > Why is it I can't buy a single one of these, on a motherboard, in a workstation format case, to use as an insane workstation? AMD doesn't have the resources to support end users for something like this. They are a public company, look at their spend. They are pouring everything they've got into trying to keep up with the Nvidia release cycle for AI chips. These chips are cutting edge, they are not perfect. They are still working through the hardware and software issues. It is hard enough to deal with all the public opinion on things as it is. Why would they add another layer of potential abuse?
- AnthonyMouse 2y ago
- behnamoh 2y agoAMD is done, no one uses their GPUs for AI because AMD were too dumb to understand the value of software lock-in like Nvidia did with CUDA.
- guywhocodes 2y agoMore like the value of drivers that doesn't require one in-house team per customer to "fix" driver crashes in the customers' particular workloads.
- numpy-thagoras 2y agoYeah, the labour involved in running non Nvidia equipment is the elephant in the room. Nvidia GPU: spin up OS, run your sims or load your LLM, gather results. AMD GPU: spin up OS, grok driver fixes, try and run your sims, grok more driver fixes, can't even gather results until you can verify software correctness of your fixes. Yeah, sometimes you need someone with specialized knowledge of numerical methods to help tune your fixes. ... What kind of maddening workflows are these? It's literally negative work: you are busy, you barely get anywhere, and you end up having to do more. In light of that, the Nvidia tax doesn't look so bad.
- DiabloD3 2y agoFunny you say that, because nobody serious about AI is actually using Nvidia unless they're already locked in with CUDA. Highest performing inference engines all use Vulkan, and are either faster per dollarwatt on the CDNA3 cards or (surprisingly) the RDNA3 cards, not Lovelace.
- behnamoh 2y ago> Funny you say that, because nobody serious about AI is actually using Nvidia unless they're already locked in with CUDA. Yeah right, so Meta and XAI buying hundreds of Nvidia's H100's was because they were not serious in AI. wtf
- 2y ago
- ChuckMcM 2y agoThat is quite a thing. I've been out of the 'design loop' for chips like this for a while so I don't know if they still do full chip simulations prior to tapeout but woah trying to simulate that thing would take quite the compute complex in itself. Hat's off to AMD for getting it out the door.
- erulabs 2y agoits interesting that two simultaneous and contradictory views are held by AI engineers: - Software is over - An impenetrable software moat protects Nvidia's market capitalization
- zaseevazarina 2y ago[dead]
- buyucu 2y agoMI300 is an insanely good GPU. There is nothing that Nvidia sells that even comes close. The H100 only has 80GB of memory, whereas MI300 has 192GB. If you are training large models, AMD is the way to go.