10 ms·
AMD Expands AI Product Lineup with GPU-Only Instinct Mi300X with 192GB Memory
- gymbeaux 3y agoI hope these new Instinct GPUs are less of a paper launch than previous Instinct GPUs. Historically they’ve been hard for consumers to buy.
- thewataccount 3y agoDoes anyone know what type of api you'd use for this? I know AMD has Rocm for their dedicated GPUs but it's barely usable from what I've heard.
- stuckinhell 3y agoROCm just doesn't have the support these days. Supposedly people are working on getting stable diffusion working on it. https://www.videogames.ai/2022/11/06/Stable-Diffusion-AMD-GPU-ROCm-Linux.html https://www.videogames.ai/2022/11/06/Stable-Diffusion-AMD-GP... But it's just too much of investment for me on something that MAY work. I ended up just buying a 4080rtx
- pja 3y agoStable Diffusion worked fine for me with Rocm & my RX580 (once I had compiled a custom torch library IIRC). But I don’t know whether it works with the more recent RDNA2 cards.
- pohuing 3y agoIt works without a hitch on my rx6900xt. Only pain was getting the amdpro drivers
- thewataccount 3y agoYeah that's kinda what I mean. I've heard to not even consider AMD for ML applications specifically ROCM, so I'm curious if these chips will use ROCM as their primary api or not. It "technically" works, but their own examples crash, you get a fraction of the performance you'd expect for the level of hardware you have, chicken&egg problem with little other software having good support of it, etc.
- delusional 3y agoI'm running stable diffusion on my 7900xtx and it's working fine. I had to screw around a little bit to get the newest ROCm and torch libraries since they aren't packed on my OS, but it wasn't that bad. I made a docker image if anybody is struggling to get it working: https://hub.docker.com/r/delusional/sd-rx7900xtx https://hub.docker.com/r/delusional/sd-rx7900xtx
- thewataccount 3y agoOut of curiosity, how many it/s do you get with DPM2 at 512x512 with a batch size of 1, and then the it/s for whatever the max batch size you can fit?
- delusional 3y agoIt does 15-16 it/s with euler a and 2.16 it/s at 8 batchsize (the max in automatic1111), and that's only using 15GiB of vram.
- brucethemoose2 3y agoStable Diffusion works fine on rocm (and intel OpenVINO), the issue is out-of-the-box support in popular UIs. TBH the whole space is kinda a mess. Tons of optimizations (like most ML compilers), even on Nvidia cards, are left on the table because the SD UI devs just dont have the throughout or motivation to implement them. At the other end, hardware makers, ml compiler devs, researchers and such are making quick demos, but are not making any integration attempts for popular frameworks. There is no one in the middle, so we are stuck with PyTorch eager mode and a perception that it only works on big Nvidia GPUs.
- sp332 3y agoThis tools page makes me think they are sticking with ROCm compilers and OpenCL. https://docs.amd.com/category/compilers_and_tools https://docs.amd.com/category/compilers_and_tools They have a tool called "hipify" that supposedly converts CUDA code to something else. But the fact it's not just integrated as a CUDA compiler makes it look like they don't really trust it to work. Also they misspelled it.
- kcb 3y agoHIP itself is a C++ language that can be utilized directly. It's extremely similar to CUDA. I think the use case AMD desires is the reverse. That software is written in HIP then converted to CUDA to also run on Nvidia GPUs.
- digitallyfree 3y agoThe issue with AMD and AI is, as always, the software stack. Even if the hardware is great ROCM simply doesn't have industry traction and accessiblity.
- wing-_-nuts 3y agoDoesn't have the traction for now. Cloud providers (ms, google, amz, etc) are quickly tiring of paying Nvidia monopoly premiums for their gpu hardware. Google has already invested in tpus and it wouldn't surprise me at all if they got together to fund ROCm development or even went so far as to develop their own NN asics. Cuda is great, but it's not strictly necessary for much of the latest AI / ML developments.
- gremlinsinc 3y agoIntel, AMD, Google, Amazon, etc should team up to create some sort of standards/consortium around an open source CUDA alternative, something that anyone who can fabricate chips could use, and the consortium could have their own team of devs/researchers to make improvements / next gen versions of their CUDA alternative. Something like the way chrome vs chromium is, or even a foundation like the linux foundation, where you have multiple distros contributing packages/etc back into the ecosystem.
- bfeynman 3y agothey are already doing that with XLA, google has TPUs, amazon has tranium/inferentia. common interface in future that you basically just cast model to `.toDevice` of an enum of accelerated computing types seems to be the goal.
- mepian 3y agoThey already did: https://www.khronos.org/sycl/ https://www.khronos.org/sycl/
- YetAnotherNick 3y ago> Cloud providers (ms, google, amz, etc) are quickly tiring of paying Nvidia monopoly premiums for their gpu hardware. I think cloud providers love exclusivity(Nvidia MSRP is significantly higher than it is available to clouds) and based on pricing compared to competitors like lambdalabs they have highest profit margin on GPU instances. Also based on availability, they likely have the highest utilisation. They definitely wouldn't want to commoditize the space. Google already has TPU that they could scale and sell to everyone but it would make the margins significantly smaller if they do it.
- bratao 3y agoI'm genuinely rooting for AMD to develop a competitive alternative to NVIDIA. Currently, NVIDIA dominates the machine learning landscape, and there doesn't seem to be a justifiable reason for the price discrepancy between the RTX 4090 and the A100.
- ChuckNorris89 3y ago>a justifiable reason for the price discrepancy between the RTX 4090 and the A100 1. they're a publicly listed company operating in a free market in an industry they themselves helped develop who's purpose is to make returns for their investors, not be liked by the public, they don't need to justify their pricing to their buyers. They're not selling essentials for survival like insulin, baby formula, or housing, they can charge as much as the market will bear for their consumer electronics products. Don't like the pricing? Don't buy it. Simple. Buy from the competition instead or older generations off the second hand market that fit your budget. 2. the price justification is that cutting edge silicon is and will always be in short supply, and buyers of the silicon in A100 form, like datacenters, use it to make money, therefore it's an investment that will yield returns, therefore they can justify spending way more money to outbid the gamers who buy the same silicon in RTX 4090 form and don't use it to make money but use it to play games therefore for them the product is worth less and it makes Nvidia smaller margins than what selling it to datacenters can. It's basic price segmentation that's been going on for decades. I also don't like the GPU pricing situation but that's the market reality I can't change and downvoting the messenger won't change it either. My 2 cents.
- ActorNightly 3y agoYeah but the issue is that they can make more money if they actually did affordable ML hardware and didn't purposefully gimp the gaming cards. LLms running on standalone affordable boxes could become a ubiquitous home accessory, not to mention the potential for things like being able to leverage learning from widespread use to retrain the models, like Tesla does with their fleet.
- ChuckNorris89 3y ago
- michelb 3y agoBit of a noob on these things here, but why 192GB? Why not, say, 256GB? I noticed the new mac pro also caps out at 192GB, is there a specific reason for that number?
- zargon 3y agoThe article says that HBM3 memory comes in 24GB stacks. (Not 32GB.)
- wmf 3y agoThe largest DRAM dies are 24 gigabit.
- Dylan16807 3y agoIn this case it's because memory scaling has slowed down, so the manufacturers have introduced sizes that are halfway between powers of two. It would simply take too long to wait for 2x before making a new product line. So the biggest HBM you can get is 24GB, and 8 of them is a reasonable max.
- brucethemoose2 3y agoThe MI300 has 8x 24GB stacks of memory, which is physically the absolute max they can do. And those 24GB stacks are brand new. I believe the Apple M series is limited by package space? The LPDDR5(X?) bus is not physically/electrically limited to 192GB like the MI300.
- ilaksh 3y agoIs it possible to do inference with Falcon 40B on this type of hardware or similar?
- seydor 3y agoThey showed Falcon 40B running on the mi300x
- ilaksh 3y agoHm. Do you have a link to that? It would be nice if this became on option for something like RunPod or Modal. Especially if it could be slightly cheaper than the Nvidia hardware somehow.
- seydor 3y agohttps://www.youtube.com/live/l3pe_qx95E0?feature=share&t=5648 https://www.youtube.com/live/l3pe_qx95E0?feature=share&t=564...
- __anon-2023__ 3y agoThe Falcon appears at t=5033: https://youtu.be/l3pe_qx95E0?t=5033 https://youtu.be/l3pe_qx95E0?t=5033
- deleted 3y ago[deleted]
- brucethemoose2 3y agoYes, but would be massive overkill. Falcon 40B takes ~35GB of VRAM to load now, and probably less in the future with better quant from llama.cpp and such. https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Large context size is becoming less of an issue now too. But maybe it would be good for batched inference?
- 3y ago
- mastax 3y agoAMD ROCm vs Nvidia CUDA has been discussed to death, but I'm curious how AMD fares compared to some of the AI training accelerator vendors. I think it would be much more damning if AMD were worse than some upstart, because the upstart wouldn't have the huge resource advantage and decade long head start of Nvidia. From my limited experience it seems like Google TPU and Cerebras are much nicer to use for AI training, from the standpoint of driver and software stability, documentation, and ecosystem support. Perhaps that's not a fair comparison. From what I know AMD and NVIDIA use GPGPU cores (now with AI-focused instructions) plus separate AI-specific accelerator blocks. Conceptually, GPGPU + NPU on one die. NPUs can be much simpler than general-purpose GPUs. So AMD's driver and software stack likely needs to be an order of magnitude more complex than the NPU vendors' in order to accommodate other non-AI use cases. But to an end user it doesn't really matter why it sucks, only that it does.
- brucethemoose2 3y ago...Not precisely. AMD implements the "AI" matrix instructions in the shaders themselves, not as big seperate blocks like Nvidia. But unlike Nvidia, the instructions are different on the consumer (RDNA3) and server line. I dont know anything about building rocm, but supporting rdna must indeed make things more difficult. Intel takes this approach too.
- fcanesin 3y agoMI300A looks really nice, if only one anyone could buy it and do a DIY "Mac Studio"-like linux machine. Imagine just swapping it for the next generation in 2 or 3 years and keeping all the rest: peripherals, case, motherboard and etc.
- rbanffy 3y agoAnd the x86/GPU version would be an awesome workstation. Just add M.2 and ethernet (and USB-C with video) and we are in business. As for a Hackintosh, I'd imagine an Nvidia Grace or Grace Hopper as a good option, even though the 500+W TDP would require a MacPro-sized heatsink. And it can have up to 960GB of RAM. edit: got a lot of details confused between the MI300 and the Grace.
- lvl102 3y agoTo be honest, if I am building a CUDA alternative from scratch, I’d build it for Apple Silicon. Apple will have a monopoly on TSM’s state of the art.
- brucethemoose2 3y agoThe issue is Apple has zero presence in datacenters, and they dont seem very interested in changing that. And they are optimizing more for power efficiency in multimedia workloads than raw AI throughout at any cost like the AI ASIC makers.
- rbzz 3y agoAMD relations with TSMC are definitely much more tight than Nvidia's. AMD has been helping with developing HPC-optimized nodes and modern packaging techniques (3d-stacking and chiplets), while Nvidia tried to force TSMC to drop prices by going to Samsung fabs for 1.5 generations (did not go too well for them). The problem with AMD is definitely not modern node access, but software, and with some investment it can probably change really fast. Apple, on the other hand, is definitely not going to release standalone apple silicon products (GPUs or CPUs), so I don't really believe for Apple Silicon as an AI platform (apart from on-device inferrence).
- brucethemoose2 3y agoMaybe they have something in the oven? I am skeptical too, but hardware design takes a looong time, and surely Apple sees the writing on the wall and the potential of their own hardware.
- rbanffy 3y agoHonestly, I find the x86+GPU parts more interesting. One could make a very Apple-like PC with a SoM and very little glue around it (a bit like what you can do with some top-of-the-line Xeons that have HBM chiplets). And with between 1 and 6 x86 chiplets, the Mi300 could have between 8 and 96 Zen 4 cores. These things are so interesting it's a shame they aren't cheap.
- the_pwner224 3y agoAn AMD Ryzen APU + RAM soldered onto the motherboard is very similar to what Apple's doing. Afaik, because the memory controller is part of the CPU, the CPU-RAM connection on the consumer chips is entirely passive - just copper traces on the motherboard with no ICs inbetween?
- NekkoDroid 3y agoIIRC the APUs aren't exactly typical APUs with shared memory, but more a CPU with a few GPU cores attached, operating independently memory whise (but I might be wrong).
- JonChesterfield 3y agoThe ones that have been shipping for a few years have one block of ddr that GPU and CPU cores both access. You can do things like synchronise code running in a thread on the CPU with a wavefront running in a kernel on the GPU using atomic operations on that shared memory.
- brucethemoose2 3y agoAMD and Intel are both making quad channel APUs akin to the M2 Pro. https://videocardz.com/newz/intel-arrow-lake-p-with-320eu-gpu-confirmed-by-a-leaked-roadmap-targeting-to-compete-with-apple-14-premium-laptops https://videocardz.com/newz/intel-arrow-lake-p-with-320eu-gp... (Sorry, I cannot find the AMD rumor link atm) But TBH the hybrid design is less interesting than you think, just because nothing really takes advantage of it. Hence Intel canceled their datacenter APU in favor of a pure Falcon Shores GPU due to a lack of interest from customers.
- zwaps 3y agoIf you want to go an run inference/train on one of these fresh and sweet LLMs using bitsandbytes/PEFT, which is really where the excitement is at, you gotta use CUDA pretty much. This is the story now. And the story for the innovation before was the same. Use CUDA, or wait for AMD to catch up a year late and with a worse version of everything. I mean, sure, you could compile your stuff to XLA or just, I dunno, set up 800 of these cards and train the whole thing on ROCM. But then would you really, really use AMD instead of some TPUs? AMD made Machine Learning either impossible, unsupported or a chore on most of their hardware for years. Their stack sucks and no one wanted to implement it, and in fact they rarely supported most of their own GPUs. Yes you have GPUs. But we need also drivers. And software. People have been yelling at AMD about this for years and years. Instead, AMD has made it clear that Machine Learning is not a priority for the company. Hence, they deserve the lack of traction. Investing in AMD hardware for ML has literally been a mistake at every point in recent history. Imagine if you bought a bunch of (insert last Gen card which is no longer supported by their stack) how dumb you'd look. Releasing a GPGPU card now? Honestly, why bother? No one is gonna buy it.
- stevenhuang 3y agoBeen playing with Stable Diffusion on 6750XT 12GB for a while now with minimal issues (low memory issues sometimes on higher res, but it's less of a problem using grid upscaler). ROCm is getting there, slowly.
- MINIMAN10000 3y agoAlso good news was that running olive and downloading the ONNX converter to run stable diffusion shows something like 120% speedup on Nvidia and 50% speedup on AMD from the numbers I see. Some UIs are beginning to support ONNX format. ONNX being the format that has Microsoft's support behind it for AI models to my understanding.
- thereisnospork 3y agoThis is a naive question, but how hard would it be for AMD to make their cards/firmware CUDA compatible? Feels like that's what they would need to do to sell hardware in the space, other than banking on sufficiently severe shortages.
- smy20011 3y agoOne thing AMD can do is working with ggml to make llama.cpp running on AMD GPUs. Compile modern ML framework is a quite complex due to number of OPs. However, running LLMs does not need a lot of OPs. Just einsum & relu & softmax. Having LLM works with llama.cpp could be done by a team within a week or so.
- lhl 3y agoI agree that AMD should be dedicating an engineer to making sure all the top popular ML/AI projects can run on their hardware, but maybe they also need to spin up a wiki listing compatibility... AMD GPU acceleration via CLBlast was merged back in mid-May in llama.cpp:master - it works and gives a boost (although not all AMD GPUs have been tuned for CLBLast - this is something that AMD should be doing tbt: https://github.com/RadeonOpenCompute/ROCm/issues/2161 https://github.com/RadeonOpenCompute/ROCm/issues/2161) There is also a hipBLAS fork, which is slightly faster (~10% on my old Radeon VII) which maybe someone at AMD should be supporting to make its way into master: https://github.com/ggerganov/llama.cpp/pull/1087 https://github.com/ggerganov/llama.cpp/pull/1087 I'll also note that exllama merged ROCm support and it runs pretty impressively - it runs 2X faster than the hipBLAS llama.cpp, and in fact, on exllama, my old Radeon VII manages to run inference >50% faster than my old (roughly equal class/FP32 perf) GTX 1080 Ti (GCN5 has 2X fp16, and 2X memory bandwidth, so there's probably more headroom even) for a relatively easy port. That's really impressive: https://github.com/turboderp/exllama/pull/7 https://github.com/turboderp/exllama/pull/7 (It's worth noting that for the latter, all you need to do is install the ROCm version of PyTorch and "it just works," which is refreshing: https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/)
- qwertox 3y ago> An engineer More like a dozen of engineers over a couple of weeks. Make sure popular LLMs can run on their hardware, as well as Stable Diffusion and other popular projects, and then they will see consumers flock to their hardware. With consumers I mean something like what "gamers" used to be during all these past decades (and still are), those who won't be using it for business cases (what Quadro used to be) but for their hobby; this, but oriented towards AI, which currently is limited to LLM, image generation and ASR. If they focus on this, the community will start helping, maybe even cleaning up their mess of repositories they have on GitHub. There are two benefits over Nvidia: They have more VRAM and they have an open source software stack. They just need to get the basics working for all those hobbyists, those who want to run little projects on their home hardware.
- deleted 3y ago[deleted]