18 ms·
Testing AMD's Giant MI300X
- pella 2y agofrom the summary: "When it is all said and done, MI300X is a very impressive piece of hardware. However, the software side suffers from a chicken-and-egg dilemma. Developers are hesitant to invest in a platform with limited adoption, but the platform also depends on their support. Hopefully the software side of the equation gets into ship shape. Should that happen, AMD would be a serious competitor to NVIDIA."
- firebaze 2y agoOTOH, the performance advantage compared to the H100 is super-impressive according to tfa. Things could become interesting again in the GPU market.
- bearjaws 2y agoIMO Nvidia is going to force companies to fix this, Nvidia have always made it clear they will increase prices and capture 90% of your profits when left free to do so. See any example from the GPU vendor space. There isn't infinite money to be spent per token, so it's not like the AI companies can just increase prices. AMD can offer this product at a 40% discount and still make money tells you all you need to know.
- amelius 2y agoI'm personally wondering when nVidia will open an AI AppStore, and every app that runs on nVidia hardware will have to be notarized first, and you'll have to pay 30% of your profits to nVidia. History has shown that this idea is not as crazy as it sounds.
- bearjaws 2y agoOh that is a great example, just wipe out Hugging Face, Ollama, Together.ai, and probably 20 more. They can host it all themselves or require their vendors to lease some time to them at cost..
- deleted 2y ago[deleted]
- autoexecbat 2y agoAMD should be giving out these units to whichever clouds are willing to host them, so they can get them in the hands of developers to induce demand via working software
- latchkey 2y agoYou are totally right.
- latchkey 2y agoThe news you've all been waiting for! We are thrilled to announce that Hot Aisle Inc. proudly volunteered our system for Chips and Cheese to use in their benchmarking and performance showcase. This collaboration has demonstrated the exceptional capabilities of our hardware and further highlighted our commitment to cutting-edge technology. Stay tuned for more exciting updates!
- JonChesterfield 2y agoThank you for loaning the box out! Has a lot more credibility than the vendor saying it runs fast
- latchkey 2y agoThanks Jon, that's exactly the idea. About $12k worth of free compute on a box that costs as much as a Ferrari. Funny that HN doesn't like my comment for some reason though.
- renewiltord 2y agoIt reads like the kind of chumbox PR you read at bottom of random website. Get a copywriter or something like writer.ai. I thought your comment was spam and nearly flagged it. It really is atrocious copy.
- jampekka 2y agoI thought it was sarcastic.
- logicallee 2y ago[retracted]
- latchkey 2y agoThis is the news that many people have been waiting for and we do have more exciting updates coming. There is another team on the system now doing testing. We have a list of 22 people currently waiting.
- elorant 2y agoEven if the community provides support it could take years to reach the maturity of CUDA. So while it's good to have some competition, I doubt it will make any difference in the immediate future. Unless some of the big corporations in the market lean in heavily and support the framework.
- JonChesterfield 2y agoSomething like Triton from Microsoft/OpenAI as a cuda bypass? Or pytorch/tensorflow targeting ROCm without user intervention. Or there's openmp or hip. In extremis opencl. I think the language stack is fine at this point. The moat isn't in cuda the tech. It's in code running reliably on nvidia's stack, without things like stray pointers needing a machine reboot. Hard to know how far off robust rocm is at this point.
- jeroenhd 2y agoIf, and that's a big if, AMD can get ROCm working well for this chip, I don't think this will be a big problem. ROCm can be spotty, especially on consumer cards, but for many models it does seem to work on their more expensive models. It may be worth it spending a few hours/days/weeks to work around the peculiarities of ROCm given the cost difference between AMD and Nvidia in this market segment. This all stands or falls with how well AMD can get ROCm to work. As this article states, it's nowhere near ready yet, but one or two updates can turn AMD's accelerators from "maybe in 5-10 years" to "we must consider this next time we order hardware". I also wonder if AMD is going to put any effort into ROCm (or a similar framework) as a response to Qualcomm and other ARM manufacturers creaming them on AI stuff. If these Copilot PCs take off, we may see AMD invest into their AI compatibility libraries because of interest from both sides.
- entropicdrifter 2y agoIf we're extremely lucky they might invest in SYCL and we'll see an Intel/AMD open-source teamup
- sorenjan 2y ago
- JonChesterfield 2y agoFantastic to see. The MI300X does memory bandwidth better than anything else by a ridiculous margin, up and down the cache hierarchy. It did not score very well on global atomics. So yeah, that seems about right. If you manage to light up the hardware, lots and lots of number crunching for you.
- jsheard 2y agoAll eyes are of course on AI, but with 192GB of VRAM I wonder if this or something like it could be good enough for high end production rendering. Pixar and co still use CPU clusters for all of their final frame rendering, even though the task is ostensibly a better fit for GPUs, mainly because their memory demands have usually been so far ahead of what even the biggest GPUs could offer. Much like with AI, Nvidia has the software side of GPU production rendering locked down tight though so that's just as much of an uphill battle for AMD.
- Havoc 2y agoI’d imagine ray tracing is a bit easier to paralize over lots of older cards. The computations aren’t as heavily linked and are more fault tolerant. So I doubt anyone is paying h100 style premiums
- bryanlarsen 2y agoPixar is paying a massive premium; they probably are using an order of magnitude or two more CPUs than they would if they could use GPUs. Using a hundred CPUs in place of a single H100 is a greater-than-h100 style premium.
- imjonse 2y agoWould Pixar's existing software run on GPUs without much work?
- jsheard 2y agoIt does already, at least on Nvidia GPUs: https://rmanwiki.pixar.com/pages/viewpage.action?mobileBypass=true&spaceKey=REN24&title=XPU https://rmanwiki.pixar.com/pages/viewpage.action?mobileBypas... They currently only use the GPU mode for quick iteration on relatively small slices of data though, and then switch back to CPU mode for the big renders.
- jsheard 2y agoThe computations are easily parallelized, sure, but the data feeding those computations isn't easily partitioned. Every parallel render node needs as much memory as a lone render node would, and GPUs typically have nowhere near enough for the highest of high end productions. Last I heard they were putting around 128GB to 256GB of RAM in their machines and that was a few years ago.
- Pesthuf 2y agoI feel like these huge graphics cards with insane amounts of RAM are the moat that AI companies have been hoping for. We can't possibly hope to run the kinds of models that run on 192GB of VRAM at home.
- dmbaggett 2y agoFor inference you could use a maxed-out Mac Ultra; the RAM is shared between the CPU and GPU.
- alecco 2y agoFor single user (batch_size = 1), sure. But that is quite expensive in $/tok.
- jsheard 2y agoApple will gladly sell you a GPU with 192GB of memory, but your wallet won't like it.
- kbenson 2y agoWon't Nvidia, and Intel, and Qualcomm, and Falanx (who make the ARM Mali GPUs from what I can see), and Imagination Technologies (PowerVR) do the same? They each make a GPU, and if you pay them enough money I have a hard time beleiving they won't figure out how to slap enough RAM on a board for one of their existing products and making whatever changes are required.
- nextaccountic 2y agoThe US government is looking into heavily limit availability of high end GPUs from now on. And the biggest and most effective bottleneck for AI right now is VRAM So maybe Apple is happy to sell huge GPUs like that but the government will probably put it under export controls like A100 and H100 already is
- rbanffy 2y ago
- tonetegeatinst 2y agoI wonder if the human body could grow artificial kidney's so that I can just sell infinite kidney's and manage to afford a couple of these so I can do AI training on my own hardware.
- pca006132 2y agowhy not infinite brains so you can have more computational power than these GPUs?
- Maken 2y agoApparently one of those costs around 15K $. I don't know if you can buy a couple or they only sell those in massive batches, but in any case, how many human kidneys you need to sell to get 30K $?
- Filligree 2y agoSo just out of curiosity, what does this thing cost?
- pheatherlite 2y agoWithout first-class CUDA translation or cross compile, AMD is just throwing more transistors at the void
- chung8123 2y agoI agree they need to work on their software but I also think that the availability as well as massive expense of the H100, AMD can undercut Nvidia and build a developer ecosystem if they wanted to. I think they need to hit the consumer market pretty hard and get all the local llama people hacking up the software and drivers to make things work. A cheaper large VRAM consumer card would go a long ways to getting a developer ecosystem behind them.
- epistasis 2y agoGiven the number of people who need the compute but are only accessing it via APIs like HuggingFace's transformers library, which supports these chips, I don't really think that CUDA support is absolutely essential. Most kernels are super quick to rewrite, and higher level abstractions like PyTorch and JAX make dealing with CUDA a pretty rare experience for most people making use of large clusters and small installs. And if you have the money to build a big cluster, you can probably also hire the engineers to port your framework to the right AMD library. The world has changed a lot! The bigger challenge is that if you are starting up, why in the world would you give yourself the additional challenge of going off the beaten path? Its not just CUDA but the whole infrastructure of clusters and networking that really gives NVIDIA an edge, in addition to knowing that they are going to stick around in the market, whereas AMD might leave it tomorrow.
- fragmede 2y agoWhen buying a supercomputer, you negotiate support contracts so it doesn't matter if AMD leaves the day after they sign the contract, you've still got your supercomputer and support for it.
- epistasis 2y agoTrue, but that works only for the current round of hardware. NVIDIA will be around for the next decade to support future clusters, too.
- alkonaut 2y agoGood. If there is even a slight suspicion that the best value is team read in 5 or 10 years then CUDA will look a lot less attractive already today.
- w-m 2y agoImpressions from last week’s CVPR, a conference with 12k attendees on computer vision - Pretty much everyone is using NVIDIA GPUs, and pretty much everyone isn’t happy with the prices, and would like some competition in the space: NVIDIA was there with 57 papers, a website dedicated to their research presented at the conference, a full day tutorial on accelerating deep learning, and ever present with shirts and backpacks in the corridors and at poster presentations. AMD had a booth at the expo part, where they were raffling off some GPUs. I went up to them to ask what framework I should look into, when writing kernels (ideally from Python) for GPGPU. They referred me to the “technical guy”, who it turns out had a demo on inference on an LLM. Which he couldn’t show me, as the laptop with the APU had crashed and wouldn’t reboot. He didn’t know about writing kernels, but told me there was a compiler guy who might be able to help, but he wasn’t to be found at that moment, and I couldn’t find him when returning to the booth later. I’m not at all happy with this situation. As long as AMDs investment into software and evangelism remains at ~$0, I don’t see how any hardware they put out will make a difference. And you’ll continue to hear people walking away from their booth, saying “oh when I win it I’m going to sell it to buy myself an NVIDIA GPU”.
- cstejerean 2y agoCompletely agree. It's been 18 years since Nvidia released CUDA. AMD has had a long time to figure this out so I'm amazed at how they continue to fumble this.
- bryanlarsen 2y ago10 years ago they were basically broke and bet the farm on Zen. That bet paid off. I doubt a bet on CUDA would have paid off in time to save the company. They definitely didn't have the resources to split that bet.
- jsheard 2y agoIt's not like the specific push for AI on GPUs came out of nowhere either, Nvidia first shipped cuDNN in 2014.
- kimixa 2y ago
- spitfire 2y agoI remember years ago one of the amd apus had the cup and gpu on the same die, and could exchange ownership of cpu and gpu memory with just a pointer change or some other small accounting. Has this returned? Because for dual gpu/cpu workloads (alpha zero, etc) that would deliver effective “infinite bandwidth” between gpu and cpu. Using an apu of course gets you huge amounts of slowish memory. But being some to fling things around with abandon would be an advantage, particularly for development.
- wmf 2y agoI assume the MI300A APU also supports zero-copy. Because MI300X is a separate chip you necessarily have to copy data over PCIe to get it into the GPU.
- rbanffy 2y agoOne day someone will build a workstation around that chip. One day…
- JonChesterfield 2y agoYou don't need to change the pointer value. The GPU and the CPU have the same page table structures and both use the same pointer representation for "somewhere in common memory". On the GPU there are additional pointer types for different local memory, e.g. LDS is a uint16_t indexing from zero. But even there you can still have a single pointer to "somewhere" and when you store to it with a single flat addressing instruction the hardware sorts out whether it's pointing to somewhere in GPU stack or somewhere on the CPU. This works really well for tables of data. It's a bit of a nuisance for code as the function pointer is aimed at somewhere in memory and whether that's to some x86 or to some gcn depends on where you got the pointer from, and jumping to gcn code from within x86 means exactly what it sounds like.
- spitfire 2y agoI'm not sure it was "pointers" but it was some very low cost way to change ownership of memory between the CPU and GPU. They had some fancy marketing name for it at the time. But it wasn't on all chips, it should have been. Even if it was dog slow between PCIe GPU and CPU the unified interface would have been the right way to go. Also, amenable to automated scheduling. The point still stand though, I want entirely unified GPU and CPU memory.
- omneity 2y agoI'm surprised at the simplicity of the formula in the paragraph below. Could someone explain the relationship between model size, memory bandwidth and token/s as they calculated here? > Taking LLaMA 3 70B as an example, in float16 the weights are approximately 140GB, and the generation context adds another ~2GB. MI300X’s theoretical maximum is 5.3TB/second, which gives us a hard upper limit of (5300 / 142) = ~37.2 tokens per second.
- wmf 2y agoAFAIK generating a single token requires reading all the weights from RAM. So 5300 GB/s total memory bandwidth / 142 GB weights = ~37.2 tokens per second.
- latchkey 2y agoFrom Cheese (they don't have a HN account, so I'm posting for them): Each weight is a FP16 float which is 2 Bytes worth of data, you have 70B tokens, so the total amount of data the weights take up is 140GB then you have a couple extra GBs for the context. Then to figure out the theoretical tokens per second you just divide the amount of memory bandwidth, 5300GB/s in MI300X's case, by the amount of data that the tokens and context take up so 5300/142 which is about 37 tokens per second.
- rbanffy 2y ago37 somethings per second doesn’t sound fast at all. You need to remember it’s 37 ridiculously difficult things per second.
- omneity 2y agoSo am I correct in understanding what they really mean is 37 full forward passes per second? In which case, if the model weights are fitting in the VRAM and are already loaded, why does the bandwidth impact the rate of tok/s?
- fancyfredbot 2y agoYou have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that. The weights are not really reused. Which means they are never in registers, or in L1/L2/L3 caches. They are always in VRAM and always need to be loaded back in again. However, if you are batching multiple separate inputs you can reuse each weight on ech input, in which case you may not be entirely bandwidth bound and this analysis breaks down a bit. Basically you can't produce a single stream of tokens any faster than this rate, but you can produce more than one stream of tokens at this rate.
- snaeker58 2y agoI hate the state of AMDs software for non gamers. RoCm is a war crime (which has improved dramatically in the last two years and still sucks). But like many have said considering AMD was almost bankrupt their performance is impressive. This really speaks for their hardware division. If only they could get the software side of things fixed! Also I wonder if NVIDIA has an employee of the decade plaque for CUDA. Because CUDA is the best thing that could’ve happened to them.
- rbanffy 2y agoWould be interesting to see a workstation based on the version with a couple x86 dies, the MI300A. Oddly enough, it’d need a discrete GPU.
- alecco 2y ago> Taking LLaMA 3 70B as an example, in float16 the weights are approximately 140GB, and the generation context adds another ~2GB. MI300X’s theoretical maximum is 5.3TB/second, which gives us a hard upper limit of (5300 / 142) = ~37.2 tokens per second. I think they mean 37.2 forward passes per second. And at 4008 tokens per second (from "LLaMA3-70B Inference" chart) it means they were using a batch size of ~138 (if using that math, but probably not correct). Right?
- mugivarra69 2y agoi worked there. they see software as a cost center, they should fix their mentality.
- Palmik 2y agoIt would be great to have real world inference benchmarks for LLMs. These aren't it. That means e.g. 8xH100 with TensorRT-LLM / vLLM vs 8xMI300X with vLLM running many concurrent requests with reasonable # of input and output tokens. Ran both in fp8 and fp16. Most of the benchmarks I've seen had setups that no one would use in production. For example running on a single MI300X or 2xH100 -- this will likely be memory bound, you need to go to higher batch sizes (more VRAM) to be compute bound to properly utilize these. Or benchmarking requests with unrealistically low # of input tokens.
- latchkey 2y agoWould you like to sign up for free time on our system to do it "right"? https://hotaisle.xyz/free-compute-offer/ https://hotaisle.xyz/free-compute-offer/