3 ms·

432 GB of RAM per chip. Tens of terabytes per rack. Clients queuing up to buy them. RAM prices are not coming down any time soon.

nl 3mo ago
Did anyone think RAM prices were coming down soon?
- ACCount37 3mo ago
  Of course. Wishful thinking is a true staple of human intelligence.
- Goronmon 3mo ago
  There are plenty of laymen that think "AI bubble pop" means that "AI" usage will go to 0 and then all of a sudden all these hardware and datacenters related companies are buying will become totally unused and flood the market.
swiftcoder 3mo ago
Hey, they are only projected to cost about $5.5million a rack. Pocket change, really
stevefan1999 3mo ago
but you have to rewrite all your software to ROCm. And ROCm, to this day, still sucks. ZLUDA basically crash on high memory demand, and only accounted for ~70% of CUDA API coverage (and it is still buggy). But hey, at least it does run on Rust-CUDA. I'm one of the few who ported it and fixed a few bugs on ZLUDA. I used it to run a simple SHA256 kernel and it ran sure, but I gave it up because of those fundamental problems on AMD GPUs. You can't believe how messy ROCm is. I wonder if Vulkan compute kernel using SPIR-V would be a better choice.
- dist-epoch 3mo ago
  Co-founder and Chief Compute Officer Anthropic: > I think the thing that we were thinking about originally was whenever we're bringing up a new hardware platform, it's a big effort. It's like a huge thing. And so as we were thinking about this, we started doing our own evaluation of MI 355. You guys generously got us a rack to start working. And we expected this to be kind of a big process. > Our actual experience was we had one engineer who start doing it. They spun up Claude, asked it, hey, bring up this machine, left it going over the weekend. And we ended up with a graph of the actual performance of our leading model on it, just going up and up and up over the weekend. https://x.com/austinsemis/status/2080336781782753635 https://x.com/austinsemis/status/2080336781782753635
  - paoliniluis 3mo ago
    Using AI slop in software that runs ML models is like buying a Ferrari with a motorbike engine. Might be good for demos or as a proof of concept, but that can’t be used for serving LLMs at scale
    - ACCount37 3mo ago
      That would be a very defensible take back in year 2023. Now though? Anthropic has Mythos. That thing's low level code "AI slop" is better than the "meatbag slop" most software developers write, and it can keep cracking at a given problem with persistence. OpenAI has GPT-5.6, and also that rabid dog of an AI model that was last seen out in the wild tearing HuggingFace open. Modern LLMs are very, very capable - not just of writing raw code, but also of persistent, methodical problem solving. Which is what you want to tackle things like "port from an exotic system A to an exotic system B and smoke test the port". Persistently hunting for testable optimizations is a good fit too.
    - InTheArena 3mo ago
      At the risk of noting that this is a PR statement from someone who needs something from AMD - you do realize that you just shat upon the CTO of the largest possible consumer of a technology talking about that technology? It's not 2022 anymore. Just pointing out, there is a reason the large LLMs keep working to try to make it so their own tools can't attack their own moats, by preventing the exact behavior that you dismiss above.
- nolist_policy 3mo ago
  Spir-v is already happening: https://rocm.blogs.amd.com/software-tools-optimization/spir-v-rocm/README.html https://rocm.blogs.amd.com/software-tools-optimization/spir-...
  - stevefan1999 3mo ago
    > PYTORCH_ROCM_ARCH="gfx900;gfx906;gfx908;gfx90a;gfx942;gfx950;\ gfx1030;gfx1100;gfx1101;gfx1102;gfx1103;\ gfx1150;gfx1151;gfx1200;gfx1201" That kicked off my PTSD to setup ROCm on my 6600XT and 9070XT, not even with different containers. I have to build an image for EACH architecture. That's duping 15GB for each arch on my already terribly small SSD.
    - ktm5j 3mo ago
      Well, that appears to be the exact problem that they're trying to solve with spir-v. > The shift SPIR-V enables is from ahead-of-time compilation per target to compile once, just-in-time specialize on device.
- DiskoHexyl 3mo ago
  That depends on the workload. When buying for on-prem inference, MI355x is quite capable. It still requires a ton more ops time, not every dc handles it cooling-wise, and you have to be careful with the model selection, but when it fits, it fits well

nl 3mo ago

Did anyone think RAM prices were coming down soon?

ACCount37 3mo ago
Of course. Wishful thinking is a true staple of human intelligence.
Goronmon 3mo ago
There are plenty of laymen that think "AI bubble pop" means that "AI" usage will go to 0 and then all of a sudden all these hardware and datacenters related companies are buying will become totally unused and flood the market.

swiftcoder 3mo ago

Hey, they are only projected to cost about $5.5million a rack. Pocket change, really

stevefan1999 3mo ago

but you have to rewrite all your software to ROCm. And ROCm, to this day, still sucks. ZLUDA basically crash on high memory demand, and only accounted for ~70% of CUDA API coverage (and it is still buggy). But hey, at least it does run on Rust-CUDA. I'm one of the few who ported it and fixed a few bugs on ZLUDA. I used it to run a simple SHA256 kernel and it ran sure, but I gave it up because of those fundamental problems on AMD GPUs. You can't believe how messy ROCm is. I wonder if Vulkan compute kernel using SPIR-V would be a better choice.

dist-epoch 3mo ago
Co-founder and Chief Compute Officer Anthropic: > I think the thing that we were thinking about originally was whenever we're bringing up a new hardware platform, it's a big effort. It's like a huge thing. And so as we were thinking about this, we started doing our own evaluation of MI 355. You guys generously got us a rack to start working. And we expected this to be kind of a big process. > Our actual experience was we had one engineer who start doing it. They spun up Claude, asked it, hey, bring up this machine, left it going over the weekend. And we ended up with a graph of the actual performance of our leading model on it, just going up and up and up over the weekend. https://x.com/austinsemis/status/2080336781782753635 https://x.com/austinsemis/status/2080336781782753635
- paoliniluis 3mo ago
  Using AI slop in software that runs ML models is like buying a Ferrari with a motorbike engine. Might be good for demos or as a proof of concept, but that can’t be used for serving LLMs at scale
  - ACCount37 3mo ago
    That would be a very defensible take back in year 2023. Now though? Anthropic has Mythos. That thing's low level code "AI slop" is better than the "meatbag slop" most software developers write, and it can keep cracking at a given problem with persistence. OpenAI has GPT-5.6, and also that rabid dog of an AI model that was last seen out in the wild tearing HuggingFace open. Modern LLMs are very, very capable - not just of writing raw code, but also of persistent, methodical problem solving. Which is what you want to tackle things like "port from an exotic system A to an exotic system B and smoke test the port". Persistently hunting for testable optimizations is a good fit too.
  - InTheArena 3mo ago
    At the risk of noting that this is a PR statement from someone who needs something from AMD - you do realize that you just shat upon the CTO of the largest possible consumer of a technology talking about that technology? It's not 2022 anymore. Just pointing out, there is a reason the large LLMs keep working to try to make it so their own tools can't attack their own moats, by preventing the exact behavior that you dismiss above.
nolist_policy 3mo ago
Spir-v is already happening: https://rocm.blogs.amd.com/software-tools-optimization/spir-v-rocm/README.html https://rocm.blogs.amd.com/software-tools-optimization/spir-...
- stevefan1999 3mo ago
  > PYTORCH_ROCM_ARCH="gfx900;gfx906;gfx908;gfx90a;gfx942;gfx950;\ gfx1030;gfx1100;gfx1101;gfx1102;gfx1103;\ gfx1150;gfx1151;gfx1200;gfx1201" That kicked off my PTSD to setup ROCm on my 6600XT and 9070XT, not even with different containers. I have to build an image for EACH architecture. That's duping 15GB for each arch on my already terribly small SSD.
  - ktm5j 3mo ago
    Well, that appears to be the exact problem that they're trying to solve with spir-v. > The shift SPIR-V enables is from ahead-of-time compilation per target to compile once, just-in-time specialize on device.
DiskoHexyl 3mo ago
That depends on the workload. When buying for on-prem inference, MI355x is quite capable. It still requires a ton more ops time, not every dc handles it cooling-wise, and you have to be careful with the model selection, but when it fits, it fits well