4 ms·

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192

> Hardware: TensorWave node equipped with 8 MI300X accelerators, 2 AMD EPYC CPU Processors (192 cores), and 2.3 TB of DDR5 RAM. > MI300X Accelerator: 192GB VRAM, 5.3 TB/s, ~1300 TFLOPS for FP16 > Hardware: Baremetal node with 8 H100 SXM5 accelerators with NVLink, 160 CPU cores, and 1.2 TB of DDR5 RAM. > H100 SXM5 Accelerator: 80GB VRAM, 3.35 TB/s, ~986 TFLOPS for FP16 I really wonder about the pricing. In theory the MI300X is supposed to be cheaper, but whether is that is really the case in practice remains to be seen.

huevosabio 2y ago
RunPod [0] is pricing MI300X at $4.89/hr vs $3.89-4.69/hr for H100s. So, probably around the same price? The tests look promising, though! [0] https://runpod.io/ https://runpod.io/
- latchkey 2y ago
  We are starting at $4.50/hr [0]. The catch is that we won't have availability until mid August. The weird thing on Runpod is the virtual CPUs, you can't run MI300x in virtual machines yet. It is a missing feature that AMD is working on. [0] https://hotaisle.xyz/pricing/ https://hotaisle.xyz/pricing/
sigmoid10 2y ago
It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.
- hyperman1 2y ago
  And this shouldn't be to hard if you know the ins and outs of the hardware and have a reasonable dev team. So why aren't they doing it?
  - rjzzleep 2y ago
    > and have a reasonable dev team probably this.
- huntertwo 2y ago
  Large corporate customers like Microsoft and Meta do not use CUDA. They all use custom software. AMD doesn’t have enough GPUs to sell them yet, that’s the real bottleneck.
  - logicchains 2y ago
    That's a pretty big claim, that Microsoft and Meta have their own proprietary cuda-replacement stack. Do you have any evidence for that claim?
    - versteegen 2y ago
      I'm guessing what they meant is that they use toolchains that are retargetable to other GPUs (and typically compile down to PTX (nVidia assembly language) on nVidia GPUs rather than go through CUDA source -- GCC and clang can both target PTX). For example XLA and most SYSCL toolchains support much more than nVidia.
      - sigmoid10 2y ago
        Even then it's an insanely bold assumption that a company other than Nvidia could build a better framework than CUDA for compiling PTX. Especially since CUDA is already so much C-like. I've never seen anyone go deeper than that outside of academia.
        dexterac 2y ago
        How many customers/consumers will care about services are be built with CUDA? If they need a ChatBot that uses a model with same accuracy and performance as on non-CUDA hardware, would they still want CUDA based hardware?
        sigmoid10 2y ago
        Who is going to build the architecture and compile the device specific kernels? You have to pay those people as well and you can save tons of money and time if you do it with cuda.
- oelang 2y ago
  Microsoft recently announced that they run chatgpt 3.5 & 4 on mi300 on Azure and the price/performance is better. https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd-instinct-mi300x-accelerators-power-microsoft-a.html https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd...
  - sigmoid10 2y ago
    I've used ChatGPT on Azure. It sucks on so many levels, everything about it was clearly enforced by some bean counters who see X dollars for Y flops with zero regard for developers. So choosing AMD here would be about par for the course. There is a reason why everyone at the top is racing to buy Nvidia cards and pay the premium.
    - dexterac 2y ago
      "Everyone" at top is also developing their own chips for inference and providing APIs for customers to not worry about using CUDA. It looks like the price to performance of inference tasks gives providers a big incentive to move away from Nvidia.
      - sigmoid10 2y ago
        There are only like 3 AI building companies who have the tech capability and resources to afford that and 2 of them don't even offer their chips to others or have gone back to Nvidia. The rest is manufacturers desperately trying to get a piece of the pie.
- mistymountains 2y ago
  Unless you develop in CUDA, you can easily train code (e.g. PyTorch) written for training on Nvidia hardware on AMD hardware. You can even keep the .cuda() calls.
  - sigmoid10 2y ago
    In theory. But if you actually work with that in practice, you're already going to have a bad experience installing the drivers. And it's all downhill from there.

huevosabio 2y ago

RunPod [0] is pricing MI300X at $4.89/hr vs $3.89-4.69/hr for H100s. So, probably around the same price? The tests look promising, though! [0] https://runpod.io/ https://runpod.io/

latchkey 2y ago
We are starting at $4.50/hr [0]. The catch is that we won't have availability until mid August. The weird thing on Runpod is the virtual CPUs, you can't run MI300x in virtual machines yet. It is a missing feature that AMD is working on. [0] https://hotaisle.xyz/pricing/ https://hotaisle.xyz/pricing/

sigmoid10 2y ago

It doesn't matter. AMD has offered better compute per dollar for a while now, but noone switched because CUDA is the real reason why all serious ML people use Nvidia. Until AMD picks up the slack on their software side, Nvidia will continue to dominate.

hyperman1 2y ago
And this shouldn't be to hard if you know the ins and outs of the hardware and have a reasonable dev team. So why aren't they doing it?
- rjzzleep 2y ago
  > and have a reasonable dev team probably this.
huntertwo 2y ago
Large corporate customers like Microsoft and Meta do not use CUDA. They all use custom software. AMD doesn’t have enough GPUs to sell them yet, that’s the real bottleneck.
- logicchains 2y ago
  That's a pretty big claim, that Microsoft and Meta have their own proprietary cuda-replacement stack. Do you have any evidence for that claim?
  - versteegen 2y ago
    I'm guessing what they meant is that they use toolchains that are retargetable to other GPUs (and typically compile down to PTX (nVidia assembly language) on nVidia GPUs rather than go through CUDA source -- GCC and clang can both target PTX). For example XLA and most SYSCL toolchains support much more than nVidia.
    - sigmoid10 2y ago
      Even then it's an insanely bold assumption that a company other than Nvidia could build a better framework than CUDA for compiling PTX. Especially since CUDA is already so much C-like. I've never seen anyone go deeper than that outside of academia.
      - dexterac 2y ago
        How many customers/consumers will care about services are be built with CUDA? If they need a ChatBot that uses a model with same accuracy and performance as on non-CUDA hardware, would they still want CUDA based hardware?
        sigmoid10 2y ago
        Who is going to build the architecture and compile the device specific kernels? You have to pay those people as well and you can save tons of money and time if you do it with cuda.
oelang 2y ago
Microsoft recently announced that they run chatgpt 3.5 & 4 on mi300 on Azure and the price/performance is better. https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd-instinct-mi300x-accelerators-power-microsoft-a.html https://www.amd.com/en/newsroom/press-releases/2024-5-21-amd...
- sigmoid10 2y ago
  I've used ChatGPT on Azure. It sucks on so many levels, everything about it was clearly enforced by some bean counters who see X dollars for Y flops with zero regard for developers. So choosing AMD here would be about par for the course. There is a reason why everyone at the top is racing to buy Nvidia cards and pay the premium.
  - dexterac 2y ago
    "Everyone" at top is also developing their own chips for inference and providing APIs for customers to not worry about using CUDA. It looks like the price to performance of inference tasks gives providers a big incentive to move away from Nvidia.
    - sigmoid10 2y ago
      There are only like 3 AI building companies who have the tech capability and resources to afford that and 2 of them don't even offer their chips to others or have gone back to Nvidia. The rest is manufacturers desperately trying to get a piece of the pie.
mistymountains 2y ago
Unless you develop in CUDA, you can easily train code (e.g. PyTorch) written for training on Nvidia hardware on AMD hardware. You can even keep the .cuda() calls.
- sigmoid10 2y ago
  In theory. But if you actually work with that in practice, you're already going to have a bad experience installing the drivers. And it's all downhill from there.