3 ms·

What's the cost benefit vs. Nvidia? Is it cheaper?

gdiamos 3y ago
Available in orders of up to 10,000 GPUs today - no shortage More than 10x cheaper than allocating machines on a tier 1 cloud - AWS, Azure, GCP, Oracle, etc More memory - 128GB HBM per GPU - means bigger models fit for training/inference without the nightmare of model parallelism over MPI/infiniband/etc Longer term - finetuning optimizations
- mardifoufs 3y ago
  Ah! The memory sounds interesting. How would that compare to similar Nvidia hardware w.r.t cost assuming the hardware was available? Does AMD provide something similar to nvlink, and even libraries like cudnn? Also, last I checked none of the public clouds offered any of the latest gens MI GPUs, so I wasn't aware that it had good availability! Azure had a preview but I'll look more into it now. Thank you for your answer btw!
  - gdiamos 3y ago
    Yeah getting around the no public cloud thing was really annoying. We had to build our own datacenter. On the plus side, it was drastically cheaper and now we can just slot in machines. I would prefer that a tier 1 cloud made MI GPUs available though. It would make it so much more accessible.
    - bitcoinmoney 3y ago
      Are you releasing your software stack to the public?
      - gdiamos 3y ago
        It's available now with an enterprise license, because we validate that it is running correctly on a system we help configure. We will open source pieces of it over time. Our strategy is open source functional core. Eventually we will have an open source dev environment that runs on a personal scale computer. We already have this for some configurations, but we don't do enough testing to ensure perf/functionality on many different systems. We are mainly bottlenecked by resources as a 12-person startup. We have released some open source SDKs here: https://github.com/orgs/lamini-ai https://github.com/orgs/lamini-ai This class has some training recipe code: https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... One thing I'd like to push back to open source is the scale out AMD SLURM support.
  - gdiamos 3y ago
    See the memory size comparison (GB) in this table: https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#Tesla https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
    - QuarterReptile 3y ago
      It blows my mind that A100 and H100 are each safely below 1000W power draw.
gardnr 3y ago
The classic economic benefits of competition: * Drives down price * Enhances product features (I see them competing on VRAM first) * Helps to insulates buyers from supply issues Nvidia has kneecapped their consumer grade hardware to ensure the gaming market still has scraps to buy in spite of crypto mining and the AI gold rush. All AMD would have to do to eat into Nvidia marketshare is remove the hardware locks in low-end cards and ship one with 64GB+ of VRAM. This of course would only work if they have comparable/usable software support. Any improvements to ROCm will be a boon for any company that doesn't already have or can't afford huge farms of high-end Nvidia chips.
light_hue_1 3y ago
You simply cannot buy nvidia GPUs at scale at the moment. We're getting quotes that are many months out, sometimes even a year+ out.
- gdiamos 3y ago
  We kept hearing 52 weeks for new shipments.

Available in orders of up to 10,000 GPUs today - no shortage More than 10x cheaper than allocating machines on a tier 1 cloud - AWS, Azure, GCP, Oracle, etc More memory - 128GB HBM per GPU - means bigger models fit for training/inference without the nightmare of model parallelism over MPI/infiniband/etc Longer term - finetuning optimizations

mardifoufs 3y ago
Ah! The memory sounds interesting. How would that compare to similar Nvidia hardware w.r.t cost assuming the hardware was available? Does AMD provide something similar to nvlink, and even libraries like cudnn? Also, last I checked none of the public clouds offered any of the latest gens MI GPUs, so I wasn't aware that it had good availability! Azure had a preview but I'll look more into it now. Thank you for your answer btw!
- gdiamos 3y ago
  Yeah getting around the no public cloud thing was really annoying. We had to build our own datacenter. On the plus side, it was drastically cheaper and now we can just slot in machines. I would prefer that a tier 1 cloud made MI GPUs available though. It would make it so much more accessible.
  - bitcoinmoney 3y ago
    Are you releasing your software stack to the public?
    - gdiamos 3y ago
      It's available now with an enterprise license, because we validate that it is running correctly on a system we help configure. We will open source pieces of it over time. Our strategy is open source functional core. Eventually we will have an open source dev environment that runs on a personal scale computer. We already have this for some configurations, but we don't do enough testing to ensure perf/functionality on many different systems. We are mainly bottlenecked by resources as a 12-person startup. We have released some open source SDKs here: https://github.com/orgs/lamini-ai https://github.com/orgs/lamini-ai This class has some training recipe code: https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... One thing I'd like to push back to open source is the scale out AMD SLURM support.
- gdiamos 3y ago
  See the memory size comparison (GB) in this table: https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#Tesla https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces...
  - QuarterReptile 3y ago
    It blows my mind that A100 and H100 are each safely below 1000W power draw.

gardnr 3y ago

The classic economic benefits of competition: * Drives down price * Enhances product features (I see them competing on VRAM first) * Helps to insulates buyers from supply issues Nvidia has kneecapped their consumer grade hardware to ensure the gaming market still has scraps to buy in spite of crypto mining and the AI gold rush. All AMD would have to do to eat into Nvidia marketshare is remove the hardware locks in low-end cards and ship one with 64GB+ of VRAM. This of course would only work if they have comparable/usable software support. Any improvements to ROCm will be a boon for any company that doesn't already have or can't afford huge farms of high-end Nvidia chips.

light_hue_1 3y ago

You simply cannot buy nvidia GPUs at scale at the moment. We're getting quotes that are many months out, sometimes even a year+ out.

gdiamos 3y ago
We kept hearing 52 weeks for new shipments.