5 ms·

1x MI300x has 192GB HBM3. 1x MI325x has 256GB HBM3e. They cost less, you can fit more into a rack and you can buy/deploy at least the 300's today and 325's

1x MI300x has 192GB HBM3. 1x MI325x has 256GB HBM3e. They cost less, you can fit more into a rack and you can buy/deploy at least the 300's today and 325's early next year. AMD and library software performance for AI is improving daily [0]. I'm still trying to wrap my head around how these companies think they are going to do well in this market without more memory. [0] https://blog.vllm.ai/2024/10/23/vllm-serving-amd.html https://blog.vllm.ai/2024/10/23/vllm-serving-amd.html

wmf 2y ago
Groq and Cerebras only make sense at massive scale which is why I guess they pivoted to being API providers so they can amortize the hardware over many customers.
- latchkey 2y ago
  Correct except that massive scale doesn't work cause it just uses up exponentially more power/space/resources. They also have a very limited use case... if things ever shift away from LLM's and into another form of engineering that their hardware does not support, what are they going to do? Just keep deploying hardware? Slippery slope.
krasin 2y ago
> I'm still trying to wrap my head around how these companies think they are going to do well in this market without more memory. Cerebras and Groq provide the fastest (by an order of magnitude) inference. This is very useful for certain workflows, which require low-latency feedback: audio chat with LLM, robotics, etc. Outside that narrow niche, AMD stuff seems to be the only contender to NVIDIA, at the moment.
- latchkey 2y ago
  > Cerebras and Groq provide the fastest (by an order of magnitude) inference. Only on smaller models, their numbers are all 70b in the article. Those numbers also need to be adjusted for the comparable amounts of capex+opex costs. If the costs are so high that they have to subsidize the usage/results, then they are just going to run out of money, fast.
  - krasin 2y ago
    > Only on smaller models, their numbers are all 70b in the article. No, they are 5x-10x faster for all the model sizes (because it's all just running from SRAM and they have more of it than NVIDIA/AMD), even though they benchmarked just up to 70B. > Those numbers also need to be adjusted for the comparable amounts of capex+opex costs. If the costs are so high that they have to subsidize the usage/results, then they are just going to run out of money, fast. True. Although, for some workloads, fast enough inference is a strict prerequisite and GPUs just don't cut it.
    - cma 2y ago
      Cerebras had less chip perimeter to hook up external memory I/O and is memory capacity limited with just SRAM. SRAM circuit size hasn't been scaling nearly as well as logic on recent nodes, but if scaling there had continued to from when Cerebras started it may have worked out better. They'll probably still have to do advanced packaging putting HBM on top to save things. They could maybe enable some cool real time inference stuff like VR SORA, but that doesn't seem like much of a product market for the cost yet. Maybe something heavy on inference iteration like an o1 style model that trades training time for more inference, used to process earnings reports the fastest or something zero sum latency war like that will be a viable market. A real time use case that may be viable with cerebras first could be with flexible robotics in ad hoc latency sensitive environments, maybe warfare. If models keep lasting ~year timescales could we ever see people going with ROM chips for the weights instead of memory? Has density and speed kept up there? Lots of stuff uses identical elements to help make the masks more cheaply, so I don't think you could use something like EUV for ROM where every few um^2 of die is distinct.
      - krasin 2y ago
        > If models keep lasting ~year timescales could we ever see people going with ROM chips for the weights instead of memory? Before ROM, there's a step where HBM for weights is replaced with Flash or Optane (but still high bandwidth, on top of the chip) and KV cache lives in SRAM - for small batch sizes, that would actually be decently cheap. In this case, even if weights change weekly, it's not a big deal at all.
- hhdhdbdb 2y ago
  How is this a narrow niche? Chain of thought type operations is in this "niche". Also anything where the value is in the follow up chat not the one shot.
arisAlexis 2y ago
The article explains in depth the issues with memory, did you read through ?
YetAnotherNick 2y ago
2x 80GB A100 is better in all the metrics than MI300x while being cheaper.

wmf 2y ago

Groq and Cerebras only make sense at massive scale which is why I guess they pivoted to being API providers so they can amortize the hardware over many customers.

latchkey 2y ago
Correct except that massive scale doesn't work cause it just uses up exponentially more power/space/resources. They also have a very limited use case... if things ever shift away from LLM's and into another form of engineering that their hardware does not support, what are they going to do? Just keep deploying hardware? Slippery slope.

krasin 2y ago

> I'm still trying to wrap my head around how these companies think they are going to do well in this market without more memory. Cerebras and Groq provide the fastest (by an order of magnitude) inference. This is very useful for certain workflows, which require low-latency feedback: audio chat with LLM, robotics, etc. Outside that narrow niche, AMD stuff seems to be the only contender to NVIDIA, at the moment.

latchkey 2y ago
> Cerebras and Groq provide the fastest (by an order of magnitude) inference. Only on smaller models, their numbers are all 70b in the article. Those numbers also need to be adjusted for the comparable amounts of capex+opex costs. If the costs are so high that they have to subsidize the usage/results, then they are just going to run out of money, fast.
- krasin 2y ago
  > Only on smaller models, their numbers are all 70b in the article. No, they are 5x-10x faster for all the model sizes (because it's all just running from SRAM and they have more of it than NVIDIA/AMD), even though they benchmarked just up to 70B. > Those numbers also need to be adjusted for the comparable amounts of capex+opex costs. If the costs are so high that they have to subsidize the usage/results, then they are just going to run out of money, fast. True. Although, for some workloads, fast enough inference is a strict prerequisite and GPUs just don't cut it.
  - cma 2y ago
    Cerebras had less chip perimeter to hook up external memory I/O and is memory capacity limited with just SRAM. SRAM circuit size hasn't been scaling nearly as well as logic on recent nodes, but if scaling there had continued to from when Cerebras started it may have worked out better. They'll probably still have to do advanced packaging putting HBM on top to save things. They could maybe enable some cool real time inference stuff like VR SORA, but that doesn't seem like much of a product market for the cost yet. Maybe something heavy on inference iteration like an o1 style model that trades training time for more inference, used to process earnings reports the fastest or something zero sum latency war like that will be a viable market. A real time use case that may be viable with cerebras first could be with flexible robotics in ad hoc latency sensitive environments, maybe warfare. If models keep lasting ~year timescales could we ever see people going with ROM chips for the weights instead of memory? Has density and speed kept up there? Lots of stuff uses identical elements to help make the masks more cheaply, so I don't think you could use something like EUV for ROM where every few um^2 of die is distinct.
    - krasin 2y ago
      > If models keep lasting ~year timescales could we ever see people going with ROM chips for the weights instead of memory? Before ROM, there's a step where HBM for weights is replaced with Flash or Optane (but still high bandwidth, on top of the chip) and KV cache lives in SRAM - for small batch sizes, that would actually be decently cheap. In this case, even if weights change weekly, it's not a big deal at all.
hhdhdbdb 2y ago
How is this a narrow niche? Chain of thought type operations is in this "niche". Also anything where the value is in the follow up chat not the one shot.

arisAlexis 2y ago

The article explains in depth the issues with memory, did you read through ?

YetAnotherNick 2y ago

2x 80GB A100 is better in all the metrics than MI300x while being cheaper.