4 ms·

AMD is coming out with the MI300x on Dec 6th, it has 192 GB of HBM3 and you put 8 of them in a chassis all connected to each other with super fast connections (

AMD is coming out with the MI300x on Dec 6th, it has 192 GB of HBM3 and you put 8 of them in a chassis all connected to each other with super fast connections (5.2 TB/s of memory bandwidth) and each chassis is networked with 800-900G connections. We don't know what the tflops will be for the 300x yet, but it seems like the limitations here are more about memory than it is about tflops.

joshhart 3y ago
Will be a long time before a large number of these are available in clouds though, not until summer at least.
- latchkey 3y ago
  I predict sooner, but that's cause it is what I'm working on now. ;-)
buildbot 3y ago
Not sure of the relationship between MI300x and MI300, but MI300 is claimed to be 2.2 PFlops FP8 (with structured sparsity, which I did not know AMD was implementing!!!): https://www.amd.com/en/claims/instinct https://www.amd.com/en/claims/instinct
- latchkey 3y ago
  There are 3 versions of MI300. X is pure gpu, A is more like an APU (cpu+gpu), C is pure cpu.
belval 3y ago
Those are all impressive numbers, but did they actually fix their software stack? Are there people here that have actually trained with those and if so how was the experience? I'm genuinely curious because it seems like every other AI Accelerator/GPU thread on HN mentions MI300X, yet I haven't talked to/met anyone who trained anything with AMD.
- latchkey 3y ago
  The MI300x aren't released yet, that's why it isn't popular. Azure is building a cloud [0]. I know someone else personally who's building a cloud too. As for training, it has been happening a bit on MI250's [1]. There is also A LOT that happens that doesn't make the larger news [2] and [3] and [4]. [0] https://techcommunity.microsoft.com/t5/azure-high-performance-computing/azure-announces-new-ai-optimized-vm-series-featuring-amd-s/ba-p/3980770 https://techcommunity.microsoft.com/t5/azure-high-performanc... [1] https://www.databricks.com/blog/training-llms-scale-amd-mi250-gpus https://www.databricks.com/blog/training-llms-scale-amd-mi25... [2] https://www.amd.com/en/resources/case-studies/kt-cloud.html https://www.amd.com/en/resources/case-studies/kt-cloud.html [3] https://www.lamini.ai/blog/lamini-llm-finetuning-on-amd-rocm-a-technical-recipe https://www.lamini.ai/blog/lamini-llm-finetuning-on-amd-rocm... [4] https://twitter.com/cHHillee/status/1730293330213531844 https://twitter.com/cHHillee/status/1730293330213531844
- latchkey 3y ago
  https://news.ycombinator.com/item?id=38479616 https://news.ycombinator.com/item?id=38479616
alfalfasprout 3y ago
The issue with AMD (and frankly AWS' custom accelerators) is more on the software side usually. ROCM is getting better but still lags far behind CUDA in adoption and often performance. With things like inferentia, although the performance numbers they have reported are really impressive-- in practice, model compilation is super finnicky. I'd love to see how seamless they can make usage of trainium.
- latchkey 3y ago
  What you said gets repeated many times. Fact is that AMD was caught out of the initial AI race and is catching up. Lisa has said over and over again that she is committed to it.

joshhart 3y ago

Will be a long time before a large number of these are available in clouds though, not until summer at least.

latchkey 3y ago
I predict sooner, but that's cause it is what I'm working on now. ;-)

buildbot 3y ago

Not sure of the relationship between MI300x and MI300, but MI300 is claimed to be 2.2 PFlops FP8 (with structured sparsity, which I did not know AMD was implementing!!!): https://www.amd.com/en/claims/instinct https://www.amd.com/en/claims/instinct

latchkey 3y ago
There are 3 versions of MI300. X is pure gpu, A is more like an APU (cpu+gpu), C is pure cpu.

belval 3y ago

Those are all impressive numbers, but did they actually fix their software stack? Are there people here that have actually trained with those and if so how was the experience? I'm genuinely curious because it seems like every other AI Accelerator/GPU thread on HN mentions MI300X, yet I haven't talked to/met anyone who trained anything with AMD.

latchkey 3y ago
The MI300x aren't released yet, that's why it isn't popular. Azure is building a cloud [0]. I know someone else personally who's building a cloud too. As for training, it has been happening a bit on MI250's [1]. There is also A LOT that happens that doesn't make the larger news [2] and [3] and [4]. [0] https://techcommunity.microsoft.com/t5/azure-high-performance-computing/azure-announces-new-ai-optimized-vm-series-featuring-amd-s/ba-p/3980770 https://techcommunity.microsoft.com/t5/azure-high-performanc... [1] https://www.databricks.com/blog/training-llms-scale-amd-mi250-gpus https://www.databricks.com/blog/training-llms-scale-amd-mi25... [2] https://www.amd.com/en/resources/case-studies/kt-cloud.html https://www.amd.com/en/resources/case-studies/kt-cloud.html [3] https://www.lamini.ai/blog/lamini-llm-finetuning-on-amd-rocm-a-technical-recipe https://www.lamini.ai/blog/lamini-llm-finetuning-on-amd-rocm... [4] https://twitter.com/cHHillee/status/1730293330213531844 https://twitter.com/cHHillee/status/1730293330213531844
latchkey 3y ago
https://news.ycombinator.com/item?id=38479616 https://news.ycombinator.com/item?id=38479616

alfalfasprout 3y ago

The issue with AMD (and frankly AWS' custom accelerators) is more on the software side usually. ROCM is getting better but still lags far behind CUDA in adoption and often performance. With things like inferentia, although the performance numbers they have reported are really impressive-- in practice, model compilation is super finnicky. I'd love to see how seamless they can make usage of trainium.

latchkey 3y ago
What you said gets repeated many times. Fact is that AMD was caught out of the initial AI race and is catching up. Lisa has said over and over again that she is committed to it.