3 ms·

On Dec 6th, AMD released the MI300x. This is effectively their entry into the AIA (ai accelerator is the new GPU) market. The previous generation (MI250) cards

On Dec 6th, AMD released the MI300x. This is effectively their entry into the AIA (ai accelerator is the new GPU) market. The previous generation (MI250) cards were good, but this new release leapfrogs everything. These cards offer 192gb of ram and pretty great performance. 8 cards in a chassis. Great for training and inference. Sure, we all know that the AMD software needs work, but I think the long bet here is on Lisa. Reminds me of the Mac vs. Windows days. If NVIDIA owns this much of the market, then I'd want to work to decentralize my AI business off a single point of failure. So, I've been working to launch a business to buy up as many of these chips as I can and make them available as bare metal rentals, even with IPMI access. This is something that hasn't been done before. Usually these sorts of AMD cards end up in super computers (ie: Frontier) and/or only accessible to a few people. Azure is catching on with their recent product, but that is a big cloud... I believe people will still want private clouds. To start with, we're loading up the MI300x chassis with tons of RAM/nvme, top end AMD CPUs and dual 400G networking. We've also got top end management servers for Ray/k8s/slurm. We are interested in feedback on what people want and can be agile enough to customize our purchases for your needs. My background is that I built a cluster of 150k AMD GPUs across 7 data centers. Deploying and running a lot of compute is something I've gotten very good at. Feel free to reach out.

fragmede 3y ago
that's awesome! what support are you giving for the rewriting that has to happen for things to run on ROCm or whatever instead of CUDA?
- latchkey 3y ago
  Thanks! Because we are just starting out, in order to stay lean, we are focused on the hardware side at this point. The capex/opex involved with deploying this level of compute is rather insane and I feel that it will be too distracting to solve both problems at the same time. I do fully recognize that in order to build the developer flywheel, we need to solve both problems in the long run. For now, there are other great companies that are focused on solving the software side, such as NeuralFabric, EmbeddedLLM, Lamini and MK1. Our intention is to partner and be friends with all of them. There is a lot of room in the picks/shovels ecosystem for all of us.
renewiltord 3y ago
Exciting news. How's software story on this? Currently we are on CUDA, etc. We have nice big AMD Epycs (1GB of cache!) but use Nvidia for GPU. Mi300x at $40k would be tempting, but hard to justify with porting effort required.
- latchkey 3y ago
  I would start to look into using ROCm/HIP to port the code. Some of it can be done automatically and some will need customizations [0]. Then, you rent time on my systems and see what the performance is. If it is good, you rent more, for longer to bring the pricing down. Effectively, we are taking the capex/opex risk off your shoulders. The key point here is that we have actual availability and we aren't just reselling someone else's hardware. Dual 9754's are indeed quite nice. ;-) [0] https://github.com/ROCm/HIPIFY https://github.com/ROCm/HIPIFY
  - renewiltord 3y ago
    Thank you for the link and advice. We're using 9684X for the cache. But dual 9754 does sound good!
    - latchkey 3y ago
      Oh yea, great chip as well. That cache certainly is expensive! Can I ask... what is the usecase for the cache?
      - renewiltord 3y ago
        We use the chip general purpose. HFT simulation code, empirically faster. The GPUs are to train a model. Separate use-case, but we use same hardware for both use-cases so that's why the big cache thing.
sandworm101 3y ago
>> We are interested in feedback on what people want and can be agile enough to customize our purchases for your needs. I would like you to not by pallets of these chips so that I and others may be able to get our hands on a couple for our own personal projects.
- latchkey 3y ago
  It was very difficult for me to get access to these chips and I'm building a whole business around it. It won't be possible for individuals for a long long time, if ever. Maybe in a few years on the secondary market. Case in point... try searching ebay for the previous generation... MI250. If you want access to these chips, you're going to have to rent them. Given that nobody is really renting them currently, at least I'm working to get you access to them at all.
  - 2devnull 3y ago
    What’s to stop the government from taking your business from you for national security reasons?
    - latchkey 3y ago
      Correct me if I'm wrong, but I don't think the US government can take over a business like that. They can take the land where we host the equipment, though if that happens, I guess I'd just move it somewhere else? I did personally sign some documentation (EAR) saying that I wouldn't export the equipment to a whole list of countries and I definitely won't do that. I also won't rent to anyone in those countries either, just to cover my bases. I have high ethics and have no desire to get in any sort of trouble.
    - xcv123 3y ago
      He's not in China
      - latchkey 3y ago
        Very very true.
KennyBlanken 3y ago
"needs work" is an understatement. ROCM only just came out for windows(!) and so far everyone in the LLM and SD scene says that ROCM on linux is a pain in the ass to get working, requires a lot of hackery and override environment variables, is buggy, and also only supports a small handful of cards - whereas CUDA runs on nearly everything NVIDIA makes, from laptop GPUs to datacenter compute cards. And there's very little support on the third party software side at the moment. That "no consumer hardware support" sounds stupid, but people getting into ML/AI, grad students, etc want to be able to mess around and develop/prototype on local hardware they already own. You cannot do that with ROCM, because they only support a very, very small number of consumer-level cards - and very oddly, it's a mix of the very high end and low end, nothing in the middle. AMD is also massively behind hardware-wise with only the current 7xxx series cards that just came out having AI-specific hardware, whereas NVIDIA has had tensor cores for three generations of their cards. But of course AMD can't do anything right, so they nerfed the GPU's graphics processing cores when they added the AI stuff, making the cards a worse deal from a pure gaming standpoint. The 7000 series cars are basically a tiny bump at best, and in many games worse, than the 6000 series equivalents. Their only advantage is better hardware video encoding support and somewhat better power consumption.
- latchkey 3y ago
  All of what you're saying is true and I can't argue that. The only difference is that previously you couldn't even rent time on these super high end AIA's. They were reserved for supercomputers only. Even the MI250x sku is government/research only and even I am unable to buy that over the MI250. So, we need to work on building that developer flywheel. My view is that if you can rent some time on one of these systems, that is a good first step. Nowhere else can you load a model into 192gb of RAM on a top end system.
- wafngar 3y ago
  For AI you have to support PyTorch. That works well now on the big AMD GPUs. Consumer gpu capabilities do not matter here.
  - latchkey 3y ago

fragmede 3y ago

that's awesome! what support are you giving for the rewriting that has to happen for things to run on ROCm or whatever instead of CUDA?

latchkey 3y ago
Thanks! Because we are just starting out, in order to stay lean, we are focused on the hardware side at this point. The capex/opex involved with deploying this level of compute is rather insane and I feel that it will be too distracting to solve both problems at the same time. I do fully recognize that in order to build the developer flywheel, we need to solve both problems in the long run. For now, there are other great companies that are focused on solving the software side, such as NeuralFabric, EmbeddedLLM, Lamini and MK1. Our intention is to partner and be friends with all of them. There is a lot of room in the picks/shovels ecosystem for all of us.

renewiltord 3y ago

Exciting news. How's software story on this? Currently we are on CUDA, etc. We have nice big AMD Epycs (1GB of cache!) but use Nvidia for GPU. Mi300x at $40k would be tempting, but hard to justify with porting effort required.

latchkey 3y ago
I would start to look into using ROCm/HIP to port the code. Some of it can be done automatically and some will need customizations [0]. Then, you rent time on my systems and see what the performance is. If it is good, you rent more, for longer to bring the pricing down. Effectively, we are taking the capex/opex risk off your shoulders. The key point here is that we have actual availability and we aren't just reselling someone else's hardware. Dual 9754's are indeed quite nice. ;-) [0] https://github.com/ROCm/HIPIFY https://github.com/ROCm/HIPIFY
- renewiltord 3y ago
  Thank you for the link and advice. We're using 9684X for the cache. But dual 9754 does sound good!
  - latchkey 3y ago
    Oh yea, great chip as well. That cache certainly is expensive! Can I ask... what is the usecase for the cache?
    - renewiltord 3y ago
      We use the chip general purpose. HFT simulation code, empirically faster. The GPUs are to train a model. Separate use-case, but we use same hardware for both use-cases so that's why the big cache thing.

sandworm101 3y ago

>> We are interested in feedback on what people want and can be agile enough to customize our purchases for your needs. I would like you to not by pallets of these chips so that I and others may be able to get our hands on a couple for our own personal projects.

latchkey 3y ago
It was very difficult for me to get access to these chips and I'm building a whole business around it. It won't be possible for individuals for a long long time, if ever. Maybe in a few years on the secondary market. Case in point... try searching ebay for the previous generation... MI250. If you want access to these chips, you're going to have to rent them. Given that nobody is really renting them currently, at least I'm working to get you access to them at all.
- 2devnull 3y ago
  What’s to stop the government from taking your business from you for national security reasons?
  - latchkey 3y ago
    Correct me if I'm wrong, but I don't think the US government can take over a business like that. They can take the land where we host the equipment, though if that happens, I guess I'd just move it somewhere else? I did personally sign some documentation (EAR) saying that I wouldn't export the equipment to a whole list of countries and I definitely won't do that. I also won't rent to anyone in those countries either, just to cover my bases. I have high ethics and have no desire to get in any sort of trouble.
  - xcv123 3y ago
    He's not in China
    - latchkey 3y ago
      Very very true.

KennyBlanken 3y ago

"needs work" is an understatement. ROCM only just came out for windows(!) and so far everyone in the LLM and SD scene says that ROCM on linux is a pain in the ass to get working, requires a lot of hackery and override environment variables, is buggy, and also only supports a small handful of cards - whereas CUDA runs on nearly everything NVIDIA makes, from laptop GPUs to datacenter compute cards. And there's very little support on the third party software side at the moment. That "no consumer hardware support" sounds stupid, but people getting into ML/AI, grad students, etc want to be able to mess around and develop/prototype on local hardware they already own. You cannot do that with ROCM, because they only support a very, very small number of consumer-level cards - and very oddly, it's a mix of the very high end and low end, nothing in the middle. AMD is also massively behind hardware-wise with only the current 7xxx series cards that just came out having AI-specific hardware, whereas NVIDIA has had tensor cores for three generations of their cards. But of course AMD can't do anything right, so they nerfed the GPU's graphics processing cores when they added the AI stuff, making the cards a worse deal from a pure gaming standpoint. The 7000 series cars are basically a tiny bump at best, and in many games worse, than the 6000 series equivalents. Their only advantage is better hardware video encoding support and somewhat better power consumption.

latchkey 3y ago
All of what you're saying is true and I can't argue that. The only difference is that previously you couldn't even rent time on these super high end AIA's. They were reserved for supercomputers only. Even the MI250x sku is government/research only and even I am unable to buy that over the MI250. So, we need to work on building that developer flywheel. My view is that if you can rent some time on one of these systems, that is a good first step. Nowhere else can you load a model into 192gb of RAM on a top end system.
wafngar 3y ago
For AI you have to support PyTorch. That works well now on the big AMD GPUs. Consumer gpu capabilities do not matter here.
- latchkey 3y ago