3 ms·

When are we likely to see MI300 available for rent and what would the cost per hour be?

photonbucket 3y ago
Which companies that already rent out H100s would risk the ire of nvidia providing them less capacity next time? edit: It seems cudocompute and dataknox is willing to rent them out
latchkey 3y ago
Thanks for asking. Soon! Estimated ship date is 2/29. Should arrive around 5 days after that. We are going to fly to the data center, get the boxes all set up and start to launch things. Pricing is TBD since there isn't really a comparable and we expect that since these GPUs are so new, people will want to do a lot of testing before they are willing to commit. Therefore, we are looking for partnerships over just customers. Disclosure: Check out my profile, I'm building a white glove bare metal service that is focused on only top end AMD compute and specifically the MI300x (and future generations). Feel free to reach out via email.
- brucethemoose2 3y ago
  We can't quite afford an 8x bare metal instance, but I am pinning this for later. Think ya'll will ever offer single MI300X instances?
  - latchkey 3y ago
    Yes! Like I said above, we fully expect people to be toe dipping at first given how new this hardware is. We offer single GPU access through a VM and PCIe pass through. Either we can provide you a basic Ubuntu VM with everything loaded into it (with regards to drivers and such) or you can send us your VM image and set it up however you like. Then you'll just have direct ssh access into it.
    - brucethemoose2 3y ago
      OK that sounds lovely. We are but a poor startup, but I am hoping we can afford to reach out to y'all soon. Maybe even share workflows/stuff that works for us. If some ML (unsloth? lorax? sglang?) work on a Mi300 with a little tinkering, that would be spectacular.
      - latchkey 3y ago
        Thanks! Again, partnerships over customers. If you're experienced and have the technical chops to make a MI300x sing, we want to work with you. Our model is that we are the capex/opex investor for businesses. As much as I love software, Hot Aisle is more of a hardware business. Running super high end large scale compute is an extreme challenge in itself. We are less interested in building the software side of things and want to foster those who can focus on that side. https://github.com/unslothai/unsloth/issues/160 https://github.com/unslothai/unsloth/issues/160 https://github.com/search?q=repo%3Apredibase%2Florax+rocm&type=code https://github.com/search?q=repo%3Apredibase%2Florax+rocm&ty... https://github.com/sgl-project/sglang/issues/157 https://github.com/sgl-project/sglang/issues/157 https://github.com/casper-hansen/AutoAWQ https://github.com/casper-hansen/AutoAWQ (supports rocm)
mlyle 3y ago
MI300 is early access in Azure right now -- https://techcommunity.microsoft.com/t5/azure-high-performance-computing/azure-announces-new-ai-optimized-vm-series-featuring-amd-s/ba-p/3980770 https://techcommunity.microsoft.com/t5/azure-high-performanc...
- latchkey 3y ago
  This sadly doesn't mean that they are actually available. I haven't seen anyone saying they are using them. Obviously, that doesn't mean much either, but I'd at least hope to see someone post some performance benchmarks.
  - mlyle 3y ago
    Early access means NDA.
    - latchkey 3y ago
      Well, that's no fun at all.

Which companies that already rent out H100s would risk the ire of nvidia providing them less capacity next time? edit: It seems cudocompute and dataknox is willing to rent them out

latchkey 3y ago

Thanks for asking. Soon! Estimated ship date is 2/29. Should arrive around 5 days after that. We are going to fly to the data center, get the boxes all set up and start to launch things. Pricing is TBD since there isn't really a comparable and we expect that since these GPUs are so new, people will want to do a lot of testing before they are willing to commit. Therefore, we are looking for partnerships over just customers. Disclosure: Check out my profile, I'm building a white glove bare metal service that is focused on only top end AMD compute and specifically the MI300x (and future generations). Feel free to reach out via email.

brucethemoose2 3y ago
We can't quite afford an 8x bare metal instance, but I am pinning this for later. Think ya'll will ever offer single MI300X instances?
- latchkey 3y ago
  Yes! Like I said above, we fully expect people to be toe dipping at first given how new this hardware is. We offer single GPU access through a VM and PCIe pass through. Either we can provide you a basic Ubuntu VM with everything loaded into it (with regards to drivers and such) or you can send us your VM image and set it up however you like. Then you'll just have direct ssh access into it.
  - brucethemoose2 3y ago
    OK that sounds lovely. We are but a poor startup, but I am hoping we can afford to reach out to y'all soon. Maybe even share workflows/stuff that works for us. If some ML (unsloth? lorax? sglang?) work on a Mi300 with a little tinkering, that would be spectacular.
    - latchkey 3y ago
      Thanks! Again, partnerships over customers. If you're experienced and have the technical chops to make a MI300x sing, we want to work with you. Our model is that we are the capex/opex investor for businesses. As much as I love software, Hot Aisle is more of a hardware business. Running super high end large scale compute is an extreme challenge in itself. We are less interested in building the software side of things and want to foster those who can focus on that side. https://github.com/unslothai/unsloth/issues/160 https://github.com/unslothai/unsloth/issues/160 https://github.com/search?q=repo%3Apredibase%2Florax+rocm&type=code https://github.com/search?q=repo%3Apredibase%2Florax+rocm&ty... https://github.com/sgl-project/sglang/issues/157 https://github.com/sgl-project/sglang/issues/157 https://github.com/casper-hansen/AutoAWQ https://github.com/casper-hansen/AutoAWQ (supports rocm)

mlyle 3y ago

MI300 is early access in Azure right now -- https://techcommunity.microsoft.com/t5/azure-high-performance-computing/azure-announces-new-ai-optimized-vm-series-featuring-amd-s/ba-p/3980770 https://techcommunity.microsoft.com/t5/azure-high-performanc...

latchkey 3y ago
This sadly doesn't mean that they are actually available. I haven't seen anyone saying they are using them. Obviously, that doesn't mean much either, but I'd at least hope to see someone post some performance benchmarks.
- mlyle 3y ago
  Early access means NDA.
  - latchkey 3y ago
    Well, that's no fun at all.