6 ms·

Great post. The ethernet section is especially interesting to me. I'm building a cluster of 16x Dell XE9680's (128 AMD MI300x GPUs) [0], with 8x 2p200G broadco

Great post. The ethernet section is especially interesting to me. I'm building a cluster of 16x Dell XE9680's (128 AMD MI300x GPUs) [0], with 8x 2p200G broadcom cards (running at 400G), all connected to a single Dell PowerSwitch Z9864F-ON, which should prevent any slowness. It will be connected over rocev2 [1]. We're going with ethernet because we believe in open standards, and few talk about the fact that the lead time on IB was last quoted to me at 50+ weeks. As kind of mentioned in the article, if you can't even deploy a cluster the speed of the network means less and less. I can't wait to do some benchmarking on the system to see if we run into similar issues or not. Thankfully, we have a great Dell partnership, with full support, so I believe that we are well covered in terms of any potential issues. Our datacenter is 100% green and low PUE and we are very proud of that as well. Hope to announce which one soon. [0] https://hotaisle.xyz/compute/ [1] https://hotaisle.xyz/networking/

omneity 2y ago
Did you procure the servers directly from Dell or through a distribution partner?
- latchkey 2y ago
  Kind of both. We started talking to Dell first and then they introduced us to Advizex. We are now effectively partnered with both companies, which is fantastic as the Advizex team have ex-Dell people working directly with us. We are lucky to have two whole teams of super talented people helping us out on this journey.
logicchains 2y ago
Meta had success building an ethernet cluster on Arista 7800 with Wedge400 and Minipack2 OCP rack switches.: https://www.datacenterdynamics.com/en/news/meta-reveals-details-of-two-new-24k-gpu-ai-clusters/ https://www.datacenterdynamics.com/en/news/meta-reveals-deta...
- latchkey 2y ago
  I actually have a call with Arista next week to learn more about their solutions. Especially those 7800's. They look awesome. One "problem" we have right now is that our cluster cannot support more than 128 GPUs. If we wanted to scale with Dell, we'd have to buy 6x more Z9864F to add one more cluster, which is crazy expensive and complicated. I want to see if Arista has something that can help us. That said, I also have to find a customer that wants more than 128 MI300x and that hasn't happened... yet.
csmpltn 2y ago
> "Our datacenter is 100% green" Cool, where can I read more about this? How do you power your DC?
- walrus01 2y ago
  Plenty of datacenters that are somewhere generally in the pacific NW can claim to be "Green" because their power supply is entirely hydroelectric. https://www.nwd.usace.army.mil/CRWM/CR-Dams/ https://www.nwd.usace.army.mil/CRWM/CR-Dams/ Many of those areas also happen to have the lowest $ per kWh electricity in North American, the only lower rate is available near a few hydroelectric dams in Quebec.
  - latchkey 2y ago
    Previously, I had multiple data centers in Quincy, WA. Those were hydro green. It is an area that hosts a whole multitude of big hyperscaler companies.
  - mulmen 2y ago
    As anyone who has driven through the Columbia River Basin can tell you wind power is also abundant in Washington. The grid is very clean here but it’s certainly not purely hydro.
  - dlkf 2y ago
    Why is green in scare-quotes?
    - uncertainrhymes 2y ago
      People make the argument that is a giant datacenter is consuming 50% of some local hydro installation, everyone else is town is buying something else that is less green. It opens up questions about grids and market efficiency, so your mileage may vary.
      - sangnoir 2y ago
        > People make the argument that is a giant datacenter is consuming 50% of some local hydro installation, everyone else is town is buying something else that is less green. I don't think that's a cogent argument. It's akin to saying a vegan commune in a small is buying is buying up 50% of the vegan food, "forcing" others to buy meat-products, and framing this to cast doubts on whether they are truly vegan. Consumers aren't in a position to solve supply problems.
dpflan 2y ago
What are you using these for? Providing compute for customers that want to train/infer? What is the level of interest and level of success customers are seeing using these services Hot Aisle offers?
- latchkey 2y ago
  We are a bare metal compute offering. They can be used for whatever people want to use them for (within legal limits, of course). Interest is much higher now that we've started to work with teams publishing benchmarks which show that H100's have a nice competitor [0]. I'll admit, it is still early days. We just finished up another free compute [1] two week stint with a benchmarking team. One thing we discovered is that saving checkpoints is slow AF. I'm guessing an issue with ROCm. Hopefully get that resolved soon. Now we are in the process of onboarding the next team. [0] https://hotaisle.xyz/benchmarks-and-analysis/ [1] https://hotaisle.xyz/free-compute-offer/
  - rvnx 2y ago
    > We would love to offer hourly on-demand rates for individual GPUs, but we can't do so at this time due to a limitation in the ROCm/AMD drivers. This limitation prevents PCIe pass-through to a virtual machine, making multi-tenancy impossible. AMD is aware of this issue and has committed to resolving it. One idea to help you: Are you sure you need a virtual machine ? Couldn't you boot the machines under PXE to solve the imaging problem ? Essentially you have TFTP server that gives a Linux image and boot on it directly
    - latchkey 2y ago
      1 chassis, 8 gpus. We want to be able to break that chassis up into individual GPUs and allocate 1 GPU to 1 "machine". I previously PXE booted 20,000 individual playstation 5 diskless blades and I'm not sure how PXE would solve this. The only alternative right now is to do what runpod (and AMD's aac) are doing and do docker containers. But that has the limitation of docker in docker, so people end up having to repackage everything. You also can't easily run different ROCm versions since that comes from the host, and if you have 8 people on a single chassis... it becomes a nightmare to manage it. We're just patiently waiting for AMD to fix the problem.
deleted 2y ago
[deleted]
rlupi 2y ago
https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/network-fabrics.html https://docs.nvidia.com/dgx-superpod/reference-architecture-... NVIDIA large GPU supercomputers have separate compute-networking (between GPUs) and storage-networking (storage to GPUs, or storage to SSD, SSD to GPUs with CPU assistance). This helps avoid networking issues, even more if not using Infiniband. From what I read here and on your website, you don't go that route. I haven't found the equivalent system level reference architecture for MI300x from AMD. I wonder if you have a link to a public document where AMD provides guidance about this choice?
- derefr 2y ago
  > even more if not using Infiniband It's interesting that the above "HPC reference architecture" shows a GPU-to-GPU Infiniband fabric, despite Nvidia also nominally pushing NVLink Switch (https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/) for the HPC use-case.
  - bee_rider 2y ago
    How does NVLink work? Because I already know MPI and I’m not going to learn anything else, lol. Edit: after googling it looks like OpenMPI has some NVLink support, so maybe it is OK.
    - zxexz 2y ago
      I use OpenMPI with no issues over multiple H100 nodes and A100 nodes, with multiple infiniband 200G and ethernet 100G/200G networks, and RDMA (though using mellanox instead of broadcom cards, but afaik broadcom supports this just the same). Side note, make sure you compile nvidia_peermem correctly if you want GDRMA to work :)
      - latchkey 2y ago
        No issues, except this minor bit of arcane knowledge that is missing from SO. :)
RobRivera 2y ago
Nice! Whats benchmark standard these days? Still Superbench or yall have something inhouse? Re: lead time quote :O I guess I got spoiled working for one of the major cloud vendors. The thought of poor b2b vendor support never entered my risk matrix. If you own your own cluster, the network bottleneck becomes less a dollar cost I suppose, since you arent being charged a premium to rent someone elses compute
- latchkey 2y ago
  We don't run benchmarks ourselves. We donate the expensive Ferrari worth of compute for others to do it. This is the most unbiased way I could think of getting useful real world data to share. The 3rd team just finished up and the 4th is getting started now. I've got 23 others in the wings. https://hotaisle.xyz/free-compute-offer/ https://hotaisle.xyz/free-compute-offer/
  - jononor 2y ago
    This seems like an excellent strategy in terms of marketing and building mindshare for the platform (as you mention elsewhere, it is an underdog).
    - latchkey 2y ago
      Thank you for the kind words.
pheatherlite 2y ago
Being out of the loop for awhile. Has amd made anything similar to cuda? Are cots frameworks such as pytorch and tensorflow on par when running on amd hardware? What makes investing in amd cluster/chips worthwhile?
- jononor 2y ago
  ROCm is the framework. Both PyTorch and Tensorflow have versions that support it. Have no experience with it, so cannot say how it works in practice.
JackYoustra 2y ago
Why xeon instead of epyc?
- latchkey 2y ago
  Sadly, the only option Dell supports today. To get to market the fastest, they took their existing H100 chassis solution, swapped out the GPUs/baseboard and called it a day.
startupsfail 2y ago
It seems the reliability, speed and scalability drops with the Ethernet are somewhat manageable. To quote the article - From our tests, we found that Infiniband was systematically outperforming Ethernet interconnects in terms of speed. When using 16 nodes / 128 GPUs, the difference varied from 3% to 10% in terms of distributed training throughput[1]. The gap was widening as we were adding more nodes: Infiniband was scaling almost linearly, while other interconnects scaled less efficiently. And then they do mention that the research team needs to debug unexplained failures on Ethernet that they’ve not seen on Infiniband. This actually can be the expensive part. Particularly if the failures are silent and cause numerical errors only.
- latchkey 2y ago
  A single switch should mitigate some of the throughput issues. As for issues, this is why I have a full professional support contract with Dell and Advizex. If there are issues in the gear, they will step in to help out. Especially on the switch, since it is a spof, we went with a 4 hour window.
teaearlgraycold 2y ago
We just set up a small cluster of our own. We’re not using infiniband but it didn’t seem like it would be a 50 week lead time to setting it up. Where did you get that number?
- latchkey 2y ago
  I hear a couple issues with your comment... "small cluster", "we're not using infinband" I'm only saying what was quoted to me. The cards are easy to get, it is the switches that are more difficult.
  - teaearlgraycold 2y ago
    I saw plenty of used switches available. Are current gen switches necessary?
    - latchkey 2y ago
      https://news.ycombinator.com/item?id=40951133 https://news.ycombinator.com/item?id=40951133
      - teaearlgraycold 2y ago
        Sounds reasonable. We’re in a very different situation. We bought a bunch of used hardware but our servers are for training models. All of our production stuff runs in AWS. We definitely don’t want downtime but a day where researchers have issues is very different than a day where your app is down and your reputation is tarnished.
        latchkey 2y ago
        Your customer is yourself. My customer is hopefully you. I do not want to piss you off, especially by telling you that the failure is because I bought used unsupported hardware.
zxexz 2y ago
If you’re OK with used equipment, I find Infiniband basically on par pricewise with used ethernet gear. Mellanox stuff is super easy to test and you can run everything with a mainline kernel. MQM8700/MQM8790 switches can be had really cheap now used. Cables are dumb expensive...except a dime a dozen used. Or buy new, 100% compatible from FS or the likes. ConnectX-6 cards are quite cheap used thanks to HFT firms constantly upgrading to the latest, and many can be set to Infiniband OR Ethernet (I _think_ some of the dual port cards suppirt simultaneous). I set up a cluster that right now has like 70 of these, at least half of them used. Have not had an issue with any of them yet (I botched a couple firmware updates but every time I saw able to just reset the card and push firmware again). Every machine is connected to the IB fabric AND to at least 100G ethernet. FWIW, used 100G Ethernet equipment is now cheap enough I’ve been upgrading my home network to be 100G. Cheaper than new consumer 10G equipment.
- latchkey 2y ago
  > If you’re OK with used equipment I'm building a business, not a home lab. (before you continue to downvote me, read what I wrote below)
  - adastra22 2y ago
    So? The difference in cost can be as much as 10x. If you're building a startup, that matters.
    - latchkey 2y ago
      What matters is support contracts and uptime. If you have a dozen customers on a server that cannot access things because of an issue, then as a startup, without a whole customer support department, you're literally screwed. I've been on HN long enough to have seen plenty of companies get complaints after growing too quickly and not being able to handle the issues they run into. I'm building this business in a way to de-risk things as much as possible. From getting the best equipment I can buy today, to support contracts, to the best data center to just scaling with revenue growth. This isn't a cost issue, it is a long term viability issue. Home lab... certainly cut as many corners as you want. Cloud service provider building top super computers for rent... not so much. There is a reason why not a lot of people start to do this... it is extremely capital intensive. That is a huge moat and getting the relationships and funding to do what I'm doing isn't easy and took me over 5 years to get to this point of just getting started. I'm not going to blow it all on cutting corners on some used equipment.

omneity 2y ago

Did you procure the servers directly from Dell or through a distribution partner?

latchkey 2y ago
Kind of both. We started talking to Dell first and then they introduced us to Advizex. We are now effectively partnered with both companies, which is fantastic as the Advizex team have ex-Dell people working directly with us. We are lucky to have two whole teams of super talented people helping us out on this journey.

logicchains 2y ago

Meta had success building an ethernet cluster on Arista 7800 with Wedge400 and Minipack2 OCP rack switches.: https://www.datacenterdynamics.com/en/news/meta-reveals-details-of-two-new-24k-gpu-ai-clusters/ https://www.datacenterdynamics.com/en/news/meta-reveals-deta...

latchkey 2y ago
I actually have a call with Arista next week to learn more about their solutions. Especially those 7800's. They look awesome. One "problem" we have right now is that our cluster cannot support more than 128 GPUs. If we wanted to scale with Dell, we'd have to buy 6x more Z9864F to add one more cluster, which is crazy expensive and complicated. I want to see if Arista has something that can help us. That said, I also have to find a customer that wants more than 128 MI300x and that hasn't happened... yet.

csmpltn 2y ago

> "Our datacenter is 100% green" Cool, where can I read more about this? How do you power your DC?

walrus01 2y ago
Plenty of datacenters that are somewhere generally in the pacific NW can claim to be "Green" because their power supply is entirely hydroelectric. https://www.nwd.usace.army.mil/CRWM/CR-Dams/ https://www.nwd.usace.army.mil/CRWM/CR-Dams/ Many of those areas also happen to have the lowest $ per kWh electricity in North American, the only lower rate is available near a few hydroelectric dams in Quebec.
- latchkey 2y ago
  Previously, I had multiple data centers in Quincy, WA. Those were hydro green. It is an area that hosts a whole multitude of big hyperscaler companies.
- mulmen 2y ago
  As anyone who has driven through the Columbia River Basin can tell you wind power is also abundant in Washington. The grid is very clean here but it’s certainly not purely hydro.
- dlkf 2y ago
  Why is green in scare-quotes?
  - uncertainrhymes 2y ago
    People make the argument that is a giant datacenter is consuming 50% of some local hydro installation, everyone else is town is buying something else that is less green. It opens up questions about grids and market efficiency, so your mileage may vary.
    - sangnoir 2y ago
      > People make the argument that is a giant datacenter is consuming 50% of some local hydro installation, everyone else is town is buying something else that is less green. I don't think that's a cogent argument. It's akin to saying a vegan commune in a small is buying is buying up 50% of the vegan food, "forcing" others to buy meat-products, and framing this to cast doubts on whether they are truly vegan. Consumers aren't in a position to solve supply problems.

dpflan 2y ago

What are you using these for? Providing compute for customers that want to train/infer? What is the level of interest and level of success customers are seeing using these services Hot Aisle offers?

latchkey 2y ago
We are a bare metal compute offering. They can be used for whatever people want to use them for (within legal limits, of course). Interest is much higher now that we've started to work with teams publishing benchmarks which show that H100's have a nice competitor [0]. I'll admit, it is still early days. We just finished up another free compute [1] two week stint with a benchmarking team. One thing we discovered is that saving checkpoints is slow AF. I'm guessing an issue with ROCm. Hopefully get that resolved soon. Now we are in the process of onboarding the next team. [0] https://hotaisle.xyz/benchmarks-and-analysis/ [1] https://hotaisle.xyz/free-compute-offer/
- rvnx 2y ago
  > We would love to offer hourly on-demand rates for individual GPUs, but we can't do so at this time due to a limitation in the ROCm/AMD drivers. This limitation prevents PCIe pass-through to a virtual machine, making multi-tenancy impossible. AMD is aware of this issue and has committed to resolving it. One idea to help you: Are you sure you need a virtual machine ? Couldn't you boot the machines under PXE to solve the imaging problem ? Essentially you have TFTP server that gives a Linux image and boot on it directly
  - latchkey 2y ago
    1 chassis, 8 gpus. We want to be able to break that chassis up into individual GPUs and allocate 1 GPU to 1 "machine". I previously PXE booted 20,000 individual playstation 5 diskless blades and I'm not sure how PXE would solve this. The only alternative right now is to do what runpod (and AMD's aac) are doing and do docker containers. But that has the limitation of docker in docker, so people end up having to repackage everything. You also can't easily run different ROCm versions since that comes from the host, and if you have 8 people on a single chassis... it becomes a nightmare to manage it. We're just patiently waiting for AMD to fix the problem.

deleted 2y ago

[deleted]

rlupi 2y ago

https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/network-fabrics.html https://docs.nvidia.com/dgx-superpod/reference-architecture-... NVIDIA large GPU supercomputers have separate compute-networking (between GPUs) and storage-networking (storage to GPUs, or storage to SSD, SSD to GPUs with CPU assistance). This helps avoid networking issues, even more if not using Infiniband. From what I read here and on your website, you don't go that route. I haven't found the equivalent system level reference architecture for MI300x from AMD. I wonder if you have a link to a public document where AMD provides guidance about this choice?

derefr 2y ago
> even more if not using Infiniband It's interesting that the above "HPC reference architecture" shows a GPU-to-GPU Infiniband fabric, despite Nvidia also nominally pushing NVLink Switch (https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/) for the HPC use-case.
- bee_rider 2y ago
  How does NVLink work? Because I already know MPI and I’m not going to learn anything else, lol. Edit: after googling it looks like OpenMPI has some NVLink support, so maybe it is OK.
  - zxexz 2y ago
    I use OpenMPI with no issues over multiple H100 nodes and A100 nodes, with multiple infiniband 200G and ethernet 100G/200G networks, and RDMA (though using mellanox instead of broadcom cards, but afaik broadcom supports this just the same). Side note, make sure you compile nvidia_peermem correctly if you want GDRMA to work :)
    - latchkey 2y ago
      No issues, except this minor bit of arcane knowledge that is missing from SO. :)

RobRivera 2y ago

Nice! Whats benchmark standard these days? Still Superbench or yall have something inhouse? Re: lead time quote :O I guess I got spoiled working for one of the major cloud vendors. The thought of poor b2b vendor support never entered my risk matrix. If you own your own cluster, the network bottleneck becomes less a dollar cost I suppose, since you arent being charged a premium to rent someone elses compute

latchkey 2y ago
We don't run benchmarks ourselves. We donate the expensive Ferrari worth of compute for others to do it. This is the most unbiased way I could think of getting useful real world data to share. The 3rd team just finished up and the 4th is getting started now. I've got 23 others in the wings. https://hotaisle.xyz/free-compute-offer/ https://hotaisle.xyz/free-compute-offer/
- jononor 2y ago
  This seems like an excellent strategy in terms of marketing and building mindshare for the platform (as you mention elsewhere, it is an underdog).
  - latchkey 2y ago
    Thank you for the kind words.

pheatherlite 2y ago

Being out of the loop for awhile. Has amd made anything similar to cuda? Are cots frameworks such as pytorch and tensorflow on par when running on amd hardware? What makes investing in amd cluster/chips worthwhile?

jononor 2y ago
ROCm is the framework. Both PyTorch and Tensorflow have versions that support it. Have no experience with it, so cannot say how it works in practice.

JackYoustra 2y ago

Why xeon instead of epyc?

latchkey 2y ago
Sadly, the only option Dell supports today. To get to market the fastest, they took their existing H100 chassis solution, swapped out the GPUs/baseboard and called it a day.

startupsfail 2y ago

It seems the reliability, speed and scalability drops with the Ethernet are somewhat manageable. To quote the article - From our tests, we found that Infiniband was systematically outperforming Ethernet interconnects in terms of speed. When using 16 nodes / 128 GPUs, the difference varied from 3% to 10% in terms of distributed training throughput[1]. The gap was widening as we were adding more nodes: Infiniband was scaling almost linearly, while other interconnects scaled less efficiently. And then they do mention that the research team needs to debug unexplained failures on Ethernet that they’ve not seen on Infiniband. This actually can be the expensive part. Particularly if the failures are silent and cause numerical errors only.

latchkey 2y ago
A single switch should mitigate some of the throughput issues. As for issues, this is why I have a full professional support contract with Dell and Advizex. If there are issues in the gear, they will step in to help out. Especially on the switch, since it is a spof, we went with a 4 hour window.

teaearlgraycold 2y ago

We just set up a small cluster of our own. We’re not using infiniband but it didn’t seem like it would be a 50 week lead time to setting it up. Where did you get that number?

latchkey 2y ago
I hear a couple issues with your comment... "small cluster", "we're not using infinband" I'm only saying what was quoted to me. The cards are easy to get, it is the switches that are more difficult.
- teaearlgraycold 2y ago
  I saw plenty of used switches available. Are current gen switches necessary?
  - latchkey 2y ago
    https://news.ycombinator.com/item?id=40951133 https://news.ycombinator.com/item?id=40951133
    - teaearlgraycold 2y ago
      Sounds reasonable. We’re in a very different situation. We bought a bunch of used hardware but our servers are for training models. All of our production stuff runs in AWS. We definitely don’t want downtime but a day where researchers have issues is very different than a day where your app is down and your reputation is tarnished.
      - latchkey 2y ago
        Your customer is yourself. My customer is hopefully you. I do not want to piss you off, especially by telling you that the failure is because I bought used unsupported hardware.

zxexz 2y ago

If you’re OK with used equipment, I find Infiniband basically on par pricewise with used ethernet gear. Mellanox stuff is super easy to test and you can run everything with a mainline kernel. MQM8700/MQM8790 switches can be had really cheap now used. Cables are dumb expensive...except a dime a dozen used. Or buy new, 100% compatible from FS or the likes. ConnectX-6 cards are quite cheap used thanks to HFT firms constantly upgrading to the latest, and many can be set to Infiniband OR Ethernet (I _think_ some of the dual port cards suppirt simultaneous). I set up a cluster that right now has like 70 of these, at least half of them used. Have not had an issue with any of them yet (I botched a couple firmware updates but every time I saw able to just reset the card and push firmware again). Every machine is connected to the IB fabric AND to at least 100G ethernet. FWIW, used 100G Ethernet equipment is now cheap enough I’ve been upgrading my home network to be 100G. Cheaper than new consumer 10G equipment.

latchkey 2y ago
> If you’re OK with used equipment I'm building a business, not a home lab. (before you continue to downvote me, read what I wrote below)
- adastra22 2y ago
  So? The difference in cost can be as much as 10x. If you're building a startup, that matters.
  - latchkey 2y ago
    What matters is support contracts and uptime. If you have a dozen customers on a server that cannot access things because of an issue, then as a startup, without a whole customer support department, you're literally screwed. I've been on HN long enough to have seen plenty of companies get complaints after growing too quickly and not being able to handle the issues they run into. I'm building this business in a way to de-risk things as much as possible. From getting the best equipment I can buy today, to support contracts, to the best data center to just scaling with revenue growth. This isn't a cost issue, it is a long term viability issue. Home lab... certainly cut as many corners as you want. Cloud service provider building top super computers for rent... not so much. There is a reason why not a lot of people start to do this... it is extremely capital intensive. That is a huge moat and getting the relationships and funding to do what I'm doing isn't easy and took me over 5 years to get to this point of just getting started. I'm not going to blow it all on cutting corners on some used equipment.