4 ms·

The deployment and operations of modern hardware certainly is a field in itself and always was. What experience do you have with that? It doesn't really seem ex

The deployment and operations of modern hardware certainly is a field in itself and always was. What experience do you have with that? It doesn't really seem exponentially more complex to me. Yes, accelerators are more common now and the heat production is similar to that of old rack-mountable mainframes but that in itself is nothing too new. Also, we have had low-latency interconnects in the HPC space for ages - that hasn't changed that much neither it is just more widely deployed now.

latchkey 2y ago
Hi, thanks for the response. 30 years of being in tech and startups. Previously deployed, managed and optimized 150,000 GPUs across 7 data centers. Power usage has changed quite a bit, such that we see articles now about how older datacenters power rack density doesn't work any more [0]. Each of our MI300x boxes are 7KW. That doesn't work with only a 20amp circuit to your rack. Liquid cooling has been in development for ages, but is now being pushed to the front due to the higher power requirements of the servers. We have to lower the PUE, in order to increase rack density. This increases complexity through the entire supply chain. Networking is more complicated. For example, we are running into sonic software issues with the need to deploy VRF to support multiple customers, and it is buggy AF in some implementations from some vendors. NICs no longer talk to CPUs, they talk directly to GPUs (RoCE), and that has more interconnects (8x), along with the way they are connected and managed at the switches. One design even puts each GPU on a separate switch, requiring even more cables. When you are at 400G/800G, which chip you choose in your switch matters. Relationships. Dealing with multiple vendors, supply chain issues, finding power/space, all these are more complicated now that demand is far far higher. 6 years ago, Cenly Chen from Supermicro said AI spend in 2025 would be $36B [1]. Current estimates now are $750B+. You can't just order a box of MI300x from your local BestBuy. These things are export controlled by the US govt and effectively classified as weapons with restricted usage. We have to KYC our customers. This is just the tip of the iceberg, I could go on and on... now that I'm in the thick of things, I feel pretty confident that we are providing a pretty necessary service. Especially at the scale we are building for. [0] https://www.datacenterknowledge.com/hardware/data-center-rack-density-has-doubled-and-its-still-not-enough https://www.datacenterknowledge.com/hardware/data-center-rac... [1] https://youtu.be/WzqBuiwkv5I?feature=shared&t=58 https://youtu.be/WzqBuiwkv5I?feature=shared&t=58
- kaliszad 2y ago
  Thanks for the details. What a career!

latchkey 2y ago

Hi, thanks for the response. 30 years of being in tech and startups. Previously deployed, managed and optimized 150,000 GPUs across 7 data centers. Power usage has changed quite a bit, such that we see articles now about how older datacenters power rack density doesn't work any more [0]. Each of our MI300x boxes are 7KW. That doesn't work with only a 20amp circuit to your rack. Liquid cooling has been in development for ages, but is now being pushed to the front due to the higher power requirements of the servers. We have to lower the PUE, in order to increase rack density. This increases complexity through the entire supply chain. Networking is more complicated. For example, we are running into sonic software issues with the need to deploy VRF to support multiple customers, and it is buggy AF in some implementations from some vendors. NICs no longer talk to CPUs, they talk directly to GPUs (RoCE), and that has more interconnects (8x), along with the way they are connected and managed at the switches. One design even puts each GPU on a separate switch, requiring even more cables. When you are at 400G/800G, which chip you choose in your switch matters. Relationships. Dealing with multiple vendors, supply chain issues, finding power/space, all these are more complicated now that demand is far far higher. 6 years ago, Cenly Chen from Supermicro said AI spend in 2025 would be $36B [1]. Current estimates now are $750B+. You can't just order a box of MI300x from your local BestBuy. These things are export controlled by the US govt and effectively classified as weapons with restricted usage. We have to KYC our customers. This is just the tip of the iceberg, I could go on and on... now that I'm in the thick of things, I feel pretty confident that we are providing a pretty necessary service. Especially at the scale we are building for. [0] https://www.datacenterknowledge.com/hardware/data-center-rack-density-has-doubled-and-its-still-not-enough https://www.datacenterknowledge.com/hardware/data-center-rac... [1] https://youtu.be/WzqBuiwkv5I?feature=shared&t=58 https://youtu.be/WzqBuiwkv5I?feature=shared&t=58

kaliszad 2y ago
Thanks for the details. What a career!