3 ms·

I'm building a business that operates as the cap/opex for businesses that do not want to invest in the huge upfront costs and effort to deploy the best compute

I'm building a business that operates as the cap/opex for businesses that do not want to invest in the huge upfront costs and effort to deploy the best compute and storage that we can get today. Especially around AI/ML/CFD type of workloads. I see this as a niche for allowing you to focus on building products with features, while we handle the rest of the details. Gone are the days of just racking some servers, as the deployment complexity has only increased exponentially over time. Especially with this newer hardware, the infrastructure needs and failure rate, debugging issues are all through the roof. I see us as kind of a middle ground between hyperscalers and building it yourself. We give full bare metal access to the hardware, as if you own it yourself and are sitting at it. This should help protect at least a small amount against hyperscalers and HPC completely dominating the market. My competitors are all focused on one company beating another in this AI compute race, as if it is a sporting competition. My focus is on running only the best of the best, I don't care who's equipment it is. We have been doing large scale deployments for a while now, but just getting started in this new business. Looking forward to seeing how things develop over time.

kaliszad 3y ago
The deployment and operations of modern hardware certainly is a field in itself and always was. What experience do you have with that? It doesn't really seem exponentially more complex to me. Yes, accelerators are more common now and the heat production is similar to that of old rack-mountable mainframes but that in itself is nothing too new. Also, we have had low-latency interconnects in the HPC space for ages - that hasn't changed that much neither it is just more widely deployed now.
- latchkey 3y ago
  Hi, thanks for the response. 30 years of being in tech and startups. Previously deployed, managed and optimized 150,000 GPUs across 7 data centers. Power usage has changed quite a bit, such that we see articles now about how older datacenters power rack density doesn't work any more [0]. Each of our MI300x boxes are 7KW. That doesn't work with only a 20amp circuit to your rack. Liquid cooling has been in development for ages, but is now being pushed to the front due to the higher power requirements of the servers. We have to lower the PUE, in order to increase rack density. This increases complexity through the entire supply chain. Networking is more complicated. For example, we are running into sonic software issues with the need to deploy VRF to support multiple customers, and it is buggy AF in some implementations from some vendors. NICs no longer talk to CPUs, they talk directly to GPUs (RoCE), and that has more interconnects (8x), along with the way they are connected and managed at the switches. One design even puts each GPU on a separate switch, requiring even more cables. When you are at 400G/800G, which chip you choose in your switch matters. Relationships. Dealing with multiple vendors, supply chain issues, finding power/space, all these are more complicated now that demand is far far higher. 6 years ago, Cenly Chen from Supermicro said AI spend in 2025 would be $36B [1]. Current estimates now are $750B+. You can't just order a box of MI300x from your local BestBuy. These things are export controlled by the US govt and effectively classified as weapons with restricted usage. We have to KYC our customers. This is just the tip of the iceberg, I could go on and on... now that I'm in the thick of things, I feel pretty confident that we are providing a pretty necessary service. Especially at the scale we are building for. [0] https://www.datacenterknowledge.com/hardware/data-center-rack-density-has-doubled-and-its-still-not-enough https://www.datacenterknowledge.com/hardware/data-center-rac... [1] https://youtu.be/WzqBuiwkv5I?feature=shared&t=58 https://youtu.be/WzqBuiwkv5I?feature=shared&t=58
  - kaliszad 3y ago
    Thanks for the details. What a career!

kaliszad 3y ago

The deployment and operations of modern hardware certainly is a field in itself and always was. What experience do you have with that? It doesn't really seem exponentially more complex to me. Yes, accelerators are more common now and the heat production is similar to that of old rack-mountable mainframes but that in itself is nothing too new. Also, we have had low-latency interconnects in the HPC space for ages - that hasn't changed that much neither it is just more widely deployed now.

latchkey 3y ago
Hi, thanks for the response. 30 years of being in tech and startups. Previously deployed, managed and optimized 150,000 GPUs across 7 data centers. Power usage has changed quite a bit, such that we see articles now about how older datacenters power rack density doesn't work any more [0]. Each of our MI300x boxes are 7KW. That doesn't work with only a 20amp circuit to your rack. Liquid cooling has been in development for ages, but is now being pushed to the front due to the higher power requirements of the servers. We have to lower the PUE, in order to increase rack density. This increases complexity through the entire supply chain. Networking is more complicated. For example, we are running into sonic software issues with the need to deploy VRF to support multiple customers, and it is buggy AF in some implementations from some vendors. NICs no longer talk to CPUs, they talk directly to GPUs (RoCE), and that has more interconnects (8x), along with the way they are connected and managed at the switches. One design even puts each GPU on a separate switch, requiring even more cables. When you are at 400G/800G, which chip you choose in your switch matters. Relationships. Dealing with multiple vendors, supply chain issues, finding power/space, all these are more complicated now that demand is far far higher. 6 years ago, Cenly Chen from Supermicro said AI spend in 2025 would be $36B [1]. Current estimates now are $750B+. You can't just order a box of MI300x from your local BestBuy. These things are export controlled by the US govt and effectively classified as weapons with restricted usage. We have to KYC our customers. This is just the tip of the iceberg, I could go on and on... now that I'm in the thick of things, I feel pretty confident that we are providing a pretty necessary service. Especially at the scale we are building for. [0] https://www.datacenterknowledge.com/hardware/data-center-rack-density-has-doubled-and-its-still-not-enough https://www.datacenterknowledge.com/hardware/data-center-rac... [1] https://youtu.be/WzqBuiwkv5I?feature=shared&t=58 https://youtu.be/WzqBuiwkv5I?feature=shared&t=58
- kaliszad 3y ago
  Thanks for the details. What a career!