6 ms·
What Happens When Hyperscalers and Clouds Buy Most Servers and Storage?
- latchkey 3y agoI'm building a business that operates as the cap/opex for businesses that do not want to invest in the huge upfront costs and effort to deploy the best compute and storage that we can get today. Especially around AI/ML/CFD type of workloads. I see this as a niche for allowing you to focus on building products with features, while we handle the rest of the details. Gone are the days of just racking some servers, as the deployment complexity has only increased exponentially over time. Especially with this newer hardware, the infrastructure needs and failure rate, debugging issues are all through the roof. I see us as kind of a middle ground between hyperscalers and building it yourself. We give full bare metal access to the hardware, as if you own it yourself and are sitting at it. This should help protect at least a small amount against hyperscalers and HPC completely dominating the market. My competitors are all focused on one company beating another in this AI compute race, as if it is a sporting competition. My focus is on running only the best of the best, I don't care who's equipment it is. We have been doing large scale deployments for a while now, but just getting started in this new business. Looking forward to seeing how things develop over time.
- kaliszad 3y agoThe deployment and operations of modern hardware certainly is a field in itself and always was. What experience do you have with that? It doesn't really seem exponentially more complex to me. Yes, accelerators are more common now and the heat production is similar to that of old rack-mountable mainframes but that in itself is nothing too new. Also, we have had low-latency interconnects in the HPC space for ages - that hasn't changed that much neither it is just more widely deployed now.
- latchkey 3y agoHi, thanks for the response. 30 years of being in tech and startups. Previously deployed, managed and optimized 150,000 GPUs across 7 data centers. Power usage has changed quite a bit, such that we see articles now about how older datacenters power rack density doesn't work any more [0]. Each of our MI300x boxes are 7KW. That doesn't work with only a 20amp circuit to your rack. Liquid cooling has been in development for ages, but is now being pushed to the front due to the higher power requirements of the servers. We have to lower the PUE, in order to increase rack density. This increases complexity through the entire supply chain. Networking is more complicated. For example, we are running into sonic software issues with the need to deploy VRF to support multiple customers, and it is buggy AF in some implementations from some vendors. NICs no longer talk to CPUs, they talk directly to GPUs (RoCE), and that has more interconnects (8x), along with the way they are connected and managed at the switches. One design even puts each GPU on a separate switch, requiring even more cables. When you are at 400G/800G, which chip you choose in your switch matters. Relationships. Dealing with multiple vendors, supply chain issues, finding power/space, all these are more complicated now that demand is far far higher. 6 years ago, Cenly Chen from Supermicro said AI spend in 2025 would be $36B [1]. Current estimates now are $750B+. You can't just order a box of MI300x from your local BestBuy. These things are export controlled by the US govt and effectively classified as weapons with restricted usage. We have to KYC our customers. This is just the tip of the iceberg, I could go on and on... now that I'm in the thick of things, I feel pretty confident that we are providing a pretty necessary service. Especially at the scale we are building for. [0] https://www.datacenterknowledge.com/hardware/data-center-rack-density-has-doubled-and-its-still-not-enough https://www.datacenterknowledge.com/hardware/data-center-rac... [1] https://youtu.be/WzqBuiwkv5I?feature=shared&t=58 https://youtu.be/WzqBuiwkv5I?feature=shared&t=58
- kaliszad 3y agoThanks for the details. What a career!
- jedberg 3y agoI work for a company (Lambda Labs) that would span all these categories. We build GPUs and sell them, sending them off to the customer. We also build them and sell them, but then host it for the customer. We also buy them for ourselves and rent them either bare metal or virtualized. The "build it and ship it to the customer" business is waning. The reason is because even the most deep pocketed customers no longer have data centers that can support the power and cooling needs of current GPUs. The latest from NVIDIA requires liquid cooling, tons of power, and solid floors (the racks are too heavy for raised floors). Being able to run a modern server is becoming a very niche skill, one that most companies don't want or need to invest in.
- echelon 3y ago> The latest from NVIDIA requires liquid cooling, tons of power, and solid floors We won't be running on GPUs forever. The innovation in the chip space will continue. With as much margin as cloud takes, there will be increased interest in on-prem and mixed environments. We run our own GPUs in production as well as having a cloud footprint.
- jedberg 3y ago> We won't be running on GPUs forever. The innovation in the chip space will continue. Sure, but I'm seeing a divergence. There are a group of chips that are getting smaller and require less power and cooling that can do a lot of workloads that you used to need huge chips for, but there are the most advanced chips for the most advanced workloads getting bigger and more power hungry. There will always be the big customers who need the big chips, and that is what will be nearly impossible for them to self host.
- PLenz 3y agoThis is the same argument that mainframe companies made in the 80s. Mass commodity systems will eventually kill the big gpu except for legacy and niche workloads. It's the tech infra circle of life.
- blackeyeblitzar 3y agoProbably what’s needed is regulation, to control their pricing and ability to accept or deny customers. They’re basically a utility.
- dilyevsky 3y agoNothing will ensure incumbency better than slapping more regs on it
- wmf 3y agoYes, you only regulate a monopoly after you've decided that competition is impossible.
- blackeyeblitzar 3y agoI think we need to be more proactive than that personally. Lots of big companies do anti competitive things that we don’t want even if they aren’t exactly a monopoly. But no one’s doing anything about that.
- blackeyeblitzar 3y agoIs incumbency a bad thing if they’ve to serve everyone equally and at prices that are fair? It’s not like they operate without any regs at all today, right?
- dilyevsky 3y agoIt is a bad thing bc it stifles innovation and ultimately hurts everyone but incumbent shareholders and their pocket legislators. And on the latter point it is also unlikely to be “fair”
- zer00eyz 3y agoWhat I find odd is that I know of a lot of companies looking a moving back to their own hardware. They are going to need a hell of a lot fewer boxes today than in 2009 (15 years ago) This is 2009: https://www.intc.com/news-events/press-releases/detail/1307/intel-introduces-core-i7-xeon-3400-and-first-core https://www.intc.com/news-events/press-releases/detail/1307/... I can get a 128 core system today, if I need the density. And that is the thing, density is through the roof, and pricing is roughly the same as 2009. The same thing has happened to storage performance and (to a degree) scale. What would have taken a few racks 15 years ago barely fills one today. 30% percent of AWS's revenue is profit. Thats what you're leaving on the table. For a lot of companies (at least the ones that can plan ahead and do basic accounting) that does not make sense. Even more so when lending is tight and every dollar counts.
- jedberg 3y ago> 30% percent of AWS's revenue is profit. If you built exactly what AWS has you wouldn't recapture that because most of that comes from operating efficiencies at scale. Yes, you can get some of that back, but it won't be 30%.
- chasd00 3y agoI think the idea is a company would only build what they use instead of replicating AWS feature for feature. If they just build what the need then I think 30% is on the low side.
- jedberg 3y agoPower and cooling are the two main things that they amortize over many customers, and everyone needs those.
- vidarh 3y agoThey're however low margin commodity services at a colo provider. Building your own data centers is the last of a long list of steps up in difficulty and one most people wont need. Renting by the rack in a colo facility with a support contract is however approachable from the point your cost is above ~$10k / months. Depending on location, once you're above ~10 racks or so per physical location, a dedicated cage can start to make sense, and a bit above that a data centre where you can rent office space for a couple of your own dedicated staff. Even many colo providers haven't owned all of their own setups for years. I think 2001 or so was the first time we had facilities in a colo building where half the building has been leased, ready made w/raised floors, HVAC, generators etc. to another colo provider which "just" had to staff it and handle sales and accounts. You can go to a commercial real estate broker and get a fractional data centre building, or you can go to a data centre operators and get a fractional functional, staffed data centre all the way from large percentages of the space with locked steel cages or walls separatibg your part from the rest down to individual 1U server slots.
- sixdimensional 3y agoThis is a good reminder that what we call the public cloud today is actually a misnomer - it’s not actually a public cloud, it’s someone else’s cloud. Just like we used to say the cloud is really just someone else’s computer. I think people started calling it the public cloud simply because of the scale and seemingly easy access to everyone, where the current clouds are seemingly in open competition with each other.
- throwaway11460 3y agoThe public waterpool is also someone else's pool but it's still public.
- jedberg 3y agoKind of different. The public pool is paid for by taxes and is publicly owned. The public cloud is still privately owned.
- throwaway11460 3y agoMy point is that the pool very often is just a private company. All three public pools around me here in central Europe are privately held companies.
- deleted 3y ago[deleted]
- toast0 3y agoPublic has many meanings. Everyone can buy or access it is a valid meaning and doesn't say anything about ownership.
- curt15 3y ago>Just like we used to say the cloud is really just someone else’s computer. I still describe cloud computing as renting someone else's computers.
- eddythompson80 3y ago
- kaliszad 3y agoThe idea is that servers should become a lot more manageable and debuggable to the very bottom. At least that's what Oxide Computer offer. (I am not affiliated with them in any way, just find the idea great.) There is a company in Germany - Cloud&Heat that offer whole rack solutions that are specially designed for efficiency/ water cooling and security. Both companies do offer interesting stuff for largeish customers.
- hintymad 3y agoI wonder why the cost of running on the cloud, engineering cost included, is not a concern to companies. Per DHH, even a modest app like 37Sigal's would cost millions more than they liked (https://world.hey.com/dhh/we-have-left-the-cloud-251760fb https://world.hey.com/dhh/we-have-left-the-cloud-251760fb) On the other hand, as increasingly only a few cloud companies know how to build out datacenters, maybe the talent pool for building datacenters will keep shrinking - a vicious cycle in which fewer companies will choose to go with their own data centers.
- lowbloodsugar 3y agoSure. Bring home legacy services that you aren’t innovating on any more. Running it on a single server. Obviously that’s a great option for a small tech company.
- acd 3y agoBuy local also data center space.
- jiayq84 3y agoI do a startup called Lepton AI. We provide AI PaaS and fast AI runtimes as a service, so we keep a close eye on the IaaS supply chain. For the last few months we see supply chain getting better and better, so the business model that worked 6 months ago - "we have gpus, come buy barebone servers" no longer work. However, a bigger problem emerges. Probably a problem that could shake the industry: people don't know how to efficiently use these machines. There are clusters of GPUs sitting idle because companies don't know how to use them. It's embarrassing to resell them too because that makes the images look bad to VCs, but secondary market is slowly happening. Essentially, people want a PaaS or SaaS on top of the barebone machines. For example, for the last couple months we were helping a customer to fully utilize their hundreds-of-card cluster. Their IaaS provider was new to the field. So we literally helped both sides to (1) understand infiniband and nccl and training code and stuff; (2) figure out control plane traffic; (3) built accelerated storage layer for training; (4) all kinds of subtle signals that needs attention. Do you know that a GPU can appear OK in nvidia-smi, but still encounter issues when you actually run a cuda or nccl kernel? That needs care. (5) fast software runtimes, like LLM runtime, finetuning script, and many others. So I think AI PaaS and SaaS is going to be a very valuable (and big) market, after people come out of the frenzy of "grabbing gpus" - and now we need to use them efficiently.
- FpUser 3y agoThis is really scary. Everything we do gravitates to a practically single point of failure in way too many areas of life. At some point single small scale unfortunate event, evil entity / person and what not might bring countries to a point of collapse. And our "servants" do not seem to give a flying fuck about it. Well they might have secured place in hideouts of the world's Zuckerbergs.
- nightshift1 3y agoI am looking at the graphs, and all I see is a substantial influx of new funds into cloud services. Non-cloud investments remain steady but are not experiencing significant growth. What does the volume indicate? Is the cost of on-premises infrastructure rising, or are people simply facing increased computing demands? I wish. I sincerely hope that owning the entire stack in a private data center won't become exclusive to a select few hyperscalers in the coming years. I am somewhat biased, because I am a Linux admin forna company that has its own datacenters that is migrating to the cloud, but I find it distressing that everyone appears to be migrating to the cloud for reasons that seem dubious, such as "Gartner told me to." Like mentioned at the end of the article, I predict a great loss of knowledge for everything that is not abput the usage of a proprietary api.
- karma_pharmer 3y agoThe upside is that these folks are extremely, extremely space-and-power constrained. Cloud data centers cycle through hardware much faster than any other industry (enterprise users, engineering workstations, render farms, etc) and when they're done with it they dump it on ebay for pennies. If you look at the large-volume sellers of rackmount servers on ebay, you'll notice that there are one or two near each major data center cluster. The best part is that they really do not care at all about any kind of cost recovery, so they dump truckloads of this stuff all at once and tank the price. If you can afford to wait you can get some utterly amazing deals on five-year-old top-of-the-line gear. If you buy enough to justify freight the (amortized) shipping is really cheap too.