6 ms·

I am building a bare metal mi300x service provider business. Anyone offering $2 GPUs is either losing money on DC space/power, or their service is so sketchy u

I am building a bare metal mi300x service provider business. Anyone offering $2 GPUs is either losing money on DC space/power, or their service is so sketchy under the covers, which they do their best to hide. It is one thing to play around with $2 gpus and another to run a business. If you're trying to do the latter, you're not considering how you are risking your business on unreliable compute. AWS really twerked people's perception of what it takes to run high end enterprise GPU infrastructure like this. People got used to the reliability hyperscaler offers. They don't consider what 999999% uptime + 45kW+ rack infrastructure truly costs. There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power...

dijit 2y ago
> 999999% uptime Assuming you mean 99.9999%; your hyperscaler isn't giving you that. MTBF is comparable. It's hardware at the end of the day, the VM hypervisor isn't giving you anything on GPU instances because those GPU instances aren't possible to live-migrate. (even normal VMs are really tricky). In a country with a decent power grid and a UPS (or if you use a colo-provider) you're going to get the same availability guarantee of a machine, maybe even slightly higher because less moving parts. I think this "cloud is god" mentality betrays the fact that server hardware is actually hugely reliable once it's working; and the cloud model literally depends on this fact. The reliability of cloud is simply the reliability of hardware; they only provided an abstraction on management not on reliability.
- llm_trw 2y ago
  I think people just don't realize how big computers have gotten since 2006. A t2.micro was an ok desktop computer back then. Today you can have something 1000 times as big for a few tens of thousands. You can easily run a company that serves the whole of the US out of a closet.
  - dartos 2y ago
    Well you’d probably also at least want a cdn in each region, so like 3 closets.
    - jgalt212 2y ago
      Cloudflare caching of static resources is cheap, so back to one closet. But three if you want to be pure and totally cloudless.
      - felixgallo 2y ago
        With one closet you can also lose the entire business if one water pipe breaks or one wire goes bad in drywall. Back to three closets.
        llm_trw 2y ago
        If only it were possible to make backups. Alas no such technology exists.
        LoganDark 2y ago
        Yes. That's what the other closets are for. Redundancy.
        dartos 2y ago
        backups do not prevent downtime.
        krab 2y ago
        Yes. That's a risk assessment every company must make. What's the probability of downtime vs the development slowdown and the operating costs of a fully redundant infrastructure? I worked for a payments company (think credit cards). We designed the system to maintain very high availability in the payment flow. Multi-region, multi-AZ in AWS. But all other flows such as user registration, customer care or even bill settlement had to stop during that one incident when our main datacenter lost power after a testing switch. The outage lasted for three hours and it happened exactly once in five years. In that specific case, investing into higher availability by architecting in more redundancy would not be worth it. We had more downtime caused by bad code and not well thought out deployments. But that risk equation will be different for everyone.
  - geodel 2y ago
    Well the problem nowadays is what can be done has become what must be done. totally bypassing on question of what should be done. So now instead of single service serving 5 million requests in a business is replaced by 20 micro services generating traffic of 150 million requests with distributed transactions, logging (MBs of log per request), monitoring, metrics and so on. All leading to massive infrastructure bloat. Do it for dozen more applications and future is cloudy now. Once management is convinced by sales people or consultants any technical argument can be brushed away as not seeing the strategic big picture of managing enterprise infrastructure.
  - JohnBooty 2y ago
    It's just wild to me how seemingly nobody is exploiting this. Our industry has really lost sight of reality and the goals we're trying to achieve. Sufficient scalability, sufficient performance, and as much developer productivity as we can manage given the other two constraints. That is the goal, not a bunch of cargo-culty complex infra. If you can achieve it with a single machine, fucking do it. A monolith-ish app, running on e.g. an Epyc with 192 cores and a couple TB of RAM???? Are you kidding me? That is so much computing power, to the point where for a lot of scenarios it can replace giant chunks of complex cloud infrastructure. And for something approaching a majority of businesses it can probably replace all of it. (Yes, I know you need at least one other "big honkin server", located elsewhere, for failover. And yes, this doesn't work for all sets of requirements, etc)
    - _3u10 2y ago
      Servers that big rarely fail anyway, everything is hotswap and redundant.
      - latchkey 2y ago
        The GPUs/OAM,UBB are not redundant and they do fail. From what I hear, Nvidia has an exceptionally high failure rate.
    - uid65534 2y ago
      I feel this every day I talk with cloud-brained coworkers. I manage an infrastructure with tens of thousands of VMs and everyone is obsessed with auto scaling and clustering and every other thing the vendor sales dept shoved down their throats while simultaneously failing to realize that they could spend <5% of what we currently do and just use the datacenter cages we _already have_ and a big fat rack of 2S 9754 1U servers. The kicker? These VMs are never more than 8 cores a piece, and applications never scale to more than 3 or 4 in a set. With sub 40% CPU utilization each. Most arguments against cloud abuse like this get ignored because VPs see Microsoft (Azure in this case) as some holy grail for everything and I frankly don't have it in me to keep fighting application dev teams that don't know anything about server admin. And that's without getting into absolutely asinine price/perf SaaS offerings like Cosmos DB.
- zaptrem 2y ago
  As someone who has done a bunch of large scale ML on hyperscaler hardware I will say the uptime is nowhere near 99.9999%. Given a cluster of only a few hundred GPUs one or multiple failures is a near certainty to the point where we spend a bunch of time on recovery time optimization.
- yencabulator 2y ago
  > (even normal VMs are really tricky) For what it's worth, GCP routinely live-migrates customer VMs to schedule hardware for maintenance/decommissioning when hardware sensors start indicating trouble. It's standard everyday basic functionality by now, but only for the vendors who built the feature in from the beginning.
  - dijit 2y ago
    I’m aware, but it won’t work for gpu accelerated workloads.
- traceroute66 2y ago
  > instances aren't possible to live-migrate Some of the cloud providers don't even do live-migration. They adhere to the cloud mantra of "oh well, its up to the customer to spin up and carry on elsewhere". I have it on good authority that some of them don't even take A+B feeds to their DC suites - and then have the chutzpah to shout at the DC provider when their only feed goes down, but that's another story... :)
- dijit 2y ago
  meta: I'm always interested how the votes go on comments like this. I've been watching periodically and it seems like I get "-2" at random intervals. This is not the first time that "low yield" karma comments have sporadic changes to their votes. It seems unlikely at the rate of change (roughly 3-5 point changes per hour) that two people would simultaneously (within a minute) have the same desire to flag a comment, so I can only speculate that: A) Some people's flag is worth -2 B) Some people, passionate about this topic, have multiple accounts C) There's bots that try to remain undetected by making only small adjustments to the conversation periodically. I'm aware that some peoples job very strongly depends on the cloud, but nothing I said could be considered off topic or controversial: Cloud for GPU compute relies on hardware reliability just like everything else does. This is fact. Regardless of this, the voting behaviour on my comments such as this are extremely suspicious.
- everforward 2y ago
  > The reliability of cloud is simply the reliability of hardware; they only provided an abstraction on management not on reliability. This isn't really true. I mean it's true in the sense that you could get the same reliability on-premise given a couple decades of engineer hours, but the vast majority of on-premise deployments I have seen have significantly lower reliability than clouds and have few plans to build out those capabilities. E.g. if I exclude public cloud operator employers, I've never worked for a company that could mimick an AZ failover on-prem and I've worked for a couple F500s. As far as I can recall, none of them have even segmented their network beyond the management plane having its own hardware. The rest of the DC network was centralized; I recall one of them in specific because an STP loop screwed up half of it at one point. Part of paying for the cloud is centralizing the costs of thinking up and implementing platform-level reliability features. Some of those things are enormously expensive and not really practical for smaller economies of scale. Just one random example is tracking hardware-level points of failure and exposing that to the scheduler. E.g. if a particular datacenter has 4 supplies from mains and each rack is only connected to a single one of those supplies, when I schedule 4 jobs to run there it will try to put each job in a rack with a separate power supply to minimize the impact of losing a mains. Ditto with network, storage, fire suppression, generators, etc, etc, etc. That kind of thing makes 0 economic sense for an individual company to implement, but it starts to make a lot of sense for a company who does basically nothing other than manage hardware failures.
- wkat4242 2y ago
  > Assuming you mean 99.9999%; your hyperscaler isn't giving you that. MTBF is comparable. Yeah we've already had about a day's worth of downtime this year on office 365 and Microsoft is definitely a hyperscaler. So that's 99.3% at best.
pico_creator 2y ago
Someone is losing the money. It’s elaborated in the article how and why this happens TLDR, VC money, is being burnt/lost
- shermantanktop 2y ago
  Tons of VC money burned in pursuit of low-probability success. It’s no wonder that some people find it easier to scam VCs than it is to build a real business.
hnaccount_rng 2y ago
Can you elaborate on the cost basis? With how little could a very lean operation still make money? I know that's basically impossible to answer generically, especially given that the recurring cost is likely already zero, given that the GPUs are already paid...
dx034 2y ago
Since most applications aren't latency sensitive, space and power can be nearly free by setting up the data center in a place where it's cold, there's nearly free electricity and few people live. Leaves you with cost for infrastructure and connectivity, but I guess electricity prices shouldn't be the issue?
- tonetegeatinst 2y ago
  I'd think cost of internet would be the big issue even if can afford the AI hardware. In rural areas or even with low population it takes forever to get fiber to roll out and if your selling access to your hardware infrastructure then you really want to get a direct connection to the nearest IX so you can offer customers the best speed for accessing data and the IX would probably be one of the few places you might be able to get 400G or higher direct fiber. But if your hooking up to a IX chances are your not an end user but a autonomous system and already are shoving moving and signing NDA's to be a peer with other Autonomous Systems in the exchange and be able to bgp announce. (Source - my old highschool networking class where I got sick of my shitty internet and looked into how I could get fiber from an exchange. I'm probably mistaken on stuff here as it was years ago and its either wrong or outdated from all those years ago.)
  - oasisbob 2y ago
    Assuming rural areas have less fiber availability isn't always a good assumption. In NW Washington state at least, the rural counties (Whatcom, Island, Skagit, etc) have had a robust market in dark fiber for over two decades. The normal telcos weren't responsive to need, so private carriers picked up the slack. When I was last involved in this market, you could get a P2P strand, including reasonable buildout, for less than a cost of a T1 line with a two-year commit. The tiny four-branch credit union I worked for had dedicated fiber loops between all our locations, no big deal. It was great.
- serjester 2y ago
  Ambient cooling can only go so far. At the end of the day if you have a rack of GPU’s using 6000 watts per node, you’re going to need some very serious active cooling regardless of your location. You’ll save a little but it’s a small percentage of your overall costs.
acd10j 2y ago
May be their business model is running compute at loss and stealing ip/code from people using platform?
michaelt 2y ago
> There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power... At the highest power settings, H100s consume 400 W. Add another 200 W for CPU/RAM. Assume you have an incredibly inefficient cooling system, so you also need 600 W of cooling. Google tells me US energy prices average around 17 cents/kWh - even if you don't locate your data centre somewhere with cheap electricity. 17 cents/kWh * 1200 watts * 1 hour is only 20.4 cents/hour.
- sandworm101 2y ago
  For the fuller math one has to include the cost of infrastructure financing, which is tied to interest rates. Given how young most of these H100 shops are, I assume that they pay more to service their debts than for power.
  - Wytwwww 2y ago
    > I assume that they pay more to service their debts than for power. Well yes, because for GPU datacentres fixed/capital costs make up a much higher fraction than power and other expenses than for CPUs. To such an extent that power usage barely even matters. A $20k that uses 1 kW ( which is way more than it would in reality ) 24x7 would cost $1.3k to run per year at 0.15$ per kWh, that's almost insignificant compared to depreciation. The premise is that nobody could make any money by renting H100s for 2$ even if they got them for free unless they only had free power. That makes no sense whatsoever when you can get 2x AMD EPYC™ 9454P servers at 2x408 W (for full system) for around $0.70 in a German data center.
- ckastner 2y ago
  That's just the power. If one expects a H100 to run for three years at full load, 24 x 365 x 3 = 26280. Assuming a price of $25K per H100, that means about $1/h to amortize costs. Hence the unless they stole them, I guess. Factor in space, networking, cooling, security, etc., and $2 really do seem undoable.
  - Negitivefrags 2y ago
bjornsing 2y ago
> There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power... That’s essentially what the OP says. But once you’ve already invested in the H100s you’re still better off renting them out for $2 per hour rather than having them idle at $0 per hour.
- Wytwwww 2y ago
  Then how come you can still get several last gen EPYC or Xeon systems that would use the same amount of power for under $1 per hour? For datacentre GPUs the energy, infrastructure and other variable costs seem to be relatively insignificant to fixed capital costs. Nvidia's GPUs are just extremely expensive relative to how much power they use (compared to CPUs). > H100s you’re still better off renting them out for $2 per hour rather than having them idle at $0 per hour. If you're barely breaking even at $2 then immediately selling them would seem like the only sensible option (depreciation alone is significantly higher than the cost power of running a H100 24x365 at 100% utilization).
  - bjornsing 2y ago
    > If you're barely breaking even at $2 then immediately selling them would seem like the only sensible option (depreciation alone is significantly higher than the cost power of running a H100 24x365 at 100% utilization). If you can then probably yes. But why would someone else buy them (at the price you want), when they can rent at $2 per hour instead?
    - Wytwwww 2y ago
      I don't think the why matters as long as people are buying them at very high prices, which they seemingly still are.
      - bjornsing 2y ago
        What makes you think they are?
scotty79 2y ago
> There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power... I think that's the point. Trying to buy and run H100s now either for yourself or for someone else to rent it is a terrible investment because of oversupply. And prices you can get for compute are not enough to cover the costs.
marcyb5st 2y ago
But it is about minimizing losses, not making profits. If you read the article, such prices happen because a lot of companies bought hardware reservations for the next few years. Instead of keeping the hardware idle (since they pay for it anyway), they rent it out on the cheap to recoup something.
rajnathani 2y ago
From your bio, your company is Hot Aisle. This company TensorWave covered by TechCrunch [0] this week sounds very similar, I almost thought it was the same! Anyway, best of luck, we need more AMD GPU compute. [0] https://techcrunch.com/2024/10/08/tensorwave-claims-its-amd-powered-cloud-for-ai-will-give-nvidia-a-run-for-its-money/ https://techcrunch.com/2024/10/08/tensorwave-claims-its-amd-...
- latchkey 2y ago
  Thanks! Definitely not the same at all.
tasuki 2y ago
> If you're trying to do the latter, you're not considering how you are risking your business on unreliable compute. What do you mean by "risking your business on unreliable compute"? Is there a reason not to use one of these to train whatever neural nets one's business needs?
- oefrha 2y ago
  Well, someone who’s building a GPU renting service right now obviously wants to scare you into using expensive and “reliable” services; the market crashing is disastrous for them. The reality is high price is hardly an indicator of reliability, and the article very clearly explains why H100 hours are being sold at $2 or less, and it’s not because of certain providers lacking reliability.
  - latchkey 2y ago
    Nah, don't be silly. No need to scare anyone into anything. Use whatever you want to use. My point in saying and of this is to simply point out that we offer this service to people who value these things.
- lazide 2y ago
  If it crashes half way through, you don’t get a useful model, and you’re still on the hook for the rental costs to get there maybe?
  - latchkey 2y ago
    Depends on the SLA.
  - tasuki 2y ago
    That's... possible? But a little unlikely. I think I'll take that risk over paying more for your allegedly more reliable GPUs anytime.
fhars 2y ago
I think this is what they are insinuating with the "Hot the Bubble Burst" in the headline. You are not expected to make money if you have invested in a bursting bubble.
wolfgangK 2y ago
For training, doesn't checkpoint saving make high reliability a moot point ? Why pay for 99.99999? uptime when you can restart your training from last/best model ?
traceroute66 2y ago
> 999999% uptime I've said it before and I've said it again.... Read the cloud provider small-print before you go around boasting about how great their SLAs are. Most of the time they are not worth the paper they are written on.
- kjs3 2y ago
  This is beyond true. Read and understand what your cloud SLAs are, not what you think they are or what you think they should be. There was significant consternation generated when I pointed out that the SLA for availability for an Azure storage blob was only 4 nines with zone redundancy. https://azure.microsoft.com/files/Features/Reliability/AzureResiliencyInfographic.pdf https://azure.microsoft.com/files/Features/Reliability/Azure...
- latchkey 2y ago
  Not just the fine print, but also look at how they present themselves. A provider with pictures of equipment and detailed specifications is always going to be more interesting than a provider with just a marketing website and a "contact us" page.
neom 2y ago
This reads exactly like what people said about DigitalOcean when we launched it.
- deleted 2y ago
  [deleted]
- imglorp 2y ago
  How was DO able to provide what AWS didn't want to? Was it purely margins?
  - neom 2y ago
    AWS just really didn't want to, very different market segment. They were doing a pure enterprise play, looking to capture most of the enterprise. We were doing a b2c play that we presumed over time would suck us up into the SMB. My theory was we had like 1% risk from them. From what I could tell Jeff and Jassy had zero interest in our segment. I left just before the IPO but when we started it, the margin was about 60%, after we figured out how many VMs we could comfortable fit on the box, Ben U just did napkin math and said "50% seems like a fine enough margin to start"
- count 2y ago
  To be fair, DO was muuuch sketchier in the past (eg https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097). Launching any multitenant system is HARD. Many of them are held together with bubble gum and good intentions….
  - neom 2y ago
    Boy I'm never going to live that one down around here huh? Hackernews always going to keep you honest, ha. :D
foobiekr 2y ago
You should consider the possibility that one outcome is that no one is going to make money offering H100s.

dijit 2y ago

> 999999% uptime Assuming you mean 99.9999%; your hyperscaler isn't giving you that. MTBF is comparable. It's hardware at the end of the day, the VM hypervisor isn't giving you anything on GPU instances because those GPU instances aren't possible to live-migrate. (even normal VMs are really tricky). In a country with a decent power grid and a UPS (or if you use a colo-provider) you're going to get the same availability guarantee of a machine, maybe even slightly higher because less moving parts. I think this "cloud is god" mentality betrays the fact that server hardware is actually hugely reliable once it's working; and the cloud model literally depends on this fact. The reliability of cloud is simply the reliability of hardware; they only provided an abstraction on management not on reliability.

llm_trw 2y ago
I think people just don't realize how big computers have gotten since 2006. A t2.micro was an ok desktop computer back then. Today you can have something 1000 times as big for a few tens of thousands. You can easily run a company that serves the whole of the US out of a closet.
- dartos 2y ago
  Well you’d probably also at least want a cdn in each region, so like 3 closets.
  - jgalt212 2y ago
    Cloudflare caching of static resources is cheap, so back to one closet. But three if you want to be pure and totally cloudless.
    - felixgallo 2y ago
      With one closet you can also lose the entire business if one water pipe breaks or one wire goes bad in drywall. Back to three closets.
      - llm_trw 2y ago
        If only it were possible to make backups. Alas no such technology exists.
        LoganDark 2y ago
        Yes. That's what the other closets are for. Redundancy.
        dartos 2y ago
        backups do not prevent downtime.
        krab 2y ago
        Yes. That's a risk assessment every company must make. What's the probability of downtime vs the development slowdown and the operating costs of a fully redundant infrastructure? I worked for a payments company (think credit cards). We designed the system to maintain very high availability in the payment flow. Multi-region, multi-AZ in AWS. But all other flows such as user registration, customer care or even bill settlement had to stop during that one incident when our main datacenter lost power after a testing switch. The outage lasted for three hours and it happened exactly once in five years. In that specific case, investing into higher availability by architecting in more redundancy would not be worth it. We had more downtime caused by bad code and not well thought out deployments. But that risk equation will be different for everyone.
- geodel 2y ago
  Well the problem nowadays is what can be done has become what must be done. totally bypassing on question of what should be done. So now instead of single service serving 5 million requests in a business is replaced by 20 micro services generating traffic of 150 million requests with distributed transactions, logging (MBs of log per request), monitoring, metrics and so on. All leading to massive infrastructure bloat. Do it for dozen more applications and future is cloudy now. Once management is convinced by sales people or consultants any technical argument can be brushed away as not seeing the strategic big picture of managing enterprise infrastructure.
- JohnBooty 2y ago
  It's just wild to me how seemingly nobody is exploiting this. Our industry has really lost sight of reality and the goals we're trying to achieve. Sufficient scalability, sufficient performance, and as much developer productivity as we can manage given the other two constraints. That is the goal, not a bunch of cargo-culty complex infra. If you can achieve it with a single machine, fucking do it. A monolith-ish app, running on e.g. an Epyc with 192 cores and a couple TB of RAM???? Are you kidding me? That is so much computing power, to the point where for a lot of scenarios it can replace giant chunks of complex cloud infrastructure. And for something approaching a majority of businesses it can probably replace all of it. (Yes, I know you need at least one other "big honkin server", located elsewhere, for failover. And yes, this doesn't work for all sets of requirements, etc)
  - _3u10 2y ago
    Servers that big rarely fail anyway, everything is hotswap and redundant.
    - latchkey 2y ago
      The GPUs/OAM,UBB are not redundant and they do fail. From what I hear, Nvidia has an exceptionally high failure rate.
  - uid65534 2y ago
    I feel this every day I talk with cloud-brained coworkers. I manage an infrastructure with tens of thousands of VMs and everyone is obsessed with auto scaling and clustering and every other thing the vendor sales dept shoved down their throats while simultaneously failing to realize that they could spend <5% of what we currently do and just use the datacenter cages we _already have_ and a big fat rack of 2S 9754 1U servers. The kicker? These VMs are never more than 8 cores a piece, and applications never scale to more than 3 or 4 in a set. With sub 40% CPU utilization each. Most arguments against cloud abuse like this get ignored because VPs see Microsoft (Azure in this case) as some holy grail for everything and I frankly don't have it in me to keep fighting application dev teams that don't know anything about server admin. And that's without getting into absolutely asinine price/perf SaaS offerings like Cosmos DB.
zaptrem 2y ago
As someone who has done a bunch of large scale ML on hyperscaler hardware I will say the uptime is nowhere near 99.9999%. Given a cluster of only a few hundred GPUs one or multiple failures is a near certainty to the point where we spend a bunch of time on recovery time optimization.
yencabulator 2y ago
> (even normal VMs are really tricky) For what it's worth, GCP routinely live-migrates customer VMs to schedule hardware for maintenance/decommissioning when hardware sensors start indicating trouble. It's standard everyday basic functionality by now, but only for the vendors who built the feature in from the beginning.
- dijit 2y ago
  I’m aware, but it won’t work for gpu accelerated workloads.
traceroute66 2y ago
> instances aren't possible to live-migrate Some of the cloud providers don't even do live-migration. They adhere to the cloud mantra of "oh well, its up to the customer to spin up and carry on elsewhere". I have it on good authority that some of them don't even take A+B feeds to their DC suites - and then have the chutzpah to shout at the DC provider when their only feed goes down, but that's another story... :)
dijit 2y ago
meta: I'm always interested how the votes go on comments like this. I've been watching periodically and it seems like I get "-2" at random intervals. This is not the first time that "low yield" karma comments have sporadic changes to their votes. It seems unlikely at the rate of change (roughly 3-5 point changes per hour) that two people would simultaneously (within a minute) have the same desire to flag a comment, so I can only speculate that: A) Some people's flag is worth -2 B) Some people, passionate about this topic, have multiple accounts C) There's bots that try to remain undetected by making only small adjustments to the conversation periodically. I'm aware that some peoples job very strongly depends on the cloud, but nothing I said could be considered off topic or controversial: Cloud for GPU compute relies on hardware reliability just like everything else does. This is fact. Regardless of this, the voting behaviour on my comments such as this are extremely suspicious.
everforward 2y ago
> The reliability of cloud is simply the reliability of hardware; they only provided an abstraction on management not on reliability. This isn't really true. I mean it's true in the sense that you could get the same reliability on-premise given a couple decades of engineer hours, but the vast majority of on-premise deployments I have seen have significantly lower reliability than clouds and have few plans to build out those capabilities. E.g. if I exclude public cloud operator employers, I've never worked for a company that could mimick an AZ failover on-prem and I've worked for a couple F500s. As far as I can recall, none of them have even segmented their network beyond the management plane having its own hardware. The rest of the DC network was centralized; I recall one of them in specific because an STP loop screwed up half of it at one point. Part of paying for the cloud is centralizing the costs of thinking up and implementing platform-level reliability features. Some of those things are enormously expensive and not really practical for smaller economies of scale. Just one random example is tracking hardware-level points of failure and exposing that to the scheduler. E.g. if a particular datacenter has 4 supplies from mains and each rack is only connected to a single one of those supplies, when I schedule 4 jobs to run there it will try to put each job in a rack with a separate power supply to minimize the impact of losing a mains. Ditto with network, storage, fire suppression, generators, etc, etc, etc. That kind of thing makes 0 economic sense for an individual company to implement, but it starts to make a lot of sense for a company who does basically nothing other than manage hardware failures.
wkat4242 2y ago
> Assuming you mean 99.9999%; your hyperscaler isn't giving you that. MTBF is comparable. Yeah we've already had about a day's worth of downtime this year on office 365 and Microsoft is definitely a hyperscaler. So that's 99.3% at best.

pico_creator 2y ago

Someone is losing the money. It’s elaborated in the article how and why this happens TLDR, VC money, is being burnt/lost

shermantanktop 2y ago
Tons of VC money burned in pursuit of low-probability success. It’s no wonder that some people find it easier to scam VCs than it is to build a real business.

hnaccount_rng 2y ago

Can you elaborate on the cost basis? With how little could a very lean operation still make money? I know that's basically impossible to answer generically, especially given that the recurring cost is likely already zero, given that the GPUs are already paid...

dx034 2y ago

Since most applications aren't latency sensitive, space and power can be nearly free by setting up the data center in a place where it's cold, there's nearly free electricity and few people live. Leaves you with cost for infrastructure and connectivity, but I guess electricity prices shouldn't be the issue?

tonetegeatinst 2y ago
I'd think cost of internet would be the big issue even if can afford the AI hardware. In rural areas or even with low population it takes forever to get fiber to roll out and if your selling access to your hardware infrastructure then you really want to get a direct connection to the nearest IX so you can offer customers the best speed for accessing data and the IX would probably be one of the few places you might be able to get 400G or higher direct fiber. But if your hooking up to a IX chances are your not an end user but a autonomous system and already are shoving moving and signing NDA's to be a peer with other Autonomous Systems in the exchange and be able to bgp announce. (Source - my old highschool networking class where I got sick of my shitty internet and looked into how I could get fiber from an exchange. I'm probably mistaken on stuff here as it was years ago and its either wrong or outdated from all those years ago.)
- oasisbob 2y ago
  Assuming rural areas have less fiber availability isn't always a good assumption. In NW Washington state at least, the rural counties (Whatcom, Island, Skagit, etc) have had a robust market in dark fiber for over two decades. The normal telcos weren't responsive to need, so private carriers picked up the slack. When I was last involved in this market, you could get a P2P strand, including reasonable buildout, for less than a cost of a T1 line with a two-year commit. The tiny four-branch credit union I worked for had dedicated fiber loops between all our locations, no big deal. It was great.
serjester 2y ago
Ambient cooling can only go so far. At the end of the day if you have a rack of GPU’s using 6000 watts per node, you’re going to need some very serious active cooling regardless of your location. You’ll save a little but it’s a small percentage of your overall costs.

acd10j 2y ago

May be their business model is running compute at loss and stealing ip/code from people using platform?

michaelt 2y ago

> There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power... At the highest power settings, H100s consume 400 W. Add another 200 W for CPU/RAM. Assume you have an incredibly inefficient cooling system, so you also need 600 W of cooling. Google tells me US energy prices average around 17 cents/kWh - even if you don't locate your data centre somewhere with cheap electricity. 17 cents/kWh * 1200 watts * 1 hour is only 20.4 cents/hour.

sandworm101 2y ago
For the fuller math one has to include the cost of infrastructure financing, which is tied to interest rates. Given how young most of these H100 shops are, I assume that they pay more to service their debts than for power.
- Wytwwww 2y ago
  > I assume that they pay more to service their debts than for power. Well yes, because for GPU datacentres fixed/capital costs make up a much higher fraction than power and other expenses than for CPUs. To such an extent that power usage barely even matters. A $20k that uses 1 kW ( which is way more than it would in reality ) 24x7 would cost $1.3k to run per year at 0.15$ per kWh, that's almost insignificant compared to depreciation. The premise is that nobody could make any money by renting H100s for 2$ even if they got them for free unless they only had free power. That makes no sense whatsoever when you can get 2x AMD EPYC™ 9454P servers at 2x408 W (for full system) for around $0.70 in a German data center.
ckastner 2y ago
That's just the power. If one expects a H100 to run for three years at full load, 24 x 365 x 3 = 26280. Assuming a price of $25K per H100, that means about $1/h to amortize costs. Hence the unless they stole them, I guess. Factor in space, networking, cooling, security, etc., and $2 really do seem undoable.
- Negitivefrags 2y ago

bjornsing 2y ago

> There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power... That’s essentially what the OP says. But once you’ve already invested in the H100s you’re still better off renting them out for $2 per hour rather than having them idle at $0 per hour.

Wytwwww 2y ago
Then how come you can still get several last gen EPYC or Xeon systems that would use the same amount of power for under $1 per hour? For datacentre GPUs the energy, infrastructure and other variable costs seem to be relatively insignificant to fixed capital costs. Nvidia's GPUs are just extremely expensive relative to how much power they use (compared to CPUs). > H100s you’re still better off renting them out for $2 per hour rather than having them idle at $0 per hour. If you're barely breaking even at $2 then immediately selling them would seem like the only sensible option (depreciation alone is significantly higher than the cost power of running a H100 24x365 at 100% utilization).
- bjornsing 2y ago
  > If you're barely breaking even at $2 then immediately selling them would seem like the only sensible option (depreciation alone is significantly higher than the cost power of running a H100 24x365 at 100% utilization). If you can then probably yes. But why would someone else buy them (at the price you want), when they can rent at $2 per hour instead?
  - Wytwwww 2y ago
    I don't think the why matters as long as people are buying them at very high prices, which they seemingly still are.
    - bjornsing 2y ago
      What makes you think they are?

scotty79 2y ago

> There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power... I think that's the point. Trying to buy and run H100s now either for yourself or for someone else to rent it is a terrible investment because of oversupply. And prices you can get for compute are not enough to cover the costs.

marcyb5st 2y ago

But it is about minimizing losses, not making profits. If you read the article, such prices happen because a lot of companies bought hardware reservations for the next few years. Instead of keeping the hardware idle (since they pay for it anyway), they rent it out on the cheap to recoup something.

rajnathani 2y ago

From your bio, your company is Hot Aisle. This company TensorWave covered by TechCrunch [0] this week sounds very similar, I almost thought it was the same! Anyway, best of luck, we need more AMD GPU compute. [0] https://techcrunch.com/2024/10/08/tensorwave-claims-its-amd-powered-cloud-for-ai-will-give-nvidia-a-run-for-its-money/ https://techcrunch.com/2024/10/08/tensorwave-claims-its-amd-...

latchkey 2y ago
Thanks! Definitely not the same at all.

tasuki 2y ago

> If you're trying to do the latter, you're not considering how you are risking your business on unreliable compute. What do you mean by "risking your business on unreliable compute"? Is there a reason not to use one of these to train whatever neural nets one's business needs?

oefrha 2y ago
Well, someone who’s building a GPU renting service right now obviously wants to scare you into using expensive and “reliable” services; the market crashing is disastrous for them. The reality is high price is hardly an indicator of reliability, and the article very clearly explains why H100 hours are being sold at $2 or less, and it’s not because of certain providers lacking reliability.
- latchkey 2y ago
  Nah, don't be silly. No need to scare anyone into anything. Use whatever you want to use. My point in saying and of this is to simply point out that we offer this service to people who value these things.
lazide 2y ago
If it crashes half way through, you don’t get a useful model, and you’re still on the hook for the rental costs to get there maybe?
- latchkey 2y ago
  Depends on the SLA.
- tasuki 2y ago
  That's... possible? But a little unlikely. I think I'll take that risk over paying more for your allegedly more reliable GPUs anytime.

fhars 2y ago

I think this is what they are insinuating with the "Hot the Bubble Burst" in the headline. You are not expected to make money if you have invested in a bursting bubble.

wolfgangK 2y ago

For training, doesn't checkpoint saving make high reliability a moot point ? Why pay for 99.99999? uptime when you can restart your training from last/best model ?

traceroute66 2y ago

> 999999% uptime I've said it before and I've said it again.... Read the cloud provider small-print before you go around boasting about how great their SLAs are. Most of the time they are not worth the paper they are written on.

kjs3 2y ago
This is beyond true. Read and understand what your cloud SLAs are, not what you think they are or what you think they should be. There was significant consternation generated when I pointed out that the SLA for availability for an Azure storage blob was only 4 nines with zone redundancy. https://azure.microsoft.com/files/Features/Reliability/AzureResiliencyInfographic.pdf https://azure.microsoft.com/files/Features/Reliability/Azure...
latchkey 2y ago
Not just the fine print, but also look at how they present themselves. A provider with pictures of equipment and detailed specifications is always going to be more interesting than a provider with just a marketing website and a "contact us" page.

neom 2y ago

This reads exactly like what people said about DigitalOcean when we launched it.

deleted 2y ago
[deleted]
imglorp 2y ago
How was DO able to provide what AWS didn't want to? Was it purely margins?
- neom 2y ago
  AWS just really didn't want to, very different market segment. They were doing a pure enterprise play, looking to capture most of the enterprise. We were doing a b2c play that we presumed over time would suck us up into the SMB. My theory was we had like 1% risk from them. From what I could tell Jeff and Jassy had zero interest in our segment. I left just before the IPO but when we started it, the margin was about 60%, after we figured out how many VMs we could comfortable fit on the box, Ben U just did napkin math and said "50% seems like a fine enough margin to start"
count 2y ago
To be fair, DO was muuuch sketchier in the past (eg https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097). Launching any multitenant system is HARD. Many of them are held together with bubble gum and good intentions….
- neom 2y ago
  Boy I'm never going to live that one down around here huh? Hackernews always going to keep you honest, ha. :D

foobiekr 2y ago

You should consider the possibility that one outcome is that no one is going to make money offering H100s.