5 ms·

H100 is nearing five years and costs more to buy a used one now than a new one when it was released :) You are completely missing the bet these companies are m

by fooker 3mo ago

H100 is nearing five years and costs more to buy a used one now than a new one when it was released :) You are completely missing the bet these companies are making. They think can outlast their competitors and capture a larger portion of the pie while the cost of inference keeps going down dramatically. If you haven't been paying attention, the cost is about 1/100th of what it was in 2024. This is the trajectory pretty much every technology has followed. Of course there will be market crashes and corrections and things like that and most companies won't survive, but the bet is that whoever survives ends up doing pretty well.

boesboes 3mo ago
What costs are 1/100th?
- fooker 3mo ago
  Of serving a (approximately) gpt4 sized model.
  - alangibson 3mo ago
    So the number is irrelevant. No one wants yesterdays newspaper. The only relevant number is the price to serve a frontier or near-frontier model.
  - barumrho 3mo ago
    Is this true? Hardware costs have only gone up during this time. Are you referring to electricity cost to serve these models? (i.e. compute got more efficient?)
  - usrusr 3mo ago
    What made the cost go down? Can't be cheaper used H100, can't be cheaper RAM. A revolutionary breakthrough in hardware use per query?
  - underlipton 3mo ago
    Does that include the capital costs of spinning up to the current models/scale or is it just running costs? Also, lost revenue from other services being degraded by shifting resources to supporting training/serving models (Google Search...)?
khurs 3mo ago
>H100 is nearing five years and costs more to buy a used one now than a new one when it was released :) Because everyone is buying as they want to run their own models and not pay for a cloud service?
- fooker 3mo ago
  Because there's no supply, data centers with these GPUs are running reasonably well.
- mattnewton 3mo ago
  Because demand for inference tokens is above supply
lardosaurusrex 3mo ago
You just stated yourself that it costs more now used than when they were new. If everyone's running local then why are these larger companies dumping cash into data centres?
- fooker 3mo ago
  Economies of scale. You need a cluster of 8-12 H100s to run the largest models locally. It doesn't make sense to run these locally yet unless your use case also involves making it available for several dozen concurrent users.
  - kingleopold 3mo ago
    not to miss, future models will be more compute hungry too. Current hardware prices are still goin up and no it's not cheaper to run your AI for like %99 of the people because of lots of costs, it's not just hardware.
    - lardosaurusrex 3mo ago
      I think this fails to take into account how many people are fine with "fast enough" vs "fastest". I've seen people happily use AI that takes several minutes to generate text or edit an image because to them they already aren't using their computer when they tell it to start; they just grab their phone and walk away and come back only to check in on it. I feel like people here and on other technology discussions -- although it's worse here -- don't seem to parse what being the minority means. They know they're one of the few to have access to such incredible hardware -- whether it be rented or purchased for way too much cash -- but they only see their own kin; their own ilk. They only compare themselves to the best. The reality is that nobody expects data centre speed nor power in their own home and are satisfied to just go "haha its thinking" and let their computer quietly tick in the background as opposed to paying outragious prices for subscriptions or hardware.
      - gmadsen 3mo ago
        That is completely discounting future capabilities and new use cases. Sure in 10 years you will have current SOTA locally, but in no way is it obvious we are anywhere near the limit of marginal value from improved capability
Imustaskforhelp 3mo ago
> They think can outlast their competitors and capture a larger portion of the pie while the cost of inference keeps going down dramatically. We are also within an arms race of training newer larger models with more speed while discontinuing older models. Gemini/Chatgpt have already discontinued their models from 2024 (iirc) because they are using all their compute in serving/training newer models. Being quite frank, nobody is serving a model from 2024 as the intended use-case while having very little moat as open source models are catching up. > Of course there will be market crashes and corrections and things like that and most companies won't survive, but the bet is that whoever survives ends up doing pretty well. How so, by raising the prices? because the current prices aren't sustainable and I feel as if there would certainly be companies which will try for one reason or other to be cheaper to capture the market share because of the larger promise of whoever is able to get as market share. I had once thought about it and I don't think that even in an ideal world, they would end up doing pretty well given no moat. Also even if a company survives and ends up being one of the survivors and makes profit in the ideal scenario you mention, then within some years other companies will try again and construct more datacenters and end up driving the prices down for everyone, so nobody knows how things might look down for 2-3 years let alone a decade, so I remain a bit skeptic currently so. I had actually thought some on the economics of datacenters and I found it to be very related to power. The only ones which seems to be making money might be the power generators actually because power is the actual bottleneck rather than GPU's in datacenters from my understanding. Though the power is raised at the cost of electricity bill increases for everybody including people living in houses. The job prospects are minimal as well, as a nation, aside from just getting investment just for the sake of it because AI's trendy right now, I feel like its a net negative deal for people living there.
maxglute 3mo ago
The fact that supply constrained GPUs holding value is negative indicator. It's like the tulips mania except bubble enthusiasts are buying wilted/dead tulips because live tulips supply constrained due to irrational demand. Now GPUs has more gross utility than tulips but seems like at current revenue/capex spend, every GPU is still negative net financial yield - they lose money - literally buying tulips and watching it wilt. Economically, better off simply not buying and losing more. That's the level of economic irrationality at place sustaining bubble, at least for hyperscaler/big tech balance sheet - small operators logic different and antagonistic to big operator demand/business model. If cost of inference goes down 100x, would need 100x more demand. This makes overspending on GPU even more irrational. Jevons this, Jevons that but ultimately irrelevant. At end of day, leading players, hungergame winner candidates is saddling themselves with so much debt, even if they survive, post crash they are immediately uncompetitive against new entrant with blank slate and newer gen, more efficient GPUs that will be cheaper to buy/operate post crash when hardware prices will revert to mean. It doesn't matter if some of the current players survive, they've basically stabbed and weakened themselves so much any healthy upstart in the future can wipe them out unless they lock in legislative protection... safety regulations, ban open source models etc. That is the new bet, regulatory capture moat, because economic bet is entirely lost, especially with open models eroding mote.
- fooker 3mo ago
  You're making a classic philosophical error here by accidentally anthropomorphizing companies :) A company losing out on a risky bet and failing, letting a new upstart rise up is pretty natural. A large fraction of experts from the failed companies continue at the new ones, business as usual. There are engineering teams at $BIGTECH now full of OS, database, or compiler experts from XP, Sun, HP etc. This natural ability of companies to take risky bets is what made silicon valley successful.
  - maxglute 3mo ago
    Companies are run by people, 100 billion dollar companies are ran by people who want to stay billionaires and will street accordingly. But I'm not sure what we're disagreeing on, yes the talent will migrate, AI industry will eventually settle on some none bubble equilibrium, but that doesn't mean current AI economics is sensible, or inflated hardware costs beyond yield is not danger indicator. Like yeah, risky bets are burning, most people are going to move and carry their technical expertise on instead of unalive themselves or flip burgers, but that's independent of whether business models and broader economy is going to explode.
justsid 3mo ago
Isn’t that the same bet that famously profitable companies like Uber did in the ride share market?
- Imustaskforhelp 3mo ago
  Uber is a two sided marketplace and was still famously unprofitable for at the same time. Over 14 years, Uber burnt ~31 Billion dollars and in nearly whereas the amount invested within AI seems to be within Trillions at this point and in near future with a product which doesn't have much moat and shaky financials on profit.
- fooker 3mo ago
  It's not the exact same bet, but it rhymes. It worked out for Uber, they are wildly profitable now after spending a decade losing money. Also worked out as Amazon managed to outlast all the dotcom era e-commerce competitors while being unprofitable.
  - disgruntledphd2 3mo ago
    > It worked out for Uber, they are wildly profitable now after spending a decade losing money. Uber have a 10% margin, which is definitely not what I'd consider wildly profitable. (Their post tax numbers look better, because of accumulated losses).
etempleton 3mo ago
This is absolutely the calculus. There is no moat. It is survival of the best financed. Open AI and Anthropic are in very precarious situations.

by fooker 3mo ago

boesboes 3mo ago

What costs are 1/100th?

fooker 3mo ago
Of serving a (approximately) gpt4 sized model.
- alangibson 3mo ago
  So the number is irrelevant. No one wants yesterdays newspaper. The only relevant number is the price to serve a frontier or near-frontier model.
- barumrho 3mo ago
  Is this true? Hardware costs have only gone up during this time. Are you referring to electricity cost to serve these models? (i.e. compute got more efficient?)
- usrusr 3mo ago
  What made the cost go down? Can't be cheaper used H100, can't be cheaper RAM. A revolutionary breakthrough in hardware use per query?
- underlipton 3mo ago
  Does that include the capital costs of spinning up to the current models/scale or is it just running costs? Also, lost revenue from other services being degraded by shifting resources to supporting training/serving models (Google Search...)?

khurs 3mo ago

>H100 is nearing five years and costs more to buy a used one now than a new one when it was released :) Because everyone is buying as they want to run their own models and not pay for a cloud service?

fooker 3mo ago
Because there's no supply, data centers with these GPUs are running reasonably well.
mattnewton 3mo ago
Because demand for inference tokens is above supply

lardosaurusrex 3mo ago

You just stated yourself that it costs more now used than when they were new. If everyone's running local then why are these larger companies dumping cash into data centres?

fooker 3mo ago
Economies of scale. You need a cluster of 8-12 H100s to run the largest models locally. It doesn't make sense to run these locally yet unless your use case also involves making it available for several dozen concurrent users.
- kingleopold 3mo ago
  not to miss, future models will be more compute hungry too. Current hardware prices are still goin up and no it's not cheaper to run your AI for like %99 of the people because of lots of costs, it's not just hardware.
  - lardosaurusrex 3mo ago
    I think this fails to take into account how many people are fine with "fast enough" vs "fastest". I've seen people happily use AI that takes several minutes to generate text or edit an image because to them they already aren't using their computer when they tell it to start; they just grab their phone and walk away and come back only to check in on it. I feel like people here and on other technology discussions -- although it's worse here -- don't seem to parse what being the minority means. They know they're one of the few to have access to such incredible hardware -- whether it be rented or purchased for way too much cash -- but they only see their own kin; their own ilk. They only compare themselves to the best. The reality is that nobody expects data centre speed nor power in their own home and are satisfied to just go "haha its thinking" and let their computer quietly tick in the background as opposed to paying outragious prices for subscriptions or hardware.
    - gmadsen 3mo ago
      That is completely discounting future capabilities and new use cases. Sure in 10 years you will have current SOTA locally, but in no way is it obvious we are anywhere near the limit of marginal value from improved capability

Imustaskforhelp 3mo ago

> They think can outlast their competitors and capture a larger portion of the pie while the cost of inference keeps going down dramatically. We are also within an arms race of training newer larger models with more speed while discontinuing older models. Gemini/Chatgpt have already discontinued their models from 2024 (iirc) because they are using all their compute in serving/training newer models. Being quite frank, nobody is serving a model from 2024 as the intended use-case while having very little moat as open source models are catching up. > Of course there will be market crashes and corrections and things like that and most companies won't survive, but the bet is that whoever survives ends up doing pretty well. How so, by raising the prices? because the current prices aren't sustainable and I feel as if there would certainly be companies which will try for one reason or other to be cheaper to capture the market share because of the larger promise of whoever is able to get as market share. I had once thought about it and I don't think that even in an ideal world, they would end up doing pretty well given no moat. Also even if a company survives and ends up being one of the survivors and makes profit in the ideal scenario you mention, then within some years other companies will try again and construct more datacenters and end up driving the prices down for everyone, so nobody knows how things might look down for 2-3 years let alone a decade, so I remain a bit skeptic currently so. I had actually thought some on the economics of datacenters and I found it to be very related to power. The only ones which seems to be making money might be the power generators actually because power is the actual bottleneck rather than GPU's in datacenters from my understanding. Though the power is raised at the cost of electricity bill increases for everybody including people living in houses. The job prospects are minimal as well, as a nation, aside from just getting investment just for the sake of it because AI's trendy right now, I feel like its a net negative deal for people living there.

maxglute 3mo ago

The fact that supply constrained GPUs holding value is negative indicator. It's like the tulips mania except bubble enthusiasts are buying wilted/dead tulips because live tulips supply constrained due to irrational demand. Now GPUs has more gross utility than tulips but seems like at current revenue/capex spend, every GPU is still negative net financial yield - they lose money - literally buying tulips and watching it wilt. Economically, better off simply not buying and losing more. That's the level of economic irrationality at place sustaining bubble, at least for hyperscaler/big tech balance sheet - small operators logic different and antagonistic to big operator demand/business model. If cost of inference goes down 100x, would need 100x more demand. This makes overspending on GPU even more irrational. Jevons this, Jevons that but ultimately irrelevant. At end of day, leading players, hungergame winner candidates is saddling themselves with so much debt, even if they survive, post crash they are immediately uncompetitive against new entrant with blank slate and newer gen, more efficient GPUs that will be cheaper to buy/operate post crash when hardware prices will revert to mean. It doesn't matter if some of the current players survive, they've basically stabbed and weakened themselves so much any healthy upstart in the future can wipe them out unless they lock in legislative protection... safety regulations, ban open source models etc. That is the new bet, regulatory capture moat, because economic bet is entirely lost, especially with open models eroding mote.

fooker 3mo ago
You're making a classic philosophical error here by accidentally anthropomorphizing companies :) A company losing out on a risky bet and failing, letting a new upstart rise up is pretty natural. A large fraction of experts from the failed companies continue at the new ones, business as usual. There are engineering teams at $BIGTECH now full of OS, database, or compiler experts from XP, Sun, HP etc. This natural ability of companies to take risky bets is what made silicon valley successful.
- maxglute 3mo ago
  Companies are run by people, 100 billion dollar companies are ran by people who want to stay billionaires and will street accordingly. But I'm not sure what we're disagreeing on, yes the talent will migrate, AI industry will eventually settle on some none bubble equilibrium, but that doesn't mean current AI economics is sensible, or inflated hardware costs beyond yield is not danger indicator. Like yeah, risky bets are burning, most people are going to move and carry their technical expertise on instead of unalive themselves or flip burgers, but that's independent of whether business models and broader economy is going to explode.

justsid 3mo ago

Isn’t that the same bet that famously profitable companies like Uber did in the ride share market?

Imustaskforhelp 3mo ago
Uber is a two sided marketplace and was still famously unprofitable for at the same time. Over 14 years, Uber burnt ~31 Billion dollars and in nearly whereas the amount invested within AI seems to be within Trillions at this point and in near future with a product which doesn't have much moat and shaky financials on profit.
fooker 3mo ago
It's not the exact same bet, but it rhymes. It worked out for Uber, they are wildly profitable now after spending a decade losing money. Also worked out as Amazon managed to outlast all the dotcom era e-commerce competitors while being unprofitable.
- disgruntledphd2 3mo ago
  > It worked out for Uber, they are wildly profitable now after spending a decade losing money. Uber have a 10% margin, which is definitely not what I'd consider wildly profitable. (Their post tax numbers look better, because of accumulated losses).

etempleton 3mo ago

This is absolutely the calculus. There is no moat. It is survival of the best financed. Open AI and Anthropic are in very precarious situations.