3 ms·

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

by m4rtink 10mo ago

nl 10mo ago
100% yes. The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) In some scenario where new investment stops flowing and some AI companies go bankrupt all that compute will be looking for a market. Inference providers are already profitable so with cheaper hardware it will mean even cheaper AI systems.
- oa335 10mo ago
  > Inference providers are already profitable. That surprises me, do you remember where you learned that?
  - nl 10mo ago
    Lots of sources, and you can do the math yourself. Here's a few good ones: https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... (suggests Deepseek is making 80% raw margin on inference) https://www.snellman.net/blog/archive/2025-06-02-llms-are-cheap/ https://www.snellman.net/blog/archive/2025-06-02-llms-are-ch... https://martinalderson.com/posts/are-openai-and-anthropic-really-losing-money-on-inference/ https://martinalderson.com/posts/are-openai-and-anthropic-re... (there's a HN discussion of this where it was pointed out this overestimates the costs) https://www.tensoreconomics.com/p/llm-inference-economics-from-first https://www.tensoreconomics.com/p/llm-inference-economics-fr... (long, but the TL;DR is that serving Lllama 3.3 70B costs around $0.28/million tokens input, $0.95 output at high utilization. These are close to what we see in the market: https://artificialanalysis.ai/models/llama-3-3-instruct-70b/providers?pricing-relationships=input-and-output-pricing https://artificialanalysis.ai/models/llama-3-3-instruct-70b/... )
- AyyEye 10mo ago
  You should probably disclose that you're a CTO at an AI startup, I had to click your bio to see that. > The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) All going into the hands of a small group of people that will soon need to pay the piper. That said, VC backed tech companies almost universally pull the rug once the money stops coming in. And historically those didn't have the trillions of dollars in future obligations that the current compute hardware oligopoly has. I can't see any universe where they don't start charging more, especially now that they've begun to make computers unaffordable for normal people. And even past the bottom dollar cost, AI provides so many fun, new, unique ways for them to rug pull users. Maybe they start forcing users to smaller/quantized models. Maybe they start giving even the paying users ads. Maybe they start inserting propaganda/ads directly into the training data to make it more subtle. Maybe they just switch out models randomly or based on instantaneous hardware demand, giving users something even more unstable than LLMs already are. Maybe they'll charge based on semantic context (I see you're asking for help with your 2015 Ford Focus. Please subscribe to our 'Mechanic+' plan for $5/month or $25 for 24 hours). Maybe they charge more for API access. Maybe they'll charge to not train on your interactions. I'll pass, thanks.
  - nl 10mo ago
    I'm not longer CTO at an AI startup. Updated, but don't actually see how that is relevant. > All going into the hands of a small group of people that will soon need to pay the piper. It's not very small! On the inference side there are many competitive providers as well as the option of hiring GPU servers yourself. > And historically those didn't have the trillions of dollars in future obligations that the current compute hardware oligopoly has. I can't see any universe where they don't start charging more, especially now that they've begun to make computers unaffordable for normal people. I can't say how strongly I disagree with this - it's just not how competition works, or how the current market is structured. Take gpt-oss-120B as an example. It's not frontier level quality but it's not far off and certainly gives a strong redline that open source models will never get less intelligent than. There is a competitive market in hosting providers, and you can see the pricing here: https://artificialanalysis.ai/models/gpt-oss-120b/providers?pricing-relationships=input-and-output-pricing https://artificialanalysis.ai/models/gpt-oss-120b/providers?... In what world is there a way in which all the providers (who are want revenue!) raise prices above the premium price Cerebas is charging for their very high speed inference? There's already Google, profitable serving at the low-end at around half the price of Cerebas (but then you have to deal with Google billing!) The fact that Azure/Amazon are all pricing exactly the same as 8(!) other providers as well as the same price https://www.voltagepark.com/blog/how-to-deploy-gpt-oss-on-a-gpu-server https://www.voltagepark.com/blog/how-to-deploy-gpt-oss-on-a-... gives for running your own server shows how the economics work on NVidia hardware. There's no subsidy going on there. This is on hardware that is already deployed. That isn't suddenly going to get more expensive unless demand increases... in which case the new hardware coming online over the next 24 months is a good investment, not a bad one!
- jeremyjh 10mo ago
  Datacenters full of GPU hosts aren't like dark fiber - they require massive ongoing expense, so the unit economics have to work really well. It is entirely possible that some overbuilt capacity will be left idle until it is obsolete.
  - nl 10mo ago
    The ongoing costs are mostly power, and aren't that massive compared to the investment. No one is leaving an H100 cluster not running because the power costs too much - this is why remnants markets like Vast.ai exist.
    - jeremyjh 10mo ago
      They absolutely will leave them idle if the market is so saturated that no one will pay enough for tokens to cover power and other operational costs. Demand is elastic but will not stretch forever. The build out assumes new applications with ROI will be found, and I'm sure they will be, but those will just drive more investment. A massive over build is inevitable.
      - nl 10mo ago
        Of course! But the operational costs are much lower than some people in this thread seem to think. You can find a safe margin for the price by looking at aggregators. https://gpus.io/gpus/h100 https://gpus.io/gpus/h100 is showing $1.83/hour lowest price, around $2.85 average. That easily pays running costs - a H100 server with cooling etc is around $0.10/hour to keep running And a massive overbuild pushes prices down not up!
- blibble 10mo ago
  > The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) which is funded by the dumping when the bubble pops: these DCs are turned off and left to rot, and your capacity drops by a factor of 8192
  - nl 10mo ago
    > which is funded by the dumping What dumping do you mean? Are you implying NVidia is selling H200s below cost? If not then you might be interested to see that Deepseek has released there inference costs here: https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... If they are losing money it's because they have a free app they are subsidizing, not because the API is underpriced.
anonzzzies 10mo ago
They will go down. Or the company will be gone.
TeMPOraL 10mo ago
Doesn't matter now. GP can revisit the math and buy some hardware once the subscription prices actually grow too high.
solatic 10mo ago
You have to remember that companies are kind of fungible in the sense that founders can close old companies and start new ones to walk away from bankruptcies in the old companies. When there's a bust and a lot of companies close up shop, because data centers were overbuilt, there's going to be a lot of GPUs being sold at firesale prices - imagine chips sold at $300k today being sold for $3k tomorrow to recoup a penny on the dollar. There's going to be a business model for someone buying those chips at $3k, then offering subscription prices at little more than the cost of electricity to keep the dumped GPUs running somewhere.
- m4rtink 10mo ago
  I do wonder how usable the hardware will be once the creditors are trying to sell it - as far as I can tell is seems the current trend is more and more custom no-matter-the cost super expensive power-inefficient hardware. The situation might be a lot different than people selling ex-crypto mining GPUs to gamers. There might be a lot of effective scrap that is no longer usable when it is no longer part of a some companies technological fever dream.

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

by m4rtink 10mo ago

Are you sure the subscription will still be affordable after the venture capital flood ends and the dumping stops?

nl 10mo ago

100% yes. The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) In some scenario where new investment stops flowing and some AI companies go bankrupt all that compute will be looking for a market. Inference providers are already profitable so with cheaper hardware it will mean even cheaper AI systems.

oa335 10mo ago
> Inference providers are already profitable. That surprises me, do you remember where you learned that?
- nl 10mo ago
  Lots of sources, and you can do the math yourself. Here's a few good ones: https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... (suggests Deepseek is making 80% raw margin on inference) https://www.snellman.net/blog/archive/2025-06-02-llms-are-cheap/ https://www.snellman.net/blog/archive/2025-06-02-llms-are-ch... https://martinalderson.com/posts/are-openai-and-anthropic-really-losing-money-on-inference/ https://martinalderson.com/posts/are-openai-and-anthropic-re... (there's a HN discussion of this where it was pointed out this overestimates the costs) https://www.tensoreconomics.com/p/llm-inference-economics-from-first https://www.tensoreconomics.com/p/llm-inference-economics-fr... (long, but the TL;DR is that serving Lllama 3.3 70B costs around $0.28/million tokens input, $0.95 output at high utilization. These are close to what we see in the market: https://artificialanalysis.ai/models/llama-3-3-instruct-70b/providers?pricing-relationships=input-and-output-pricing https://artificialanalysis.ai/models/llama-3-3-instruct-70b/... )
AyyEye 10mo ago
You should probably disclose that you're a CTO at an AI startup, I had to click your bio to see that. > The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) All going into the hands of a small group of people that will soon need to pay the piper. That said, VC backed tech companies almost universally pull the rug once the money stops coming in. And historically those didn't have the trillions of dollars in future obligations that the current compute hardware oligopoly has. I can't see any universe where they don't start charging more, especially now that they've begun to make computers unaffordable for normal people. And even past the bottom dollar cost, AI provides so many fun, new, unique ways for them to rug pull users. Maybe they start forcing users to smaller/quantized models. Maybe they start giving even the paying users ads. Maybe they start inserting propaganda/ads directly into the training data to make it more subtle. Maybe they just switch out models randomly or based on instantaneous hardware demand, giving users something even more unstable than LLMs already are. Maybe they'll charge based on semantic context (I see you're asking for help with your 2015 Ford Focus. Please subscribe to our 'Mechanic+' plan for $5/month or $25 for 24 hours). Maybe they charge more for API access. Maybe they'll charge to not train on your interactions. I'll pass, thanks.
- nl 10mo ago
  I'm not longer CTO at an AI startup. Updated, but don't actually see how that is relevant. > All going into the hands of a small group of people that will soon need to pay the piper. It's not very small! On the inference side there are many competitive providers as well as the option of hiring GPU servers yourself. > And historically those didn't have the trillions of dollars in future obligations that the current compute hardware oligopoly has. I can't see any universe where they don't start charging more, especially now that they've begun to make computers unaffordable for normal people. I can't say how strongly I disagree with this - it's just not how competition works, or how the current market is structured. Take gpt-oss-120B as an example. It's not frontier level quality but it's not far off and certainly gives a strong redline that open source models will never get less intelligent than. There is a competitive market in hosting providers, and you can see the pricing here: https://artificialanalysis.ai/models/gpt-oss-120b/providers?pricing-relationships=input-and-output-pricing https://artificialanalysis.ai/models/gpt-oss-120b/providers?... In what world is there a way in which all the providers (who are want revenue!) raise prices above the premium price Cerebas is charging for their very high speed inference? There's already Google, profitable serving at the low-end at around half the price of Cerebas (but then you have to deal with Google billing!) The fact that Azure/Amazon are all pricing exactly the same as 8(!) other providers as well as the same price https://www.voltagepark.com/blog/how-to-deploy-gpt-oss-on-a-gpu-server https://www.voltagepark.com/blog/how-to-deploy-gpt-oss-on-a-... gives for running your own server shows how the economics work on NVidia hardware. There's no subsidy going on there. This is on hardware that is already deployed. That isn't suddenly going to get more expensive unless demand increases... in which case the new hardware coming online over the next 24 months is a good investment, not a bad one!
jeremyjh 10mo ago
Datacenters full of GPU hosts aren't like dark fiber - they require massive ongoing expense, so the unit economics have to work really well. It is entirely possible that some overbuilt capacity will be left idle until it is obsolete.
- nl 10mo ago
  The ongoing costs are mostly power, and aren't that massive compared to the investment. No one is leaving an H100 cluster not running because the power costs too much - this is why remnants markets like Vast.ai exist.
  - jeremyjh 10mo ago
    They absolutely will leave them idle if the market is so saturated that no one will pay enough for tokens to cover power and other operational costs. Demand is elastic but will not stretch forever. The build out assumes new applications with ROI will be found, and I'm sure they will be, but those will just drive more investment. A massive over build is inevitable.
    - nl 10mo ago
      Of course! But the operational costs are much lower than some people in this thread seem to think. You can find a safe margin for the price by looking at aggregators. https://gpus.io/gpus/h100 https://gpus.io/gpus/h100 is showing $1.83/hour lowest price, around $2.85 average. That easily pays running costs - a H100 server with cooling etc is around $0.10/hour to keep running And a massive overbuild pushes prices down not up!
blibble 10mo ago
> The amount of compute in the world is doubling over 2 years because of the ongoing investment in AI (!!) which is funded by the dumping when the bubble pops: these DCs are turned off and left to rot, and your capacity drops by a factor of 8192
- nl 10mo ago
  > which is funded by the dumping What dumping do you mean? Are you implying NVidia is selling H200s below cost? If not then you might be interested to see that Deepseek has released there inference costs here: https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... If they are losing money it's because they have a free app they are subsidizing, not because the API is underpriced.

anonzzzies 10mo ago

They will go down. Or the company will be gone.

TeMPOraL 10mo ago

Doesn't matter now. GP can revisit the math and buy some hardware once the subscription prices actually grow too high.

solatic 10mo ago

You have to remember that companies are kind of fungible in the sense that founders can close old companies and start new ones to walk away from bankruptcies in the old companies. When there's a bust and a lot of companies close up shop, because data centers were overbuilt, there's going to be a lot of GPUs being sold at firesale prices - imagine chips sold at $300k today being sold for $3k tomorrow to recoup a penny on the dollar. There's going to be a business model for someone buying those chips at $3k, then offering subscription prices at little more than the cost of electricity to keep the dumped GPUs running somewhere.

m4rtink 10mo ago
I do wonder how usable the hardware will be once the creditors are trying to sell it - as far as I can tell is seems the current trend is more and more custom no-matter-the cost super expensive power-inefficient hardware. The situation might be a lot different than people selling ex-crypto mining GPUs to gamers. There might be a lot of effective scrap that is no longer usable when it is no longer part of a some companies technological fever dream.