3 ms·

My prediction is that hardware costs will make open source models impractical for the foreseeable future. Yes, tinkerers and enthusiasts will continue to make

My prediction is that hardware costs will make open source models impractical for the foreseeable future. Yes, tinkerers and enthusiasts will continue to make use of them, but frontier companies will maintain near total dominance because they will be the only ones with access to the hardware.

honeycrispy 3mo ago
There will be plenty of model providers with prices that undercut Anthropic/OpenAI's prices.
- root_axis 3mo ago
  Doubtful for the same reasons. The frontier providers are working at a hardware scale that will make it impractical to undercut them. I wouldn't rule out the possibility completely, but it won't be very common.
  - honeycrispy 3mo ago
    They also have to pay for staff / model training. Those are not cheap.
repeekad 3mo ago
Meta is selling their now excess compute, other compute has been on the market for a while. The current hardware cost bubble is temporary, especially once people are forced to pay the real inference price instead of majorly subsidized subscriptions.
- bigyabai 3mo ago
  Lowering the cost of hardware still won't solve the issue. HBM and DDR5 was never cheap, even before the shortages, so selling a full inference system is beyond the acceptable price range for most casual customers. We're going to see Apple and Google compete over services and AI/OS integration instead, it will probably be years before your OEM takes local models seriously.
  - tanseydavid 3mo ago
    Apple and Google (via smartphones) are in literally everyone's pocket. Running KIMI on a phone is not possible today and I agree with you that it will "probably be years before..." it is. But how many years do you guess? I personally do not think it will take even 10 years for the situation to be commonplace.
    - bigyabai 3mo ago
      > I personally do not think it will take even 10 years for the situation to be commonplace. Do you personally remember how far smartphones progressed in the past 10 years? It's not as long a time as you think it is, the limits of what a smartphone GPU is capable of did not substantially change in that time. Nor did the amount of onboard RAM that we include in the package. This is true even for Nvidia's ARM SOCs, frankly. Apple, Microsoft and Google all eventually want to enforce OS-level lock-in for the most profitable AI services (eg. their own). It's much more attainable and profitable to use that lock-in to sell you exclusive service integration, the local AI revolution probably won't begin on their hardware.
      - fragmede 3mo ago
        Only if you narrowly define AI as talking with a chatbot. Today, on an iPhone, there are a number of features that only work due to some sort of a model running locally. OCR and intelligent object selection in a photos, and summarization of texts are local models running on the iPhone hardware, they're just not a chatbot. Yes, there's also cloud backing various features, but the idea of running models locally isn't foreign to Apple. On Google's side, the Pixel 10 Pro is powerful enough to run quantized local chatbot models locally today. Local translate is a model, and has been for a while. Both corporations are going to sell whatever customers are willing to pay for, in money or via ads, and if local models get good enough that customers actually are willing to pay for it, I have no doubt that it's Apple and Google will go that direction. It's Google that's releasing Gemma models for download, and they're a big enough organization that the left hand doesn't know what the right hand is doing, so the one conspiracy theory that they'll never do local models because they only want to profit off hosted models is too simplistic.
- program_whiz 3mo ago
  yep, all those coders paying $200-$500 per day to use claude once subsidy ends will be seriously rethinking how much they really want to vibe-code "rewriting X in rust". Helping people write word docs, recipes, and emails isn't going to justify $15K per month subscriptions either.
  - apitman 3mo ago
    > Helping people write word docs, recipes, and emails isn't going to justify $15K per month subscriptions either. Those things can all be done today on a $250 used video card and pennies of electricity
    - root_axis 3mo ago
      Not really. Models that can run on a $250 card may be able to produce a decent word doc sometimes, but the fidelity of such models is so poor that it's no longer very attractive for general use on those types of tasks.
      - andrekandre 3mo ago
        i might have agreed with you 4 months ago, but today with open source models i am surprised as anyone with the progress compared to frontier... so much so that i also picked up one of those $250 cards!
      - gf000 3mo ago
        Then I buy a very expensive GPU rack and timeshare it with a couple of friends.
      - apitman 3mo ago
        Have you tried qwen3.6-27b or Gemma 4? gemma4-26b-a4b is particularly impressive for its balance of speed and performance
  - km3r 3mo ago
    The subscription is certainly subsidized but its no where near 100x cheaper than API prices. Heck, most large enterprise moved to usage based billing and are still happily paying for it. They are force multipliers for your top talent, and when a top engineer is being paid $500k a year, doubling their output for $500/day is a no brainer.
switchbak 3mo ago
For the near future it seems that the new models will consume whatever improved hardware capacity we have. Competing with that is challenging, but I also think there will be strong economic incentives towards cheaper but adequate models on other providers. I don't think we'll see home users being able to match even the low end clouds for a long time. Longer term I think we'll see these uses of AI cluster into a few groups: - maximal code / reasoning quality, at high prices (Fable) - typical code / agents (sub-Opus, Terra) - cheap but decent enough quality (think Deepseek / GLM / Luna) - so cheap I don't care about utilization (Deepseek, and friends) And also more niche ones: - ultra fast with high quality answers (typically sub-SOTA). Cerebras / dedicated silicon type approaches, expensive. - ultra fast with mostly-adequate answers, and an openness to retries, moving up to better models I think the open models will dominate (not with individuals, but low cost providers) all except the top 1-2 of those categories, and there will be a continuous erosion on the big player's moats. The top categories are also where all the money is, but I'm not sure it can justify those investments long-term. I also think they will have to squeeze more money out of them to justify the investments, which will also drive people down the list. Edit: clarifications.
- asadotzler 3mo ago
  >For the near future it seems that the new models will consume whatever improved hardware capacity we have. But they're not. Meta, SpaceX, Microsoft, Amazon, they're all leasing out capacity to others. If they were truly constrained, we wouldn't see that happening.
mark_l_watson 3mo ago
You may be correct but the combination of local models when they are fast enough and work, combined with paying close to zero for deepseek v4 flash from US providers is pretty good. When you need it, glm5.2 is cheap to use and very good for working with larger projects.

honeycrispy 3mo ago

There will be plenty of model providers with prices that undercut Anthropic/OpenAI's prices.

root_axis 3mo ago
Doubtful for the same reasons. The frontier providers are working at a hardware scale that will make it impractical to undercut them. I wouldn't rule out the possibility completely, but it won't be very common.
- honeycrispy 3mo ago
  They also have to pay for staff / model training. Those are not cheap.

repeekad 3mo ago

Meta is selling their now excess compute, other compute has been on the market for a while. The current hardware cost bubble is temporary, especially once people are forced to pay the real inference price instead of majorly subsidized subscriptions.

bigyabai 3mo ago
Lowering the cost of hardware still won't solve the issue. HBM and DDR5 was never cheap, even before the shortages, so selling a full inference system is beyond the acceptable price range for most casual customers. We're going to see Apple and Google compete over services and AI/OS integration instead, it will probably be years before your OEM takes local models seriously.
- tanseydavid 3mo ago
  Apple and Google (via smartphones) are in literally everyone's pocket. Running KIMI on a phone is not possible today and I agree with you that it will "probably be years before..." it is. But how many years do you guess? I personally do not think it will take even 10 years for the situation to be commonplace.
  - bigyabai 3mo ago
    > I personally do not think it will take even 10 years for the situation to be commonplace. Do you personally remember how far smartphones progressed in the past 10 years? It's not as long a time as you think it is, the limits of what a smartphone GPU is capable of did not substantially change in that time. Nor did the amount of onboard RAM that we include in the package. This is true even for Nvidia's ARM SOCs, frankly. Apple, Microsoft and Google all eventually want to enforce OS-level lock-in for the most profitable AI services (eg. their own). It's much more attainable and profitable to use that lock-in to sell you exclusive service integration, the local AI revolution probably won't begin on their hardware.
    - fragmede 3mo ago
      Only if you narrowly define AI as talking with a chatbot. Today, on an iPhone, there are a number of features that only work due to some sort of a model running locally. OCR and intelligent object selection in a photos, and summarization of texts are local models running on the iPhone hardware, they're just not a chatbot. Yes, there's also cloud backing various features, but the idea of running models locally isn't foreign to Apple. On Google's side, the Pixel 10 Pro is powerful enough to run quantized local chatbot models locally today. Local translate is a model, and has been for a while. Both corporations are going to sell whatever customers are willing to pay for, in money or via ads, and if local models get good enough that customers actually are willing to pay for it, I have no doubt that it's Apple and Google will go that direction. It's Google that's releasing Gemma models for download, and they're a big enough organization that the left hand doesn't know what the right hand is doing, so the one conspiracy theory that they'll never do local models because they only want to profit off hosted models is too simplistic.
program_whiz 3mo ago
yep, all those coders paying $200-$500 per day to use claude once subsidy ends will be seriously rethinking how much they really want to vibe-code "rewriting X in rust". Helping people write word docs, recipes, and emails isn't going to justify $15K per month subscriptions either.
- apitman 3mo ago
  > Helping people write word docs, recipes, and emails isn't going to justify $15K per month subscriptions either. Those things can all be done today on a $250 used video card and pennies of electricity
  - root_axis 3mo ago
    Not really. Models that can run on a $250 card may be able to produce a decent word doc sometimes, but the fidelity of such models is so poor that it's no longer very attractive for general use on those types of tasks.
    - andrekandre 3mo ago
      i might have agreed with you 4 months ago, but today with open source models i am surprised as anyone with the progress compared to frontier... so much so that i also picked up one of those $250 cards!
    - gf000 3mo ago
      Then I buy a very expensive GPU rack and timeshare it with a couple of friends.
    - apitman 3mo ago
      Have you tried qwen3.6-27b or Gemma 4? gemma4-26b-a4b is particularly impressive for its balance of speed and performance
- km3r 3mo ago
  The subscription is certainly subsidized but its no where near 100x cheaper than API prices. Heck, most large enterprise moved to usage based billing and are still happily paying for it. They are force multipliers for your top talent, and when a top engineer is being paid $500k a year, doubling their output for $500/day is a no brainer.

switchbak 3mo ago

For the near future it seems that the new models will consume whatever improved hardware capacity we have. Competing with that is challenging, but I also think there will be strong economic incentives towards cheaper but adequate models on other providers. I don't think we'll see home users being able to match even the low end clouds for a long time. Longer term I think we'll see these uses of AI cluster into a few groups: - maximal code / reasoning quality, at high prices (Fable) - typical code / agents (sub-Opus, Terra) - cheap but decent enough quality (think Deepseek / GLM / Luna) - so cheap I don't care about utilization (Deepseek, and friends) And also more niche ones: - ultra fast with high quality answers (typically sub-SOTA). Cerebras / dedicated silicon type approaches, expensive. - ultra fast with mostly-adequate answers, and an openness to retries, moving up to better models I think the open models will dominate (not with individuals, but low cost providers) all except the top 1-2 of those categories, and there will be a continuous erosion on the big player's moats. The top categories are also where all the money is, but I'm not sure it can justify those investments long-term. I also think they will have to squeeze more money out of them to justify the investments, which will also drive people down the list. Edit: clarifications.

asadotzler 3mo ago
>For the near future it seems that the new models will consume whatever improved hardware capacity we have. But they're not. Meta, SpaceX, Microsoft, Amazon, they're all leasing out capacity to others. If they were truly constrained, we wouldn't see that happening.

mark_l_watson 3mo ago

You may be correct but the combination of local models when they are fast enough and work, combined with paying close to zero for deepseek v4 flash from US providers is pretty good. When you need it, glm5.2 is cheap to use and very good for working with larger projects.