3 ms·

> I don't think cloud models are going away; the hardware for good perf is expensive I think local AI will win in its niche by repurposing users' existing hard

> I don't think cloud models are going away; the hardware for good perf is expensive I think local AI will win in its niche by repurposing users' existing hardware, especially as cloud hardware itself gets increasingly bottlenecked in all sorts of ways and the price of cloud tokens rises. You don't have to care about "bad" performance when you've got dedicated hardware that runs your workloads 24/7. Time-critical work that also requires the latest and greatest model can stay on the cloud, but a vast amount of AI work just isn't that critical.

ai_fry_ur_brain 6mo ago
[flagged]
- CamperBob2 6mo ago
  Other sites beckon.
reissbaker 6mo ago
Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.
- otabdeveloper4 6mo ago
  > paying $100/month There will not ever be a monthly subscription for LLM tokens. The economics isn't there. Local tokens will always be cheaper.
  - entrope 6mo ago
    What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours of service per day.
    - zozbot234 6mo ago
      Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.
      - entrope 6mo ago
        Both my experience, and Anthropic's off-peak promotion, indicate that there are very uneven levels of demand for peak hours versus off-peak hours. How close do you think they are?
        zozbot234 6mo ago
        But that's demand for cloud inference that's priced on a flat-rate basis with some adjustments (like "off-peak hours"). Not a local rig where inference is effectively free aside from the cost of power whenever the system isn't congested.
        reissbaker 6mo ago
        The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig. Therefore cloud tokens have a huge capital efficiency advantage over local rigs. That will continue basically forever, since there will always be large cloud companies willing to spend millions of dollars for more capital-efficient hardware, so Nvidia and friends will continue to spare no expense producing it, meaning the cloud hardware will be way too expensive if you're not a large inference company. You can also buy local rigs, but they will be less capital efficient per token, not more. (This is a generous argument: it also ignores the massive software stack optimization the cloud companies do that doesn't trickle down to local-rig-sized deployments; for example, prefill/decode disaggregation, which would double the VRAM requirements for a local rig — if you could even do it on a local rig, which you can't, because local rigs don't have Infiniband. But at scale, prefill/decode disaggregation improves capital efficiency, since you can tune the compute-bound prefill node differently than the memory-bound decode node.) The advantage of local rigs is not capital-efficient tokens. It's privacy. But then again, you can get zero-data-retention options from many inference companies, so for many use cases it may not matter unless you need strict guarantees the data never leaves the building...
  - reissbaker 6mo ago
    There already are many subscriptions for LLM tokens: OpenAI, Claude, Synthetic (shameless plug), Zai... I'm not sure what you mean by "There will not ever be a monthly subscription for LLM tokens." That already exists!
    - otabdeveloper4 6mo ago
      Monthly subscriptions is a "first hit is free" promo. In the future LLMs will be priced per token, not all-you-can-eat.
      - reissbaker 6mo ago
        LLM subscriptions are not "all you can eat," they have rate limits — and fundamentally there is no difference between subscription-with-rate-limit and typical usage-based business practices. Subscriptions are simply usage-based pricing with volume discounts in exchange for upfront payment; every single usage-based provider of pretty much anything offers the same kind of discounting for buying volume commitments upfront. Although from a business sense, subscriptions are even better than volume discounts... Because they're recurring, whereas reserved volume might not recur. Subscriptions aren't gonna go away. They're great for businesses. Rate limits or pricing might change but the underlying business model is very good. The reason usage-based is so much more expensive than subscription isn't that usage-based is the "true" cost and subscription is a loss leader — just like a buying 30 consecutive day passes to a gym being more expensive than a monthly membership isn't a result of memberships being a loss leader. Memberships are the business model! The day passes are overpriced to steer you into buying the membership.
- zozbot234 6mo ago
  You said you need $80k in hardware for "good performance". I'm saying the local AI inference workflow will be a lot more flexible about performance than that, and can get away with something vastly cheaper and in line with what the user owns already.

ai_fry_ur_brain 6mo ago

[flagged]

CamperBob2 6mo ago
Other sites beckon.

reissbaker 6mo ago

Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.

otabdeveloper4 6mo ago
> paying $100/month There will not ever be a monthly subscription for LLM tokens. The economics isn't there. Local tokens will always be cheaper.
- entrope 6mo ago
  What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours of service per day.
  - zozbot234 6mo ago
    Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.
    - entrope 6mo ago
      Both my experience, and Anthropic's off-peak promotion, indicate that there are very uneven levels of demand for peak hours versus off-peak hours. How close do you think they are?
      - zozbot234 6mo ago
        But that's demand for cloud inference that's priced on a flat-rate basis with some adjustments (like "off-peak hours"). Not a local rig where inference is effectively free aside from the cost of power whenever the system isn't congested.
        reissbaker 6mo ago
        The local rig is not free and requires very large capital expenditures while producing very low token throughput for large models. Within any time budget, you can get many orders of magnitude more large-model tokens off an 8xB200 than off a local rig. Therefore cloud tokens have a huge capital efficiency advantage over local rigs. That will continue basically forever, since there will always be large cloud companies willing to spend millions of dollars for more capital-efficient hardware, so Nvidia and friends will continue to spare no expense producing it, meaning the cloud hardware will be way too expensive if you're not a large inference company. You can also buy local rigs, but they will be less capital efficient per token, not more. (This is a generous argument: it also ignores the massive software stack optimization the cloud companies do that doesn't trickle down to local-rig-sized deployments; for example, prefill/decode disaggregation, which would double the VRAM requirements for a local rig — if you could even do it on a local rig, which you can't, because local rigs don't have Infiniband. But at scale, prefill/decode disaggregation improves capital efficiency, since you can tune the compute-bound prefill node differently than the memory-bound decode node.) The advantage of local rigs is not capital-efficient tokens. It's privacy. But then again, you can get zero-data-retention options from many inference companies, so for many use cases it may not matter unless you need strict guarantees the data never leaves the building...
- reissbaker 6mo ago
  There already are many subscriptions for LLM tokens: OpenAI, Claude, Synthetic (shameless plug), Zai... I'm not sure what you mean by "There will not ever be a monthly subscription for LLM tokens." That already exists!
  - otabdeveloper4 6mo ago
    Monthly subscriptions is a "first hit is free" promo. In the future LLMs will be priced per token, not all-you-can-eat.
    - reissbaker 6mo ago
      LLM subscriptions are not "all you can eat," they have rate limits — and fundamentally there is no difference between subscription-with-rate-limit and typical usage-based business practices. Subscriptions are simply usage-based pricing with volume discounts in exchange for upfront payment; every single usage-based provider of pretty much anything offers the same kind of discounting for buying volume commitments upfront. Although from a business sense, subscriptions are even better than volume discounts... Because they're recurring, whereas reserved volume might not recur. Subscriptions aren't gonna go away. They're great for businesses. Rate limits or pricing might change but the underlying business model is very good. The reason usage-based is so much more expensive than subscription isn't that usage-based is the "true" cost and subscription is a loss leader — just like a buying 30 consecutive day passes to a gym being more expensive than a monthly membership isn't a result of memberships being a loss leader. Memberships are the business model! The day passes are overpriced to steer you into buying the membership.
zozbot234 6mo ago
You said you need $80k in hardware for "good performance". I'm saying the local AI inference workflow will be a lot more flexible about performance than that, and can get away with something vastly cheaper and in line with what the user owns already.