3 ms·

> demand for tokens/AI inference capacity exceeds CURRENT supply That's not at all obvious to me; costs as a consumer are going down, rather than up. Can someo

> demand for tokens/AI inference capacity exceeds CURRENT supply That's not at all obvious to me; costs as a consumer are going down, rather than up. Can someone steel-man this guy's argument for me?

matusp 1y ago
There is only so many requests people want to process with LLMs.
- Earw0rm 1y ago
  Even though I'm a touch sceptical as to how good LLMs are at coding, imagine developing world demand for LLMs via a cheap Android handset for $1/month. There's a lot of use case for the information retrieval side, especially for people whose educational level and mindset means well-crafted search queries were never a thing for them. But there's also the other side of the trade. How much are suppliers willing to pay for upranking? It's a dirty business model, Enshittification 3.0+, but when did that stop anyone?
antonvs 1y ago
Cost to consumers is being subsidized by investors trying to grab market share - penetration pricing.
- petesergeant 1y ago
  I don’t think that’s true, given the wide range of inference providers on OpenRouter. These are mostly undifferentiated and commoditized, with no chance of gaining market share at the exclusion of others.
  - antonvs 1y ago
    Same issue, you're not seeing what it actually costs, yet. All of those are based on foundation models that have very high training costs - tens to hundreds of millions of dollars. Even smaller models like Mistral apparently cost several million just for compute. The user-facing economics and inference costs are only part of the story.
wmf 1y ago
"Demand" for inference seems infinite because you can always think harder or insert AI for free into some workflow where the customer didn't request it (e.g. Google search). It's like a gas that expands to fill the container. Then during the crash people are going to discover that some fraction of that demand wasn't needed and they'll just turn it off. It would be a good sign if paid demand exceeds supply but I don't know if we can measure that.

> demand for tokens/AI inference capacity exceeds CURRENT supply That's not at all obvious to me; costs as a consumer are going down, rather than up. Can someone steel-man this guy's argument for me?

matusp 1y ago

There is only so many requests people want to process with LLMs.

Earw0rm 1y ago
Even though I'm a touch sceptical as to how good LLMs are at coding, imagine developing world demand for LLMs via a cheap Android handset for $1/month. There's a lot of use case for the information retrieval side, especially for people whose educational level and mindset means well-crafted search queries were never a thing for them. But there's also the other side of the trade. How much are suppliers willing to pay for upranking? It's a dirty business model, Enshittification 3.0+, but when did that stop anyone?

antonvs 1y ago

Cost to consumers is being subsidized by investors trying to grab market share - penetration pricing.

petesergeant 1y ago
I don’t think that’s true, given the wide range of inference providers on OpenRouter. These are mostly undifferentiated and commoditized, with no chance of gaining market share at the exclusion of others.
- antonvs 1y ago
  Same issue, you're not seeing what it actually costs, yet. All of those are based on foundation models that have very high training costs - tens to hundreds of millions of dollars. Even smaller models like Mistral apparently cost several million just for compute. The user-facing economics and inference costs are only part of the story.

wmf 1y ago

"Demand" for inference seems infinite because you can always think harder or insert AI for free into some workflow where the customer didn't request it (e.g. Google search). It's like a gas that expands to fill the container. Then during the crash people are going to discover that some fraction of that demand wasn't needed and they'll just turn it off. It would be a good sign if paid demand exceeds supply but I don't know if we can measure that.