4 ms·
The cost of inference seems like a major barrier in making these things work commercially. If you put one in the user facing flow you'll need an extraordinary a
by jcgrillo 2y ago
The cost of inference seems like a major barrier in making these things work commercially. If you put one in the user facing flow you'll need an extraordinary amount of compute that scales extremely poorly in number of users, given that each query takes O(nm^2) where m is the number of model weights (many) and n is the number of tokens in the query. So it seems clear if each user session implies multiple LLM queries, those user sessions had better be exceptionally valuable. This seems like a completely different business with very different constraints and margins.
The problem is it doesn't seem like the value added is all that great. So how do you justify the ruinous cost? I would like to know more about how people actually intend on using these things profitably than to hear about how they're going to wake up and become "intelligent". Does anyone have any success stories actually using this thing? It's hard to find any discussion of this amidst all the bullshit hype, and it's really the only question--can you make money with this or not? And I don't mean raising billions in venture capital, I mean can you integrate these things profitably and sustainably into a website?
- tmnvdb 2y agoThe costs are a problem. We don't have hard evidence that this will be solved, but with algorithmic efficiency and raw compute costs both changing rapidly, the cost per token has gone down by about a factor of 10 per year for the last 3 years, i.e. 1000x over 3 years.
- jcgrillo 2y agoAs far as I can tell, those charts are merely describing the price per token that LLM hosting companies are charging, not what running the model actually costs. The distinction is important for two reasons: 1. These companies are heavily subsidized by huge amounts of venture investment 2. If I'm integrating this technology into my web product there's absolutely no way I'll be adding a 3rd party company as a dependency. This is all way too new and bubbly to trust any of the current offerings will still exist in O(years). Are there any similar studies showing not sticker price but actual compute/performance decrease?
- tmnvdb 2y agoThis is incorrect, there are real effiency gains. Slightly old by the standards of this field, but a good overview: https://arxiv.org/abs/2403.05812 https://arxiv.org/abs/2403.05812
- jcgrillo 2y agoI'm sorry, what is incorrect? And the paper you linked appears to be about training, not inference? I wouldn't train a model in a user's session, instead I'd run the model. That's the cost that seems a blocker, all the models are already trained, I dont need to invest a dime in that.