3 ms·
Character.ai shows how to reduce LLM costs by 13 times
- gnabgib 2y agoArticle title: Optimizing AI Inference Discussion (63 points, 14 days ago) https://news.ycombinator.com/item?id=40739225 https://news.ycombinator.com/item?id=40739225
- uberman 2y agoIf I reduce the cost of something by 1x it becomes free...
- Oksitaine 2y agoYes it's true, i mean he LLM costs for Character.ai are 13 times cheaper (thanks to their optimization with KV cache) than if they had simply used the API of an LLM service like Anthropic or ChatGPT.
- uberman 2y agoI think it is a great optimization. My point is that they can reduce the cost by 92% which is less than 1x (a fantastic optimization) but if they reduce the cost by 1x (100%) it would mean that the service would be free. Reducing the cost by 13x means that for every dollar I spend they would give me 13 dollars. This is a not uncommon error and i would not mention it, but it is the direct claim in the heading.