4 ms·
This is really cool! I'd like to understand how you're dealing with the costs. Since you're using an LLM and this is getting viral on HN, aren't API costs getti
by fievelk 9mo ago
This is really cool! I'd like to understand how you're dealing with the costs. Since you're using an LLM and this is getting viral on HN, aren't API costs getting higher? I'm working on some LLM project myself and trying to figure out how to make the demo accessible without spending too much.
Great job.
- alecf 9mo agoAt least for the sake of this demo, the costs are actually pretty small, especially given that many of the ... even at $1.75/1M tokens you'd have to have 2 megabytes of text across all users before this even broke $1...
- storystarling 9mo agoThe unit price looks low but the cost driver in these apps is usually the context window. You have to re-send the full history every turn to maintain state, so you end up paying for the same tokens dozens of times. By the end of a long session the cost per interaction is significantly higher than at the start since you're re-processing the entire game log for every new command.
- selcuka 9mo agoCached tokens are cheaper. Also there are ways to compress the context (not sure if this game employs any of those techniques).