3 ms·
How much do those input tokens cost? According to https://ai.google.dev/pricing https://ai.google.dev/pricing it's $0.70/million input tokens (for a long conte
by ianbicking 2y ago
How much do those input tokens cost?
According to https://ai.google.dev/pricing https://ai.google.dev/pricing it's $0.70/million input tokens (for a long context). That will be per-exchange, so every little back and forth will cost around that much (if you're using a substantial portion of the context window).
And while I haven't tested Gemini, most LLMs get increasingly wonky as the context goes up, more likely to fixate, more likely to forget instructions.
That big context window could definitely be great for certain tasks (especially information extraction), but it doesn't feel like a generally useful feature.
- lxgr 2y agoIs there a way to amortize that cost over several queries, i.e. "pre-bake" a document into a context persisted in some form to allow cheaper follow-up queries about it?
- inlined 2y agoThough I'm not familiar with the specifics, they announced "context caching"
- simonw 2y agoThey announced that today, calling it "context caching" - but it looks like it's only going to be available for Gemini Pro 1.5, not for Gemini Flash. It reduces prompt costs by half for those shared prefix tokens, but you have to pay $4.50/million tokens/hour to keep that cache warm - so probably not a useful optimization for most lower traffic applications. https://ai.google.dev/gemini-api/docs/caching https://ai.google.dev/gemini-api/docs/caching
- dragonwriter 2y ago> It reduces prompt costs by half for those shared prefix tokens, but you have to pay $4.50/million tokens/hour to keep that cache warm - so probably not a useful optimization for most lower traffic applications That's on a model with $3.5/1M input token cost, so half price on cached prefix tokens for $4.5/1M/hour breaks even at a little over 2.5 requests/hour using the cached prefix.
- gcanyon 2y agoDepending on the output window limit, the first query could be something like: "Summarize this down to its essential details" -- then use that to feed future queries. Tediously, it would be possible to do this chapter by chapter in order to exceed the output limit building something for future inputs. Of course, the summary might not fulfill the same functionality as the original source document. YMMV
- mcbuilder 2y agoThat per exchange context cost is what really puts me off using cloud LLM for anything serious. I know batching and everything is needed in the data center, and important for keeping around KVQ cache, you basically need to fully take over machine to get an interactive session to get the context costs to scale with sequence length. So it's useful, but more in the case of a local LLaMA type situation if you want a conversation.
- falcor84 2y agoI wonder if we could implement the equivalent of a JIT compilation, whereby context sequences which get repeatedly reused would be used for an online fine-tuning.
- HumanOstrich 2y ago[flagged]
- nostrebored 2y agoThey are asking if you can take the context being passed per interaction and train it into a session in real time (via an online algorithm). Essentially bake the context passed in to the attention layer so that you can pass only the relevant chat context. Your post wasn’t a particularly charitable interpretation.
- sp332 2y agoNo, but you can just cache the state after processing the prompt. https://github.com/ggerganov/llama.cpp/tree/master/examples/main#prompt-caching https://github.com/ggerganov/llama.cpp/tree/master/examples/...
- neverokay 2y agoIt makes building any app that requires generous user prompting impossible to build for regular developers (cloud pricing). $20 hosting can serve thousands of users per month. $20 llm sub services just one person. This is fucking impossible.
- bredren 2y agoCan anyone speculate on how G arrived at this price, and perhaps how it contrasts with how OAI arrived at its updated pricing? (realizing it can't be held up directly to GPT x at the moment)