4 ms·
With 1M tokens you can dump 2000 pages of documents into the context windows before starting a chat. Gemini's strength isn't in being able to answer logic puzz
by Workaccount2 2y ago
With 1M tokens you can dump 2000 pages of documents into the context windows before starting a chat.
Gemini's strength isn't in being able to answer logic puzzles, it's strength is in its context length. Studying for an exam? Just put the entire textbook in the chat. Need to use a dead language for an old test system with no information on the internet? Drop the 1300 page reference manual in and ask away.
- tulip4attoo 2y ago[flagged]
- dang 2y agoPlease make your substantive points without crossing into personal attack. Your comment would be fine without the first sentence. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- ianbicking 2y agoHow much do those input tokens cost? According to https://ai.google.dev/pricing https://ai.google.dev/pricing it's $0.70/million input tokens (for a long context). That will be per-exchange, so every little back and forth will cost around that much (if you're using a substantial portion of the context window). And while I haven't tested Gemini, most LLMs get increasingly wonky as the context goes up, more likely to fixate, more likely to forget instructions. That big context window could definitely be great for certain tasks (especially information extraction), but it doesn't feel like a generally useful feature.
- lxgr 2y agoIs there a way to amortize that cost over several queries, i.e. "pre-bake" a document into a context persisted in some form to allow cheaper follow-up queries about it?
- inlined 2y agoThough I'm not familiar with the specifics, they announced "context caching"
- simonw 2y agoThey announced that today, calling it "context caching" - but it looks like it's only going to be available for Gemini Pro 1.5, not for Gemini Flash. It reduces prompt costs by half for those shared prefix tokens, but you have to pay $4.50/million tokens/hour to keep that cache warm - so probably not a useful optimization for most lower traffic applications. https://ai.google.dev/gemini-api/docs/caching https://ai.google.dev/gemini-api/docs/caching
- dragonwriter 2y ago> It reduces prompt costs by half for those shared prefix tokens, but you have to pay $4.50/million tokens/hour to keep that cache warm - so probably not a useful optimization for most lower traffic applications That's on a model with $3.5/1M input token cost, so half price on cached prefix tokens for $4.5/1M/hour breaks even at a little over 2.5 requests/hour using the cached prefix.
- gcanyon 2y agoDepending on the output window limit, the first query could be something like: "Summarize this down to its essential details" -- then use that to feed future queries. Tediously, it would be possible to do this chapter by chapter in order to exceed the output limit building something for future inputs. Of course, the summary might not fulfill the same functionality as the original source document. YMMV
- mcbuilder 2y agoThat per exchange context cost is what really puts me off using cloud LLM for anything serious. I know batching and everything is needed in the data center, and important for keeping around KVQ cache, you basically need to fully take over machine to get an interactive session to get the context costs to scale with sequence length. So it's useful, but more in the case of a local LLaMA type situation if you want a conversation.
- falcor84 2y agoI wonder if we could implement the equivalent of a JIT compilation, whereby context sequences which get repeatedly reused would be used for an online fine-tuning.
- HumanOstrich 2y ago[flagged]
- nostrebored 2y agoThey are asking if you can take the context being passed per interaction and train it into a session in real time (via an online algorithm). Essentially bake the context passed in to the attention layer so that you can pass only the relevant chat context. Your post wasn’t a particularly charitable interpretation.
- sp332 2y agoNo, but you can just cache the state after processing the prompt. https://github.com/ggerganov/llama.cpp/tree/master/examples/main#prompt-caching https://github.com/ggerganov/llama.cpp/tree/master/examples/...
- neverokay 2y agoIt makes building any app that requires generous user prompting impossible to build for regular developers (cloud pricing). $20 hosting can serve thousands of users per month. $20 llm sub services just one person. This is fucking impossible.
- bredren 2y agoCan anyone speculate on how G arrived at this price, and perhaps how it contrasts with how OAI arrived at its updated pricing? (realizing it can't be held up directly to GPT x at the moment)
- tk90 2y agoIsn't there retrieval degradation with such a large context size? I would still think that a RAG system on 128K is still better than No Rag + 1M context window, no? (assuming text only)