3 ms·
So, will this outperform any RAG approach as long as the data fits inside the context window?
by megaman821 3y ago
So, will this outperform any RAG approach as long as the data fits inside the context window?
- saliagato 3y agobasically, yes. Pinecone? Dead. Azure AI Search? Dead. Quadrant? Dead.
- _boffin_ 3y agoPrompt token cost still a variable.
- ArcaneMoose 3y agoCost would still be a big concern
- TheGeminon 3y agoOutperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation. But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.
- megaman821 3y agoDoesn't their needle/haystack benchmark seem to suggest there is almost no dilution? They pushed that demo out to 10M tokens.
- CuriouslyC 3y agoA perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.
- chasd00 3y agoare you going to upload 10M tokens to Gemini on every request? That's a lot of data moving around when the user is expecting a near realtime response. Seems like it would still be better to only set the context with information relevant to the user's prompt which is what plain rag does.