4 ms·
I've been disappointed in some of these services before, because in the early days and I suspect now still as well they used RAG for obvious reasons. I feel lik
by ZeroCool2u 2y ago
I've been disappointed in some of these services before, because in the early days and I suspect now still as well they used RAG for obvious reasons. I feel like for research it's rare that RAG works really well.
What I'm really excited about is a tool like Elicit using the new Google Gemini 1.5 Pro/Ultra models with the 2 Million token window sizes, filtering down the papers using traditional search and high quality meta-data, then critically, prompt/activation caching to make the tool economically viable.
Maybe it won't work better, but I'm willing to bet it'll find those really specific ideas/needles in the haystack a lot more often than vanilla RAG will.
- inimino 2y agoIt's the economically viable part that I think is currently hard, but agreed this is the right approach. The basic problem is that scaling up understanding over a large dataset requires scaling the application of an LLM and tokens are expensive.
- lysecret 2y agoAnd the minutes of latency you currently get with this context lengths.
- ZeroCool2u 2y agoYeah, this is why I mentioned Gemini with the context caching. It's not out yet, but supposedly launching soon. You pay a lower rate for storing the system prompt or whatever you dump in before the user query, plus you don't have to wait the full minute or so for all your research to be ingested every time. https://ai.google.dev/gemini-api/docs/caching#get-started https://ai.google.dev/gemini-api/docs/caching#get-started