4 ms·
> The 10M context ability wipes out most RAG stack complexity immediately. Remains to be seen. Large contexts are not always better. For starters, it takes l
by CharlieDigital 3y ago
> The 10M context ability wipes out most RAG stack complexity immediately.
Remains to be seen.
Large contexts are not always better. For starters, it takes longer to process. But secondly, even with RAG and the large context of GPT4 Turbo, providing it a more relevant and accurate context always yields better output.
What you get with RAG is faster response times and more accurate answers by pre-filtering out the noise.
- behnamoh 3y agoDon't forget that Gemini also has access to the internet, so a lot of RAGging becomes pointless anyway.
- beppo 3y agoInternet search is a form of RAG, though. 10M tokens is very impressive, but you're not fitting a database, let alone the entire internet into a prompt anytime soon.
- behnamoh 3y agoYou shouldn't fit an entire database in the context anyway. btw, 10M tokens is 78 times more context window than the newest GPT-4-turbo (128K). In a way, you don't need 78 GPT-4 API calls, only one batch call to Gemini 1.5.
- rvnx 3y agoWell it's nice, just sad nobody can use it
- cchance 3y agoI don't get this why is it people think that you need to put an entire database in the short-term memory of the AI to be useful? When you work with a DB are you memorizing the entire f*cking database, no, you know the summaries of it and how to access and use it. People also seem to forget that the average is 1b words that are read by people in their entire LIFETIME, and at 10m, with nearly 100% recall thats pretty damn amazing, i'm pretty sure I don't have perfect recall of 10m words myself lol
- Qwero 3y agoIt increases the use cases. It can also be a good alternative for fine-tuning. And the use case of a code base is a good example: if the ai understands the whole context, it can do basically everything. Let me pay 5€ for a android app rewritten into iOS.
- choilive 3y agoYou certainly don't need that much context for it to be useful, but it definitely opens up a LOT more possibilities without the compromises of implementing some type of RAG. In addition, don't we want our AI to have superhuman capabilities? The ability to work on 10M+ tokens of context at a time could enable superhuman performance in many tasks. Why stop at 10M tokens? Imagine if AI could work on 1B tokens of context like you said?
- CharlieDigital 3y agoThis may be useful in a generalized use case, but a problem is that many of those results again will add noise. For any use case where you want contextual results, you need to be able to either filter the search scope or use RAG to pre-define the acceptable corpus.
- panarky 3y ago> you need to be able to either filter the search scope or use RAG ... Unless you can get nearly perfect recall with millions of tokens, which is the claim made here.
- killerstorm 3y agoHopefully we can get a better RAG out of it. Currently people do incredibly primitive stuff like chunking text into chunks of a fixed size and adding them to vector DB. An actually useful RAG would be to convert text to Q&A and use Q's embeddings as an index. Large context can make use of in-context learning to make better Q&A.
- mediaman 3y agoA lot of people in RAG already do this. I do this with my product: we process each page and create lists of potential questions that the page would answer, and then embed that. We also embed the actual text, though, because I found that only doing the questions resulted in inferior performance.
- CharlieDigital 3y agoSo in this case, what your workflow might look like is: 1. Get text from page/section/chunk 2. Generate possible questions related to the page/section/chunk 3. Generate an embedding using { each possible question + page/section/chunk } 4. Incoming question targets the embedding and matches against { question + source } Is this roughly it? How many questions do you generate? Do you save a separate embedding for each question? Or just stuff all of the questions back with the page/section/chunk?
- mediaman 3y agoRight now I just throw the different questions together in a single embedding for a given chunk, with the idea that there’s enough dimensionality to capture them all. But I haven’t tested embedding each question, matching on that vector, and then returning the corresponding chunk. That seems like it’d be worth testing out.