4 ms·

There's a fair amount of skepticism towards the efficacy of RAG in these comments—often in contrast to simply using a model with a huge context window to analyz

There's a fair amount of skepticism towards the efficacy of RAG in these comments—often in contrast to simply using a model with a huge context window to analyze the corpus in one giant chunk. But that will not be a viable alternative in all use cases. For example, one might need to analyze/search a very large corpus composed of many documents which, as a whole, is very unlikely to fit within any realistic context window. Or one might be constrained to only use local models and may not have access to models with these huge windows. Or both! In cases like these, can anyone recommend a more promising approach than RAG?

make3 3y ago
rag is also so much faster and cheaper and energy efficient than running a giant transformer over every single document, it's like not remotely close
Atotalnoob 3y ago
RAG. Large context windows can cause the LLMs to get confused or use part of the window. I think googles new 1M context window is promising for its recall but needs more testing to be certain. Other llms have shown they reduce performance on larger contexts. Additionally, the LLM might discard instructions and hallucinate. Also, if you are concatenating user input, you are opening yourself up to prompt injection. The larger the context, the larger the prompt injection can be. There is also the cost. Why load up 1M tokens and pay for a massive request when you can load a fraction that is actually relevant? So regardless if google’s 1M context is perfect and others can match it, I would steer away from just throwing it into the massive context. To me, it’s easier to mentally think of this as SQL query performance. Is a full table scan better or worse than using an index? Even if you can do a full table scan in a reasonable amount of time, why bother? Just do things the right way…

make3 3y ago

rag is also so much faster and cheaper and energy efficient than running a giant transformer over every single document, it's like not remotely close

Atotalnoob 3y ago

RAG. Large context windows can cause the LLMs to get confused or use part of the window. I think googles new 1M context window is promising for its recall but needs more testing to be certain. Other llms have shown they reduce performance on larger contexts. Additionally, the LLM might discard instructions and hallucinate. Also, if you are concatenating user input, you are opening yourself up to prompt injection. The larger the context, the larger the prompt injection can be. There is also the cost. Why load up 1M tokens and pay for a massive request when you can load a fraction that is actually relevant? So regardless if google’s 1M context is perfect and others can match it, I would steer away from just throwing it into the massive context. To me, it’s easier to mentally think of this as SQL query performance. Is a full table scan better or worse than using an index? Even if you can do a full table scan in a reasonable amount of time, why bother? Just do things the right way…