RAG. Large context windows can cause the LLMs to get confused or use part of the window. I think googles new 1M context window is promising for its recall but

RAG. Large context windows can cause the LLMs to get confused or use part of the window. I think googles new 1M context window is promising for its recall but needs more testing to be certain. Other llms have shown they reduce performance on larger contexts. Additionally, the LLM might discard instructions and hallucinate. Also, if you are concatenating user input, you are opening yourself up to prompt injection. The larger the context, the larger the prompt injection can be. There is also the cost. Why load up 1M tokens and pay for a massive request when you can load a fraction that is actually relevant? So regardless if google’s 1M context is perfect and others can match it, I would steer away from just throwing it into the massive context. To me, it’s easier to mentally think of this as SQL query performance. Is a full table scan better or worse than using an index? Even if you can do a full table scan in a reasonable amount of time, why bother? Just do things the right way…