4 ms·
This is interesting, but unless I'm misreading the paper, it looks like they're training an LLM on the corpus. I can easily see why that would result in better
by reissbaker 3y ago
This is interesting, but unless I'm misreading the paper, it looks like they're training an LLM on the corpus. I can easily see why that would result in better performance than an off-the-shelf embeddings model, but... it won't work for a corpus that changes frequently, since you'll constantly have to retrain the LLM. That's sort of the point of RAG: how do you get the right information into an LLM as context, for data that changes so frequently that you can't directly train on it?
- jerpint 3y agoI’m also not sure with this method if tje LLM can exactly cite its source, which is another great benefit of RAG
- vintermann 3y agoThere are some interesting corpora that I would like to search smartly, that shouldn't change too often. For instance "all Norwegian newspapers printed before 1980".
- HellsMaddy 3y agoThis was my takeaway as well. None of the other retrieval methods this paper benchmarks against are specifically trained on the corpus. I think it would be more fair to compare "Self-Retrieval" against models which have been fine-tuned on the corpus.