3 ms·
"What is the best way to augment the model with new knowledge other than at runtime using RAG? I'm afraid the answer is "at runtime using RAG". Don't fall int
by simonw 1y ago
"What is the best way to augment the model with new knowledge other than at runtime using RAG?
I'm afraid the answer is "at runtime using RAG".
Don't fall into the trap of assuming that RAG has to mean janky vector embeddings though. There are many different ways to implement RAG. Good old fashioned FTS search (using tools like Elasticsearch or Solar or even PostgreSQL/MySQL/SQLite FTS) it's a lot less complicated and less expensive to set up and can provide extremely good results.
A lot of of the common RAG techniques were put together a couple of years ago when models were less capable and input limits were still around 8000 tokens.
The models today are much cheaper, far better and mostly have 100,000+ token input limits. This opens up all sorts of new RAG possibilities.
I am very excited at the moment by tool-driven RAG: implement a "search" tool for an LLM to use and prompt it to try several iterations on its search terms before it gives up.
o3 and o4-mini do this in ChatGPT with their web search tool and the results are extremely convincing.
- simianwords 1y agoI agree that RAG does not have to be embeddings, RAG to me simply means augmenting new knowledge at run time no matter the method. I would like to convince you that RAG may not be ideal and is simply an approximation of real learned data. RAG is inherently constrained by context length which means any understanding has to happen within chunks of size 100k tokens (as you pointed out). Keep in mind that you still lose high level semantic understanding as you increase the prompt token length to 100k even if needle in the haystack type of problems are solved at this level. RAG introduces a severe limitation in understanding higher level semantic understanding across chunks. For instance, imagine a global variable shared across many modules causing some race conditions. This is extremely hard for RAG because it has to put many random modules in its context to deeply understand how the race condition happens. (to convince myself I must show that the linux codebase benefits from being indexed by an LLM and can solve hard to debug race conditions) Another situation where RAG fails is where the you don't even know what to put in your context to get the answer. Imagine a prompt like "tell me two movies released in 2025 that are surprisingly similar in terms of story line". Maybe O3 can solve this particular problem but imagine I start adding more constraints?
- simonw 1y agoSure, RAG isn't ideal. I don't know of an alternative. Attempting to constantly train it fine-tune entire new models to update their knowledge doesn't appear to be practical - I've not seen anyone demonstrate that working. I think long context plus tricks with tools is the best solution we have right now.
- simianwords 1y agoThe balance may tip in favour of fine tuning once we have made small breakthroughs in this space. It might be especially useful in enterprise contexts where you can have one model per company trained through all Wiki, code, documentation etc.
- simonw 1y agoThat right there is the thing I'm most skeptical of. It's so very obviously what every company wants: a custom model fine-tuned on their internal documentation and code. And yet stories of it actually working are incredibly rare! The closest I've heard to a success story in that space is Jane's Street who fine-tuned their own model because they use OCaml more than anyone else: https://www.youtube.com/watch?v=0ML7ZLMdcl4 https://www.youtube.com/watch?v=0ML7ZLMdcl4 I am confident that any startup today who could provably demonstrate that "we can fine tune a model on your company's internal code and documentation and have it answer questions about them" would have enormous financial success. I'll believe it works when I see that!