4 ms·

Related: what is the best way to augment the model with new knowledge other than at runtime using RAG?

scosman 1y ago
Context window + prompt caching if you can't use RAG. Can add a lot to long context models, and their needle in haystack metrics keep getting better. Why can't you use RAG?
- simianwords 1y ago
  you lose coherence across chunks of context size. i wish i could spend compute to pre-train on some knowledge.
ijk 1y ago
Depends on the definition of "knowledge"; there's a lot of factors that go into it. Some of the common approaches are continued/continual pretraining and model editing (https://arxiv.org/pdf/2502.12598 https://arxiv.org/pdf/2502.12598). * Models are bad at learning that A=B implies B=A, let alone more complicated relations; augmenting the dataset with multiple examples with different phrasing/perspectives is important (https://arxiv.org/abs/2404.00213 https://arxiv.org/abs/2404.00213). The frequency that a relation occurs in the dataset affects the results (https://arxiv.org/html/2504.09597v2 https://arxiv.org/html/2504.09597v2). * You have to be able to balance preserving existing knowledge against the new knowledge (https://arxiv.org/abs/2502.14502 https://arxiv.org/abs/2502.14502). There are techniques like making sure your data mix corresponds to the original training data, but new data is primed by existing data so it gets complicated (https://arxiv.org/abs/2504.09522 https://arxiv.org/abs/2504.09522). * Curriculum training (a la Phi) can be quite effective for training knowledge into base models at the very least. * Continued pretraining is much more difficult than most finetuning, though it is possible (https://unsloth.ai/blog/contpretraining https://unsloth.ai/blog/contpretraining). * Model editing of individual facts is possible but tricky because everything is interconnected but the model isn't great at figuring out reciprocal relationships (https://arxiv.org/abs/2310.16218 https://arxiv.org/abs/2310.16218). There's been some slow progress, though I find that few people are aware that it is even possible, despite the progress that has been made (https://github.com/zjunlp/KnowledgeEditingPapers https://github.com/zjunlp/KnowledgeEditingPapers). The keywords you want are knowledge injection, domain adaptation, continual pretraining, model editing.
- simianwords 1y ago
  This is exactly what I was talking about. I wonder why no one has tried to inject a critical code repository (at least 1 million LOC) and compare to common RAG methods? The ones you have shown here are nice and simple like world cup statistics. Maybe we are nowhere near solving such complicated scenarios?
simonw 1y ago
"What is the best way to augment the model with new knowledge other than at runtime using RAG? I'm afraid the answer is "at runtime using RAG". Don't fall into the trap of assuming that RAG has to mean janky vector embeddings though. There are many different ways to implement RAG. Good old fashioned FTS search (using tools like Elasticsearch or Solar or even PostgreSQL/MySQL/SQLite FTS) it's a lot less complicated and less expensive to set up and can provide extremely good results. A lot of of the common RAG techniques were put together a couple of years ago when models were less capable and input limits were still around 8000 tokens. The models today are much cheaper, far better and mostly have 100,000+ token input limits. This opens up all sorts of new RAG possibilities. I am very excited at the moment by tool-driven RAG: implement a "search" tool for an LLM to use and prompt it to try several iterations on its search terms before it gives up. o3 and o4-mini do this in ChatGPT with their web search tool and the results are extremely convincing.
- simianwords 1y ago
  I agree that RAG does not have to be embeddings, RAG to me simply means augmenting new knowledge at run time no matter the method. I would like to convince you that RAG may not be ideal and is simply an approximation of real learned data. RAG is inherently constrained by context length which means any understanding has to happen within chunks of size 100k tokens (as you pointed out). Keep in mind that you still lose high level semantic understanding as you increase the prompt token length to 100k even if needle in the haystack type of problems are solved at this level. RAG introduces a severe limitation in understanding higher level semantic understanding across chunks. For instance, imagine a global variable shared across many modules causing some race conditions. This is extremely hard for RAG because it has to put many random modules in its context to deeply understand how the race condition happens. (to convince myself I must show that the linux codebase benefits from being indexed by an LLM and can solve hard to debug race conditions) Another situation where RAG fails is where the you don't even know what to put in your context to get the answer. Imagine a prompt like "tell me two movies released in 2025 that are surprisingly similar in terms of story line". Maybe O3 can solve this particular problem but imagine I start adding more constraints?
  - 1y ago

Context window + prompt caching if you can't use RAG. Can add a lot to long context models, and their needle in haystack metrics keep getting better. Why can't you use RAG?

simianwords 1y ago
you lose coherence across chunks of context size. i wish i could spend compute to pre-train on some knowledge.

ijk 1y ago

Depends on the definition of "knowledge"; there's a lot of factors that go into it. Some of the common approaches are continued/continual pretraining and model editing (https://arxiv.org/pdf/2502.12598 https://arxiv.org/pdf/2502.12598). * Models are bad at learning that A=B implies B=A, let alone more complicated relations; augmenting the dataset with multiple examples with different phrasing/perspectives is important (https://arxiv.org/abs/2404.00213 https://arxiv.org/abs/2404.00213). The frequency that a relation occurs in the dataset affects the results (https://arxiv.org/html/2504.09597v2 https://arxiv.org/html/2504.09597v2). * You have to be able to balance preserving existing knowledge against the new knowledge (https://arxiv.org/abs/2502.14502 https://arxiv.org/abs/2502.14502). There are techniques like making sure your data mix corresponds to the original training data, but new data is primed by existing data so it gets complicated (https://arxiv.org/abs/2504.09522 https://arxiv.org/abs/2504.09522). * Curriculum training (a la Phi) can be quite effective for training knowledge into base models at the very least. * Continued pretraining is much more difficult than most finetuning, though it is possible (https://unsloth.ai/blog/contpretraining https://unsloth.ai/blog/contpretraining). * Model editing of individual facts is possible but tricky because everything is interconnected but the model isn't great at figuring out reciprocal relationships (https://arxiv.org/abs/2310.16218 https://arxiv.org/abs/2310.16218). There's been some slow progress, though I find that few people are aware that it is even possible, despite the progress that has been made (https://github.com/zjunlp/KnowledgeEditingPapers https://github.com/zjunlp/KnowledgeEditingPapers). The keywords you want are knowledge injection, domain adaptation, continual pretraining, model editing.

simianwords 1y ago
This is exactly what I was talking about. I wonder why no one has tried to inject a critical code repository (at least 1 million LOC) and compare to common RAG methods? The ones you have shown here are nice and simple like world cup statistics. Maybe we are nowhere near solving such complicated scenarios?

simonw 1y ago

"What is the best way to augment the model with new knowledge other than at runtime using RAG? I'm afraid the answer is "at runtime using RAG". Don't fall into the trap of assuming that RAG has to mean janky vector embeddings though. There are many different ways to implement RAG. Good old fashioned FTS search (using tools like Elasticsearch or Solar or even PostgreSQL/MySQL/SQLite FTS) it's a lot less complicated and less expensive to set up and can provide extremely good results. A lot of of the common RAG techniques were put together a couple of years ago when models were less capable and input limits were still around 8000 tokens. The models today are much cheaper, far better and mostly have 100,000+ token input limits. This opens up all sorts of new RAG possibilities. I am very excited at the moment by tool-driven RAG: implement a "search" tool for an LLM to use and prompt it to try several iterations on its search terms before it gives up. o3 and o4-mini do this in ChatGPT with their web search tool and the results are extremely convincing.

simianwords 1y ago
I agree that RAG does not have to be embeddings, RAG to me simply means augmenting new knowledge at run time no matter the method. I would like to convince you that RAG may not be ideal and is simply an approximation of real learned data. RAG is inherently constrained by context length which means any understanding has to happen within chunks of size 100k tokens (as you pointed out). Keep in mind that you still lose high level semantic understanding as you increase the prompt token length to 100k even if needle in the haystack type of problems are solved at this level. RAG introduces a severe limitation in understanding higher level semantic understanding across chunks. For instance, imagine a global variable shared across many modules causing some race conditions. This is extremely hard for RAG because it has to put many random modules in its context to deeply understand how the race condition happens. (to convince myself I must show that the linux codebase benefits from being indexed by an LLM and can solve hard to debug race conditions) Another situation where RAG fails is where the you don't even know what to put in your context to get the answer. Imagine a prompt like "tell me two movies released in 2025 that are surprisingly similar in terms of story line". Maybe O3 can solve this particular problem but imagine I start adding more constraints?
- 1y ago