6 ms·

Solving the out-of-context chunk problem for RAG

260 points by zmccormick7 2y ago

Satam 2y ago
RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.
- repeekad 2y ago
  What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
  - jstummbillig 2y ago
    I don't think the parents idea was to bake all information into the model, just that current RAG feels cumbersome to use (but then again, so do most things AI right now) and information access should be intrinsic part of the model.
    - deleted 2y ago
      [deleted]
    - randomdata 2y ago
      Is there a specific shortcoming of the model that could be improved, or are we simply seeking better APIs?
  - PaulHoule 2y ago
    One of my favorite cases is sports chat. I'd expect ChatGPT to be able to talk about sports legends but not be able to talk about a game that happened last weekend. Copilot usually does a good job because it can look up the game on Bing and them summarize but the other day i asked it "What happened last week in the NFL" and it told me about a Buffalo Bills game from last year (did it know I was in the Bills geography?) Some kind of incremental fine tuning is probably necessary to keep a model like ChatGPT up to date but I can't picture it happening each time something happens in the news.
    - ec109685 2y ago
      For the current game, it seems solvable by providing it the Boxscore and the radio commentary as context, perhaps with some additional data derived from recent games and news. I think you’d get a close approximation of speaking with someone who was watching the game with you.
- viraptor 2y ago
  That's so vague I can't tell what you're suggesting. What specifically do you think needs solving at the model level? What should work differently?
  - Satam 2y ago
    There’s probably lack of cpabalities on multiple fronts. RAG might have the right general idea but currently the retrieval seems to be too seperated from the model itself. I don’t know how our brains do it, but retrieval looks to be more integrated there. Models currently also have no way to update themselves with new info besides us putting data into their context window. They don’t learn after the initial training. It seems if they could, say, read documentation and internalize it, the need for RAG or even large context windows would decrease. Humans somehow are able to build understanding of extensive topics with what feels to be a much shorter context-window.
    - simonw 2y ago
      Don't forget the importance of data privacy. Updating a model with fresh information makes that information available to ALL users of that model. This often isn't what you want - you can run RAG against a user's private email to answer just their queries, without making that email "baked in" to the model.
      - viraptor 2y ago
        You don't need to update the whole model for everyone. Fine tuning exists and is even available as a service in openai. The updates are only visible in the specific models you see.
        simonw 2y ago
        Maintaining a fine-tuned model for every one of your users - even with techniques like LoRA - sounds complicated and expensive to me!
        2y ago
- ac1spkrbox 2y ago
  The set of techniques for retrieval is immature, but it's important to note that just relying on model context or few-shot prompting has many drawbacks. Perhaps the most important is that retrieval as a task should not rely on generative outputs.
  - danielbln 2y ago
    It's also subject to significantly more hallucination when the knowledge is baked into the model, vs being injected into the context at runtime.
- l72 2y ago
  I've described it this way to my colleagues: RAG is a bit like having a pretty smart person take an open book test on a subject they are not an expert in. If your book has a good chapter layout and index, you probably do an ok job trying to find relevant information, quickly read it, and try to come up with an answer. But your not going to be able to test for a deep understanding of the material. This person is going to struggle if each chapter/concept builds on the previous concept, as you can't just look up something in Chapter 10 and be able to understand it without understanding Chapter 1-9. Fine-tuning is a bit more like having someone go off and do a phd and specialize in a specific area. They get a much deeper understanding for the problem space and can conceptualize at a different level.
  - CGamesPlay 2y ago
    What you said about RAG makes sense, but my understanding is that fine-tuning is actually not very good at getting deeper understanding out of LLMs. It's more useful for teaching general instructions like output format rather than teaching deep concepts like a new domain of science.
    - bashfulpup 2y ago
      This is true if you don't know what you're doing, so it is good advice for the vast majority. Fine tuning is just training. You can completely change the model if you want make learn anything you want. But there are MANY challenges in doing so.
      - CGamesPlay 2y ago
        This isn't true either, because if you don't have access to the original data set, the model will overfit on your fine tuning data set and (in the extreme cases) lose its ability to even do basic reasoning.
        bashfulpup 2y ago
        Again, that's why I said it is challenging. I regularly do fine tuning on a model with fine results and little damage to the base functionality. It is possible, but it's too complex for the majority of users. It requires a lot of work per dataset you want trained on.
- williamtrask 2y ago
  Fwiw, I used to think this way too but LLMs are more RAG-like internally than we initially realised. Attention is all you need ~= RAG is a big attention mechanism. Models have reverse curse, memorisation issues etc. I personally think of LLMs as a kind of decomposed RAG. Check out DeepMind’s RETRO paper for an even closer integration.
- jejeyyy77 2y ago
  The biggest problem with RAG is that the bottleneck for your product is now the RAG (i.e, results are only as good as what your vector store sends to the LLM). This is a step backwards. Source: built a few products using RAG+LLM products.
- zby 2y ago
  I guess you can imagine an LLM that contains all information there is - but it would have to be at least as big as all information there is or it would have to hallucinate. And also you Not to mention that it seems that you would also require it to learn everything immediately. I don't see any realistic way to reach that goal. To reach their potential LLMs need to know how to use external sources. Update: After some more thinking - if you required it to know information about itself - then this would lead to some paradox - I am sure.
  - bashfulpup 2y ago
    A CL agent is next generation AI. When CL is properly implemented in an LLM agent format, most of these systems vanish.
CharlieDigital 2y ago
The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.
- williamcotton 2y ago
  Contextual chunk headers The idea here is to add in higher-level context to the chunk by prepending a chunk header. This chunk header could be as simple as just the document title, or it could use a combination of document title, a concise document summary, and the full hierarchy of section and sub-section titles. That is from the article. Is this different from your suggested approach?
  - CharlieDigital 2y ago
    No, but this is also not really a novel solution.
- lmeyerov 2y ago
  So subtle! The article is on doing that, which is something we are doing a lot on right now... though it seems to snatch defeat from the jaws of victory: If we think about what this is about, it is basically entity augmentation & lexical linking / citations. Ex: A patient document may be all about patient id 123. That won't be spelled out in every paragraph, but by carrying along the patient ID (semantic entity) and the document (citation), the combined model gets access to them. A naive one-shot retrieval over a naive chunked vector index would want it at the text/embedding, while a smarter one also in the entry metadata. And as others write, this helps move reasoning from the symbolic domain to the semantic domain, so less of a hack. We are working on some fun 'pure-vector' graph RAG work here to tackle production problems around scale, quality, & always-on scenarios like alerting - happy to chat!
  - Reyessproooo 2y ago
aster0id 2y ago
I'd like to see more evaluation data. There are 100s of RAG strategies, most of them only work on specific types of queries.
- gillesjacobs 2y ago
  Yeah exactly, existing benchmark datasets available are underutilized (eg KILT, Natural questions, etc.). But it is only natural that different QA use cases require different strategies. I built 3 production RAG systems / virtual assistant now, and 4 that didn't make it past PoC and what advanced techniques works really depends on document type, text content and genre, use case, source knowledgebase structure and metadata to exploit etc. Current go-to is semantic similarity chunking (with overlap) + title or question generation > retriever with fusion on bienc vector sim + classic bm25 + condensed question reformulated QA agent. If you don't get some decent results with that setup there is no hope. For every project we start the creation of a use-case eval set immediately in parallel with the actual RAG agent, but sometimes the client doesn't think this is priority. We convinced them all it's highly important though, because it is. Having an evaluation set is doubly important in GenAI projects: a generative system will do unexpected things and an objective measure is needed. Your client will run into weird behaviour when testing and they will get hung up on a 1-in-100 undesirable generation.
  - drittich 2y ago
    How do you weight results between vector search and bm25? Do you fall back to bm25 when vector similarity is below a threshold, or maybe you tweak the weights by hand for each data set?
    - gillesjacobs 2y ago
      The algorithm I use to get a final ranking from multiple rankings is called "reciprocal ranked fusion". I use the implementation described here: https://docs.llamaindex.ai/en/stable/examples/low_level/fusion_retriever/#step-3-perform-fusion https://docs.llamaindex.ai/en/stable/examples/low_level/fusi... Which is the implementation from the original paper.
siquick 2y ago
I can’t imagine any serious RAG application is not doing this - adding a contextual title, summary, keywords, and questions to the metadata of each chunk is a pretty low effort/high return implementation.
- visarga 2y ago
  Text embeds don't capture inferred data, like "second letter of this text" does not embed close to "e". LLM chain of thought is required to deduce the meaning more completely.
  - derefr 2y ago
    Given current SOTA, no, they don’t. But there’s no reason why they couldn’t — just capture the vectors of some of the earlier hidden layers during the RAG encoder’s inference run, and append these intermediate vectors to the final embedding vector of the output layer to become the vectors you throw into your vector DB. (And then do the same at runtime for embedding your query prompts.) Probably you’d want to bias those internal-layer vectors, giving them an increasingly-high “artificial distance” coefficient for increasingly-early layers — so that a document closely matching in token space or word space or syntax-node space improves its retrieval rank a bit, but not nearly as much as if the document were a close match in concept space. (But maybe do something nonlinear instead of multiplication here — you might want near-identical token-wise or syntax-wise matches to show up despite different meanings, depending on your use-case.) Come to think, you could probably build a pretty good source-code search RAG off of this approach. (Also, it should hopefully be obvious here that if you fine-tuned an encoder-decoder LLM to label matches based on criteria where some of those criteria are only available in earlier layers, then you’d be training pass-through vector dimensions into the intermediate layers of the encoder — such that using such an encoder on its own for RAG embedding should produce the same effect as capturing + weighting the intermediate layers of a non-fine-tuned LLM.)
- samx18 2y ago
  I agree, most production RAG systems have been doing this since last year
- 2y ago
gillesjacobs 2y ago
I really want to see some evaluation benchmark comparisons on in-chunk augmentation approaches like this (and question, title, header-generation) and the hybrid retrieval approach where you match at multiple levels: first retrieve/filter on a higher-level summary, title or header, then match the related chunks. The pure vector approach of in-chunk text augmentation is much simpler of course, but my hypothesis is that the resulting vector will cause too much false positives in retrieval. In my experience retrieval precision is most commonly the problem not recall with vector similarity. This method will indeed improve recall for out-of-context chunks, but for me recall has not been a problem very often.
bob1029 2y ago
I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then, look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any value without having this working first.
- k__ 2y ago
  I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
  - te_chris 2y ago
    Honestly you clocked the secret: it doesn’t. It makes sense for the hype, though. As we got LLM’s we also got wayyyy better embedding models, but they’re not dependencies.
  - simonw 2y ago
    It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/ https://simonwillison.net/2024/Jun/21/search-based-rag/
    - rcarmo 2y ago
      I've been using SQLite FTS (which is essentially BM25) and it works so well I haven't really bothered with vector databases, or Postgres, or anything else yet. Maybe when my corpus exceeds 2GB...
    - ianbutler 2y ago
      In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs. That seems like a better stacking of the technologies even now
ankit219 2y ago
As is typical with any RAG strategy/algorithm, the implicit thing is it works on a specific dataset. Then, it solves a very specific use case. The thing is, if you have a dataset and a use case, you can have a very custom algorithm which would work wonders in terms of output you need. There need not be anything generic. My instinct at this point is, these algos look attractive because we are constrained to giving a user a wow moment where they upload something and get to chat with the doc/dataset within minutes. As attractive as that is, it is a distinct second priority to building a system that works 99% of the time, even if takes a day or two to set up. You get a feel of the data, have a feel of type of questions that may be asked, and create an algo that works for a specific type of dataset-usecase combo (assuming any more data you add in this system would be similar and work pretty well). There is no silver bullet that we seem to be searching for.
- cl42 2y ago
  100% agree with you. I've built a # of RAG systems and find that simple Q&A-style use cases actually do fine with traditional chunking approaches. ... and then you have situations where people ask complex questions with multiple logical steps, or knowledge gathering requirements, and using some sort of hierarchical RAG strategy works better. I think a lot of solutions (including this post) abstract to building knowledge graphs of some sort... But knowledge graphs still require an ontology associated to the problem you're solving and will fail outside of those domains.
oshams 2y ago
Have you considered this approach? Worked well for us: https://news.ycombinator.com/item?id=40998497 https://news.ycombinator.com/item?id=40998497
Sharlin 2y ago
“An Outside Context Problem was the sort of thing most civilisations encountered just once, and which they tended to encounter rather in the same way a sentence encountered a full stop.” https://www.goodreads.com/quotes/9605621-an-outside-context-problem-was-the-sort-of-thing-most https://www.goodreads.com/quotes/9605621-an-outside-context-... (Sorry, I just had to post this quote because it was the first thing that came to my mind when I saw the title, and I've been re-reading Banks lately.)
unixhero 2y ago
I have identified a painpoint where my RAGs are insufficiently answering what I already had with a long tail DAG in production.
CGamesPlay 2y ago
An interesting paper that was recently published that talks about a different approach: Human-like Episodic Memory for Infinite Context LLMs <https://arxiv.org/abs/2407.09450 https://arxiv.org/abs/2407.09450> This wasn't focused on RAG, but there seems to be a lot of crossover to me. Using the LLM to make "episodes" is a similar problem to chunking, and letting the LLM decide the boundary might also yield good results.
brhsagain 2y ago
I've never seen so many epicycles in my life...
iAkashPaul 2y ago
One quick way to improve results greatly is to ask questions with 2/3 chunks & in the lookup for these chunks mention the IDs of the other chunks, qdrant allows for easy metadata addition. So just generate a synthetic question bank & then do vSearch against the same instead of hoping for the chunks to match up with user questions.
oiphl 2y ago
[flagged]
hackernoteng 2y ago
I experience worse IR performance adding title/headers to chunks. It really depends on the nature of the documents. The only successful RAG systems I see are ones specifically tuned to a single domain and document type. If your document collection is diverse in domains or formats, good luck.

260 points by zmccormick7 2y ago

RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.

repeekad 2y ago
What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
- jstummbillig 2y ago
  I don't think the parents idea was to bake all information into the model, just that current RAG feels cumbersome to use (but then again, so do most things AI right now) and information access should be intrinsic part of the model.
  - deleted 2y ago
    [deleted]
  - randomdata 2y ago
    Is there a specific shortcoming of the model that could be improved, or are we simply seeking better APIs?
- PaulHoule 2y ago
  One of my favorite cases is sports chat. I'd expect ChatGPT to be able to talk about sports legends but not be able to talk about a game that happened last weekend. Copilot usually does a good job because it can look up the game on Bing and them summarize but the other day i asked it "What happened last week in the NFL" and it told me about a Buffalo Bills game from last year (did it know I was in the Bills geography?) Some kind of incremental fine tuning is probably necessary to keep a model like ChatGPT up to date but I can't picture it happening each time something happens in the news.
  - ec109685 2y ago
    For the current game, it seems solvable by providing it the Boxscore and the radio commentary as context, perhaps with some additional data derived from recent games and news. I think you’d get a close approximation of speaking with someone who was watching the game with you.
viraptor 2y ago
That's so vague I can't tell what you're suggesting. What specifically do you think needs solving at the model level? What should work differently?
- Satam 2y ago
  There’s probably lack of cpabalities on multiple fronts. RAG might have the right general idea but currently the retrieval seems to be too seperated from the model itself. I don’t know how our brains do it, but retrieval looks to be more integrated there. Models currently also have no way to update themselves with new info besides us putting data into their context window. They don’t learn after the initial training. It seems if they could, say, read documentation and internalize it, the need for RAG or even large context windows would decrease. Humans somehow are able to build understanding of extensive topics with what feels to be a much shorter context-window.
  - simonw 2y ago
    Don't forget the importance of data privacy. Updating a model with fresh information makes that information available to ALL users of that model. This often isn't what you want - you can run RAG against a user's private email to answer just their queries, without making that email "baked in" to the model.
    - viraptor 2y ago
      You don't need to update the whole model for everyone. Fine tuning exists and is even available as a service in openai. The updates are only visible in the specific models you see.
      - simonw 2y ago
        Maintaining a fine-tuned model for every one of your users - even with techniques like LoRA - sounds complicated and expensive to me!
        2y ago
ac1spkrbox 2y ago
The set of techniques for retrieval is immature, but it's important to note that just relying on model context or few-shot prompting has many drawbacks. Perhaps the most important is that retrieval as a task should not rely on generative outputs.
- danielbln 2y ago
  It's also subject to significantly more hallucination when the knowledge is baked into the model, vs being injected into the context at runtime.
l72 2y ago
I've described it this way to my colleagues: RAG is a bit like having a pretty smart person take an open book test on a subject they are not an expert in. If your book has a good chapter layout and index, you probably do an ok job trying to find relevant information, quickly read it, and try to come up with an answer. But your not going to be able to test for a deep understanding of the material. This person is going to struggle if each chapter/concept builds on the previous concept, as you can't just look up something in Chapter 10 and be able to understand it without understanding Chapter 1-9. Fine-tuning is a bit more like having someone go off and do a phd and specialize in a specific area. They get a much deeper understanding for the problem space and can conceptualize at a different level.
- CGamesPlay 2y ago
  What you said about RAG makes sense, but my understanding is that fine-tuning is actually not very good at getting deeper understanding out of LLMs. It's more useful for teaching general instructions like output format rather than teaching deep concepts like a new domain of science.
  - bashfulpup 2y ago
    This is true if you don't know what you're doing, so it is good advice for the vast majority. Fine tuning is just training. You can completely change the model if you want make learn anything you want. But there are MANY challenges in doing so.
    - CGamesPlay 2y ago
      This isn't true either, because if you don't have access to the original data set, the model will overfit on your fine tuning data set and (in the extreme cases) lose its ability to even do basic reasoning.
      - bashfulpup 2y ago
        Again, that's why I said it is challenging. I regularly do fine tuning on a model with fine results and little damage to the base functionality. It is possible, but it's too complex for the majority of users. It requires a lot of work per dataset you want trained on.
williamtrask 2y ago
Fwiw, I used to think this way too but LLMs are more RAG-like internally than we initially realised. Attention is all you need ~= RAG is a big attention mechanism. Models have reverse curse, memorisation issues etc. I personally think of LLMs as a kind of decomposed RAG. Check out DeepMind’s RETRO paper for an even closer integration.
jejeyyy77 2y ago
The biggest problem with RAG is that the bottleneck for your product is now the RAG (i.e, results are only as good as what your vector store sends to the LLM). This is a step backwards. Source: built a few products using RAG+LLM products.
zby 2y ago
I guess you can imagine an LLM that contains all information there is - but it would have to be at least as big as all information there is or it would have to hallucinate. And also you Not to mention that it seems that you would also require it to learn everything immediately. I don't see any realistic way to reach that goal. To reach their potential LLMs need to know how to use external sources. Update: After some more thinking - if you required it to know information about itself - then this would lead to some paradox - I am sure.
- bashfulpup 2y ago
  A CL agent is next generation AI. When CL is properly implemented in an LLM agent format, most of these systems vanish.

CharlieDigital 2y ago

The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.

williamcotton 2y ago
Contextual chunk headers The idea here is to add in higher-level context to the chunk by prepending a chunk header. This chunk header could be as simple as just the document title, or it could use a combination of document title, a concise document summary, and the full hierarchy of section and sub-section titles. That is from the article. Is this different from your suggested approach?
- CharlieDigital 2y ago
  No, but this is also not really a novel solution.
lmeyerov 2y ago
So subtle! The article is on doing that, which is something we are doing a lot on right now... though it seems to snatch defeat from the jaws of victory: If we think about what this is about, it is basically entity augmentation & lexical linking / citations. Ex: A patient document may be all about patient id 123. That won't be spelled out in every paragraph, but by carrying along the patient ID (semantic entity) and the document (citation), the combined model gets access to them. A naive one-shot retrieval over a naive chunked vector index would want it at the text/embedding, while a smarter one also in the entry metadata. And as others write, this helps move reasoning from the symbolic domain to the semantic domain, so less of a hack. We are working on some fun 'pure-vector' graph RAG work here to tackle production problems around scale, quality, & always-on scenarios like alerting - happy to chat!
- Reyessproooo 2y ago

aster0id 2y ago

I'd like to see more evaluation data. There are 100s of RAG strategies, most of them only work on specific types of queries.

gillesjacobs 2y ago
Yeah exactly, existing benchmark datasets available are underutilized (eg KILT, Natural questions, etc.). But it is only natural that different QA use cases require different strategies. I built 3 production RAG systems / virtual assistant now, and 4 that didn't make it past PoC and what advanced techniques works really depends on document type, text content and genre, use case, source knowledgebase structure and metadata to exploit etc. Current go-to is semantic similarity chunking (with overlap) + title or question generation > retriever with fusion on bienc vector sim + classic bm25 + condensed question reformulated QA agent. If you don't get some decent results with that setup there is no hope. For every project we start the creation of a use-case eval set immediately in parallel with the actual RAG agent, but sometimes the client doesn't think this is priority. We convinced them all it's highly important though, because it is. Having an evaluation set is doubly important in GenAI projects: a generative system will do unexpected things and an objective measure is needed. Your client will run into weird behaviour when testing and they will get hung up on a 1-in-100 undesirable generation.
- drittich 2y ago
  How do you weight results between vector search and bm25? Do you fall back to bm25 when vector similarity is below a threshold, or maybe you tweak the weights by hand for each data set?
  - gillesjacobs 2y ago
    The algorithm I use to get a final ranking from multiple rankings is called "reciprocal ranked fusion". I use the implementation described here: https://docs.llamaindex.ai/en/stable/examples/low_level/fusion_retriever/#step-3-perform-fusion https://docs.llamaindex.ai/en/stable/examples/low_level/fusi... Which is the implementation from the original paper.

siquick 2y ago

I can’t imagine any serious RAG application is not doing this - adding a contextual title, summary, keywords, and questions to the metadata of each chunk is a pretty low effort/high return implementation.

visarga 2y ago
Text embeds don't capture inferred data, like "second letter of this text" does not embed close to "e". LLM chain of thought is required to deduce the meaning more completely.
- derefr 2y ago
  Given current SOTA, no, they don’t. But there’s no reason why they couldn’t — just capture the vectors of some of the earlier hidden layers during the RAG encoder’s inference run, and append these intermediate vectors to the final embedding vector of the output layer to become the vectors you throw into your vector DB. (And then do the same at runtime for embedding your query prompts.) Probably you’d want to bias those internal-layer vectors, giving them an increasingly-high “artificial distance” coefficient for increasingly-early layers — so that a document closely matching in token space or word space or syntax-node space improves its retrieval rank a bit, but not nearly as much as if the document were a close match in concept space. (But maybe do something nonlinear instead of multiplication here — you might want near-identical token-wise or syntax-wise matches to show up despite different meanings, depending on your use-case.) Come to think, you could probably build a pretty good source-code search RAG off of this approach. (Also, it should hopefully be obvious here that if you fine-tuned an encoder-decoder LLM to label matches based on criteria where some of those criteria are only available in earlier layers, then you’d be training pass-through vector dimensions into the intermediate layers of the encoder — such that using such an encoder on its own for RAG embedding should produce the same effect as capturing + weighting the intermediate layers of a non-fine-tuned LLM.)
samx18 2y ago
I agree, most production RAG systems have been doing this since last year
2y ago

gillesjacobs 2y ago

I really want to see some evaluation benchmark comparisons on in-chunk augmentation approaches like this (and question, title, header-generation) and the hybrid retrieval approach where you match at multiple levels: first retrieve/filter on a higher-level summary, title or header, then match the related chunks. The pure vector approach of in-chunk text augmentation is much simpler of course, but my hypothesis is that the resulting vector will cause too much false positives in retrieval. In my experience retrieval precision is most commonly the problem not recall with vector similarity. This method will indeed improve recall for out-of-context chunks, but for me recall has not been a problem very often.

bob1029 2y ago

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then, look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any value without having this working first.

k__ 2y ago
I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
- te_chris 2y ago
  Honestly you clocked the secret: it doesn’t. It makes sense for the hype, though. As we got LLM’s we also got wayyyy better embedding models, but they’re not dependencies.
- simonw 2y ago
  It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/ https://simonwillison.net/2024/Jun/21/search-based-rag/
  - rcarmo 2y ago
    I've been using SQLite FTS (which is essentially BM25) and it works so well I haven't really bothered with vector databases, or Postgres, or anything else yet. Maybe when my corpus exceeds 2GB...
  - ianbutler 2y ago
    In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs. That seems like a better stacking of the technologies even now

ankit219 2y ago

As is typical with any RAG strategy/algorithm, the implicit thing is it works on a specific dataset. Then, it solves a very specific use case. The thing is, if you have a dataset and a use case, you can have a very custom algorithm which would work wonders in terms of output you need. There need not be anything generic. My instinct at this point is, these algos look attractive because we are constrained to giving a user a wow moment where they upload something and get to chat with the doc/dataset within minutes. As attractive as that is, it is a distinct second priority to building a system that works 99% of the time, even if takes a day or two to set up. You get a feel of the data, have a feel of type of questions that may be asked, and create an algo that works for a specific type of dataset-usecase combo (assuming any more data you add in this system would be similar and work pretty well). There is no silver bullet that we seem to be searching for.

cl42 2y ago
100% agree with you. I've built a # of RAG systems and find that simple Q&A-style use cases actually do fine with traditional chunking approaches. ... and then you have situations where people ask complex questions with multiple logical steps, or knowledge gathering requirements, and using some sort of hierarchical RAG strategy works better. I think a lot of solutions (including this post) abstract to building knowledge graphs of some sort... But knowledge graphs still require an ontology associated to the problem you're solving and will fail outside of those domains.

oshams 2y ago

Have you considered this approach? Worked well for us: https://news.ycombinator.com/item?id=40998497 https://news.ycombinator.com/item?id=40998497

Sharlin 2y ago

“An Outside Context Problem was the sort of thing most civilisations encountered just once, and which they tended to encounter rather in the same way a sentence encountered a full stop.” https://www.goodreads.com/quotes/9605621-an-outside-context-problem-was-the-sort-of-thing-most https://www.goodreads.com/quotes/9605621-an-outside-context-... (Sorry, I just had to post this quote because it was the first thing that came to my mind when I saw the title, and I've been re-reading Banks lately.)

unixhero 2y ago

I have identified a painpoint where my RAGs are insufficiently answering what I already had with a long tail DAG in production.

CGamesPlay 2y ago

An interesting paper that was recently published that talks about a different approach: Human-like Episodic Memory for Infinite Context LLMs <https://arxiv.org/abs/2407.09450 https://arxiv.org/abs/2407.09450> This wasn't focused on RAG, but there seems to be a lot of crossover to me. Using the LLM to make "episodes" is a similar problem to chunking, and letting the LLM decide the boundary might also yield good results.

brhsagain 2y ago

I've never seen so many epicycles in my life...

iAkashPaul 2y ago

One quick way to improve results greatly is to ask questions with 2/3 chunks & in the lookup for these chunks mention the IDs of the other chunks, qdrant allows for easy metadata addition. So just generate a synthetic question bank & then do vSearch against the same instead of hoping for the chunks to match up with user questions.

oiphl 2y ago

[flagged]

hackernoteng 2y ago

I experience worse IR performance adding title/headers to chunks. It really depends on the nature of the documents. The only successful RAG systems I see are ones specifically tuned to a single domain and document type. If your document collection is diverse in domains or formats, good luck.