9 ms·

I did a hobby RAG project a little while back, and I'll just share my experience here. 1. First ask the LLM to answer your questions without RAG. It is easy to

I did a hobby RAG project a little while back, and I'll just share my experience here. 1. First ask the LLM to answer your questions without RAG. It is easy to do and you may be surprised (I was, but my data was semi-public). This also gives you a baseline to beat. 2. Chunking of your data needs to be smart. Just chunking every N characters wasn't especially fruitful. My data was a book, so it was hierarchal (by heading level). I would chunk by book section and hand it to the LLM. 3. Use the context window effectively. There is a weighted knapsack problem here, there are chunks of various sizes (chars/tokens) with various weightings (quality of match). If your data supports it, then the problem is also hierarchal. For example, I have 4 excellent matches in this chapter, so should I include each match, or should I include the whole chapter? 4. Quality of input data counts. I spent 30 minutes copy-pasting the entire book into markdown format. This was only a small project. I'd be interested to hear any other thoughts/tips.

nindalf 3y ago
It seems like you put a lot of thought and effort into this. Were you happy with the results? If you could put the entire book into the context window of Gemini (1M tokens), how do you think it would compare with your approach? It would cost like $15 to do this, so not cheap but also cost effective considering the time you’ve spent chunking it.
- s_tim 3y ago
  But you would need to spend the $15 on every request whereas the RAG approach would be most likely significantly cheaper per request.
- adamcharnock 3y ago
  At the time I was working with either GPT 3.5 or 4, and – looking at my old code - I was limiting myself to around 14k tokens. > Were you happy with the results? It was somewhat ok. I was testing the system from a (copyrighted) psychiatry textbook book and getting feedback on the output from a psychotherapist. The idea was to provide a tool to help therapists prep for a session, rather than help patients directly. As usual it was somewhat helpful but a little too vague sometimes, or missed important specific information for situation. It is possible that it could be improved with a larger context window, having more data to select, or different prompting. But the frequent response was along the lines of, "this is good advice, but it just doesn't drill down enough". Ultimately we found that GPT3.5/4 could produce responses that matched or exceeded our RAG-based solution. This was surprising as it is quite domain specific, but also it seemed pretty clear that GPT must have be trained on very data very similar the (copyrighted) content we were using. Further steps would be: 1. Use other LLM models. Is it just GPT3.5/4 that is reluctant to drill down? 2. Used specifically-trained LLMs(/LORA) based on the expected response style I'd be careful of entering this kind of arms race. It seems to be a fight against mediocre results, and at any moment OpenAI at al may release a new model that eats your lunch.
  - CuriouslyC 3y ago
    The thing you have to keep in mind is that OpenAI (and all the big tech companies) are risk averse. That's the reason that model alignment is so overbearing. The foundation models are also trying to be good at everything, which means they won't be sensitive to the nuances of specific types of questions. RAG and chatbot memory systems are here to stay, and they will always provide a benefit.
  - WhitneyLand 3y ago
    Did you try or consider fine-tuning Gpt 3.5 as a complementary approach?
kjqgqkejbfefn 3y ago
>tree-based approach to organize and summarize text data, capturing both high-level and low-level details. https://twitter.com/parthsarthi03/status/1753199233241674040 https://twitter.com/parthsarthi03/status/1753199233241674040 processes documents, organizing content and improving readability by handling sections, paragraphs, links, tables, lists, page continuations, and removing redundancies, watermarks, and applying OCR, with additional support for HTML and other formats through Apache Tika: https://github.com/nlmatics/nlm-ingestor https://github.com/nlmatics/nlm-ingestor
- osigurdson 3y ago
  I don't understand. Why build up text chunks from different, non-contiguous sections?
  - infecto 3y ago
    If those non-contiguous sections share similar semantic/other meaning, it can make sense from a search perspective to group them?
    - bageler 3y ago
      it starts to look like a graph problem
  - kjqgqkejbfefn 3y ago
    On the level of paper, not everything is laid out linearly. The main text is often laid out in column, the flow can be be offset with pictures with a caption, additional text can be placed in inserts, etc ... You need a human eye to figure that out and this is the task nlm-ingestor tackles. As for the content, semantic contiguity is not always guaranteed. A typical example of this are conversations, where people engage in narrative/argumentative competitions. Topics get nested as the conversation advances, along the lines of "Hey, this remind me of ...". Building up a stack that can be popped once subtopics have been exhausted: "To get back to the topic of ...". This is explored at length by Kebrat-Orecchioni in: https://www.cambridge.org/core/journals/language-in-society/article/abs/kerbrat-orecchionicatherine-les-interactions-verbales-vol-i-paris-armand-colin-1990-pp-318/16BCCCE4EEF5B0177A6AB7CF477C4A26 https://www.cambridge.org/core/journals/language-in-society/... And an explanation is offered by Dessalles in: https://telecom-paris.hal.science/hal-03814068/document https://telecom-paris.hal.science/hal-03814068/document
osigurdson 3y ago
>> Just chunking every N characters wasn't especially fruitful Is there any science associated with creative effective embedding sets? For a book, you could do every sentence, every paragraph, every page or every chapter (or all of these options). Eventually people will want to just point their RAG system at data and everything works.
- CuriouslyC 3y ago
  The easy answer is just use a model to chunk your data for you. Phi-2 can chunk and annotate with pre/post summary context in one pass, and it's pretty fast/cheap. There is an optimal chunk size, which IIRC is ~512 tokens depending on some factors. You could hierarchically model your data with embeddings by chunking the data, then generating summaries of those chunks and chunking the summaries, and repeating that process ad nauseum until you only have a small number of top level chunks.
  - drittich 3y ago
    How does this work when there is a limited context window. You do some pre-chunking?
    - CuriouslyC 3y ago
      Phi can ingest 2k tokens and the optimal chunk size is between 512-1024 depending on the model/application, so you just give it a big chunk and tell it to break it down into smaller chunks that are semantically related, leaving enough room for book-end sentences to enrich the context of the chunk. Then you start the next big chunk with the remnants of the previous one that the model couldn't group.
      - drittich 3y ago
        Isn't "give it a big chunk" just the same problem at a higher level? How do you handle, say, a book?
CuriouslyC 3y ago
Pro tip, you can use small models like phi-2 to do semantic chunking rather than just chunk on based on size, and chunking works even better if you book-end chunks with summaries of text coming before/after to enrich the context. Another pro tip, you can again use a small model to summarize/extract RAG content to get more actual data in the context.
- hack_edu 3y ago
  Could you share a bit more about semantic chunking with Phi? Any recommendations/examples of prompts?
  - CuriouslyC 3y ago
    Sure, it'll look something like this: """ Task: Divide the provided text into semantically coherent chunks, each containing between 250-350 words. Aim to preserve logical and thematic continuity within each chunk, ensuring that sentences or ideas that belong together are not split across different chunks. Guidelines: 1. Identify natural text breaks such as paragraph ends or section divides to initiate new chunks. 2. Estimate the word count as you include content in a chunk. Begin a new chunk when you reach approximately 250 words, preferring to end on a natural break close to this count, without exceeding 350 words. 3. In cases where text does not neatly fit within these constraints, prioritize maintaining the integrity of ideas and sentences over strict adherence to word limits. 4. Adjust the boundaries iteratively, refining your initial segmentation based on semantic coherence and word count guidelines. Your primary goal is to minimize disruption to the logical flow of content across chunks, even if slight deviations from the word count range are necessary to achieve this. """
    - eurekin 3y ago
      Is phi actually able to follow those instructions? How do you handle errors?
      - CuriouslyC 3y ago

nindalf 3y ago

It seems like you put a lot of thought and effort into this. Were you happy with the results? If you could put the entire book into the context window of Gemini (1M tokens), how do you think it would compare with your approach? It would cost like $15 to do this, so not cheap but also cost effective considering the time you’ve spent chunking it.

s_tim 3y ago
But you would need to spend the $15 on every request whereas the RAG approach would be most likely significantly cheaper per request.
adamcharnock 3y ago
At the time I was working with either GPT 3.5 or 4, and – looking at my old code - I was limiting myself to around 14k tokens. > Were you happy with the results? It was somewhat ok. I was testing the system from a (copyrighted) psychiatry textbook book and getting feedback on the output from a psychotherapist. The idea was to provide a tool to help therapists prep for a session, rather than help patients directly. As usual it was somewhat helpful but a little too vague sometimes, or missed important specific information for situation. It is possible that it could be improved with a larger context window, having more data to select, or different prompting. But the frequent response was along the lines of, "this is good advice, but it just doesn't drill down enough". Ultimately we found that GPT3.5/4 could produce responses that matched or exceeded our RAG-based solution. This was surprising as it is quite domain specific, but also it seemed pretty clear that GPT must have be trained on very data very similar the (copyrighted) content we were using. Further steps would be: 1. Use other LLM models. Is it just GPT3.5/4 that is reluctant to drill down? 2. Used specifically-trained LLMs(/LORA) based on the expected response style I'd be careful of entering this kind of arms race. It seems to be a fight against mediocre results, and at any moment OpenAI at al may release a new model that eats your lunch.
- CuriouslyC 3y ago
  The thing you have to keep in mind is that OpenAI (and all the big tech companies) are risk averse. That's the reason that model alignment is so overbearing. The foundation models are also trying to be good at everything, which means they won't be sensitive to the nuances of specific types of questions. RAG and chatbot memory systems are here to stay, and they will always provide a benefit.
- WhitneyLand 3y ago
  Did you try or consider fine-tuning Gpt 3.5 as a complementary approach?

kjqgqkejbfefn 3y ago

>tree-based approach to organize and summarize text data, capturing both high-level and low-level details. https://twitter.com/parthsarthi03/status/1753199233241674040 https://twitter.com/parthsarthi03/status/1753199233241674040 processes documents, organizing content and improving readability by handling sections, paragraphs, links, tables, lists, page continuations, and removing redundancies, watermarks, and applying OCR, with additional support for HTML and other formats through Apache Tika: https://github.com/nlmatics/nlm-ingestor https://github.com/nlmatics/nlm-ingestor

osigurdson 3y ago
I don't understand. Why build up text chunks from different, non-contiguous sections?
- infecto 3y ago
  If those non-contiguous sections share similar semantic/other meaning, it can make sense from a search perspective to group them?
  - bageler 3y ago
    it starts to look like a graph problem
- kjqgqkejbfefn 3y ago
  On the level of paper, not everything is laid out linearly. The main text is often laid out in column, the flow can be be offset with pictures with a caption, additional text can be placed in inserts, etc ... You need a human eye to figure that out and this is the task nlm-ingestor tackles. As for the content, semantic contiguity is not always guaranteed. A typical example of this are conversations, where people engage in narrative/argumentative competitions. Topics get nested as the conversation advances, along the lines of "Hey, this remind me of ...". Building up a stack that can be popped once subtopics have been exhausted: "To get back to the topic of ...". This is explored at length by Kebrat-Orecchioni in: https://www.cambridge.org/core/journals/language-in-society/article/abs/kerbrat-orecchionicatherine-les-interactions-verbales-vol-i-paris-armand-colin-1990-pp-318/16BCCCE4EEF5B0177A6AB7CF477C4A26 https://www.cambridge.org/core/journals/language-in-society/... And an explanation is offered by Dessalles in: https://telecom-paris.hal.science/hal-03814068/document https://telecom-paris.hal.science/hal-03814068/document

osigurdson 3y ago

>> Just chunking every N characters wasn't especially fruitful Is there any science associated with creative effective embedding sets? For a book, you could do every sentence, every paragraph, every page or every chapter (or all of these options). Eventually people will want to just point their RAG system at data and everything works.

CuriouslyC 3y ago
The easy answer is just use a model to chunk your data for you. Phi-2 can chunk and annotate with pre/post summary context in one pass, and it's pretty fast/cheap. There is an optimal chunk size, which IIRC is ~512 tokens depending on some factors. You could hierarchically model your data with embeddings by chunking the data, then generating summaries of those chunks and chunking the summaries, and repeating that process ad nauseum until you only have a small number of top level chunks.
- drittich 3y ago
  How does this work when there is a limited context window. You do some pre-chunking?
  - CuriouslyC 3y ago
    Phi can ingest 2k tokens and the optimal chunk size is between 512-1024 depending on the model/application, so you just give it a big chunk and tell it to break it down into smaller chunks that are semantically related, leaving enough room for book-end sentences to enrich the context of the chunk. Then you start the next big chunk with the remnants of the previous one that the model couldn't group.
    - drittich 3y ago
      Isn't "give it a big chunk" just the same problem at a higher level? How do you handle, say, a book?

CuriouslyC 3y ago

Pro tip, you can use small models like phi-2 to do semantic chunking rather than just chunk on based on size, and chunking works even better if you book-end chunks with summaries of text coming before/after to enrich the context. Another pro tip, you can again use a small model to summarize/extract RAG content to get more actual data in the context.

hack_edu 3y ago
Could you share a bit more about semantic chunking with Phi? Any recommendations/examples of prompts?
- CuriouslyC 3y ago
  Sure, it'll look something like this: """ Task: Divide the provided text into semantically coherent chunks, each containing between 250-350 words. Aim to preserve logical and thematic continuity within each chunk, ensuring that sentences or ideas that belong together are not split across different chunks. Guidelines: 1. Identify natural text breaks such as paragraph ends or section divides to initiate new chunks. 2. Estimate the word count as you include content in a chunk. Begin a new chunk when you reach approximately 250 words, preferring to end on a natural break close to this count, without exceeding 350 words. 3. In cases where text does not neatly fit within these constraints, prioritize maintaining the integrity of ideas and sentences over strict adherence to word limits. 4. Adjust the boundaries iteratively, refining your initial segmentation based on semantic coherence and word count guidelines. Your primary goal is to minimize disruption to the logical flow of content across chunks, even if slight deviations from the word count range are necessary to achieve this. """
  - eurekin 3y ago
    Is phi actually able to follow those instructions? How do you handle errors?
    - CuriouslyC 3y ago