7 ms·
You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are s
by ilaksh 3y ago
You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for.
You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG.
Test it out. If it really and truly doesn't work, search for a script that creates question and answer pairs automatically with gpt-4. Then try using that for qLoRA. I have never heard of anyone successfully using that for a private document knowledgebase though. Only for skills like math, reasoning, Python, etc. I think the issue is that you need a LOT of data and it needs to repeat concepts or any facts you need to learn many, many times in different supporting ways.
What absolutely does not work is trying to just feed a set of documents into fine tuning. I personally have proven that dozens of times because I had a client who is determined to do it. He has been mislead.
What it will do is learn the patterns that are in those documents.
- min76 3y agoWell said. The problem is, there are way too many alternatives. Any idea how llamaindex's ingestion engine compares to unstructured.io? ( Which is used in langchain)
- ilaksh 3y agoI think they may be using the same thing.
- dfhg 3y agoAre there public examples of working products using RAG, compared with fine-tuning or training from scratch?
- c0pium 3y agoCopilots use RAG: https://www.microsoft.com/en-us/research/group/dynamics-insights-apps-artificial-intelligence-machine-learning/articles/prompt-engineering-improving-our-ability-to-communicate-with-an-llm/ https://www.microsoft.com/en-us/research/group/dynamics-insi...
- tinco 3y agoThe OpenAI assistants API is an implementation of a RAG pipeline. It performs both RAG on any documents you upload, and on any conversation you have with it that exceeds the context.
- regularfry 3y agoAmazon Q is (at least partially) a RAG implementation.
- aik 3y agoNot public but internally I wrote a tool to help us respond to RFPs. You pass in a question from a new RFP and it outputs surprisingly great answers most of the time. Is writing 75%+ of our RFP responses now (naturally we review and adjust sometimes and as needed). And best of all it was very quickly hacked together and it’s actually useful. Copied questions/answers from all previous ones into a doc, and am using OpenAI embeddings api + FAISS vector db + GPT-4 to load the chunks + store the embeddings + process the resulting chunks.
- deleted 3y ago[deleted]
- dfhg 3y agoAnother question, which one is preferred, LlamaIndex or Langchain, for RAG? Thanks in advance for your insights.
- isoprophlex 3y agoYou basically don't use langchain for anything besides 30 minute demos that you copied from someone else's github. It has a completely spaghettified API, is not performant, and forces you into excessive mental contortions to reason about otherwise simple tasks. LlamaIndex is pretty good.
- monkeydust 3y agoYea discovered this with Langchain last week. Was great for a demo then started to push it harder and spent ages trawling Reddit, discord, GitHub trying to find solutions to issues only to discover what was supposed to be supported was deprecated. Got a massive headache for what should have been a simple change. Moved on now.
- sophiabits 3y agoYeah +1 We originally started out building features with LangChain (loading chains from YAML sounded good—it felt like it would be easy to get non-engineers to help with prompt development) but in practice it’s just way too complicated. Nice idea, but the execution feels lacking. It also doesn’t help that LangChain is evolving so rapidly. When we first started using it a lot of code samples on the internet couldn’t be copy/pasted because of import paths changing, and at one point we had to bump by ~60 patch versions to get a bug fix, which was painful because it broke all kinds of stuff
- simonw 3y agoLlamaIndex is mainly focused on RAG. LangChain does a ton of other stuff too. I'd focus on LlamaIndex first.
- mathis-l 3y agoHaystack [1] is another good option. It‘s modular, doesn’t get in your way and is particularly strong at retrieval. People like the documentation too. Disclaimer: I work at deepset [1] https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack
- benjaminwootton 3y agoTo sing the praises of Bedrock again, it does have continuous pre-training as well as RAG “knowledge bases”. The former is based on JSON fragments and the RAG stuff is PDFs and other document formats. With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed. I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cases and for situations across the knowledge base. It did feel like it broke down and got confused at a certain point of complexity of questioning, but I still think it’s already useful as a “copilot” or search engine and surely it will only improve over time.
- ilaksh 3y agoI forgot about the continuous pre-training thing. How long and how much did they cost on Bedrock? I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning. Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?
- benjaminwootton 3y agoI used the RAG knowledge bases for most of my testing described above. I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.
- hbamoria 3y agoOh Great! How did you evaluate the LLM responses? I'm cofounder of an evaluation and monitoring platform - Athina AI (www.athina.ai) You can use our monitoring dashboard and evals to check your LLM performance and iterate quickly.
- seedless-sensat 3y agoWhat is RAG? That's hard to search for
- deleted 3y ago[deleted]
- accrual 3y agoRetrieval-augmented generation, RAG + LLM will turn up more results.
- rjzzleep 3y agoThis one seems like a good summary Retrieval-Augmented Generation for Large Language Models: A Survey https://arxiv.org/abs/2312.10997 https://arxiv.org/abs/2312.10997 The photos of this post are also good for a high level look https://twitter.com/dotey/status/1738400607336120573/photo/2 https://twitter.com/dotey/status/1738400607336120573/photo/2 From the various posts I have seen people claim that phi-2 is a good model to start off from. If you just want to do embeddings, there are various tutorials to use pgvector for that.
- nmstoker 3y agoSeems fairly easy to search for to me - top results are all relevant: https://kagi.com/search?q=ml+rag https://kagi.com/search?q=ml+rag https://www.google.com/search?q=ml+rag https://www.google.com/search?q=ml+rag
- tomduncalf 3y agoRetrieval Augmented Generation - in brief, using some kind of search to find relevant documents to the user’s question (often vector DB search, which can search by “meaning”, by also other forms of more traditional search), then injecting those into the prompt to the LLM alongside the question, so it hopefully has facts to refer to (and its “generation” can be “augmented” by documents you’ve “retrieved”, I guess!)
- whartung 3y agoSo, as a contrived example, with RAG you make some queries, in some format, like “Who is Sauron?” And then start feeding in what books he’s mentioned in, paragraphs describing him from Tolkien books, things he has done. Then you start making more specific queries? How old is he, how tall is he, etc. And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions might this paragraph answer”, and then turn around and feed those questions and text back into the system. Is that a 30,000 foot view really of how this works?
- sroecker 3y agoWe just held a workshop about this a few weeks ago: https://red.ht/llmappdev https://red.ht/llmappdev We created a simple chatbot using local models with Ollama (llamacpp), LlamaIndex and streamlit. Have a look at the streamlit folder, it's super easy. I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language SQL querying now.
- 3abiton 3y agoLlamaindex has so mucu potential. Any benchmarks on performance compared to fine-tuning?
- sroecker 3y agoYou probably don't need fine-tuning, at least if it's just new content (and no new instructions). It may even be detrimental, since LLMs are als good at forgetting: https://twitter.com/abacaj/status/1739015011748499772 https://twitter.com/abacaj/status/1739015011748499772
- agilob 3y agolooks very promising, do you plan to keep this single repo up to date as new things are released?
- sroecker 3y agoGood question, as you can see I haven't touched it for a month. I wanted to show what's possible then with open source and (open) local models and there's already so much new stuff out there. I'll probably fix some things this week and then either update it or start from scratch. Guided generation, structured extraction, function calling and multi-modal are things I wanted to add and chainlit looks interesting.
- spacecadet 3y agoOuch your client! I had one earlier this year like this. We were doing some audio processing for word matching, he had also been mislead before coming to us, he fully believed that this was going to be some form of super AI trained on his 5 audio records of him repeating the words over and over... We did all we could to steer him toward a correct path of understanding. Sadly we launched a working product but he doesn't understand it and continues to miss represent and miss sell it. After continuing to give him time and follow up with him (I tend to personally do this with Clients like this), I can tell he is starting to realize his lack of understanding...
- jasonjmcghee 3y agoYou don't just feed documents in, you need to build a dataset representative of how you want to interact with it. So likely using gpt-4 or something to create: a chunk of a document, a question that can be answered by that chunk and a good answer. (Or something)
- stevenhuang 3y ago> What absolutely does not work is trying to just feed a set of documents into fine tuning. Not quite. It does work, albeit likely not optimal. See https://github.com/bublint/ue5-llama-lora https://github.com/bublint/ue5-llama-lora
- ilaksh 3y agoHave you tried that? I find the results hard to believe because he says he is using Llama 7b and asking it questions but that is not a chat model. Also he does not appear at all in the issues when people ask him about reproducing the results. Instead there are people in the issues recommending RAG or creating a QA dataset.
- treprinum 3y agoLlamaIndex can't do chunk-level metadata, only document-level metadata, so you can't put precise references to where materials the LLM synthesized answers from originated, e.g. HTML anchors. Just write your own RAG with Pinecone and OpenAI APIs directly.
- seldo 3y agoLlamaIndex lets you attach metadata to Nodes which are basically chunks, although that fact is poorly documented! Will fix.
- treprinum 3y agoThanks! Even with a better documentation, document importers don't extract node metadata so one needs to write their own "text and metadata extractor" as well. It's then easier to skip LlamaIndex altogether, or just get inspiration from some re-ranking etc. you guys did.
- anjanb 3y agocan you elaborate please
- jncfhnb 3y agoRAG is a funny thing. It’s like going back to Watson for specifics but letting the LLM handle the generic stuff.
- thorum 3y agoHas anyone tried using an LLM for the retrieval stage? Instead of using vector embeddings, have a (small, fast) LLM literally scan the entire corpus in chunks extracting relevant sections?
- spullara 3y agoThat would still be very slow with any reasonably large corpus.
- warthog 3y agoWhy Llamaindex instead of Langchain?
- moltar 3y agoAnother super easy option for RAG is AWS Bedrock Knowledge Base. It can ingest docs from S3. Just don’t use the OpenSearch serverless store it’s $$$. Can use a low end RDS with pgvector extension.
- legendofbrando 3y agoWhat’s the benefit of llamaindex over just storing documents in chroma and using chroma to query? I’ve done the latter and trying to understand if there’s a performance gain to the former?
- sivasurend 3y agoNot much, actually. For lower volumes of documents, vector stores like Chroma or Weaviate provide inbuilt RAG. Things get messy when the number and type of documents increase. Below are the reasons why you may need advanced RAG. 1. Intelligent Data Parser 2. Chunking efficiently 3. Choice of embedding models 4. Query transformation 5. RAG technique 6. Prompt design 7. Feedback loop Check out my blog on the 27 parameters, considerations and techniques one could follow to build a State-of-the-Art Chatbot. https://www.lyzr.ai/27-parameters-techniques-considerations-for-building-a-llm-powered-state-of-the-art-sota-chatbot/ https://www.lyzr.ai/27-parameters-techniques-considerations-... So here is the quick guide. For simpler usecases - inbuilt vector database RAG is sufficient For more complex ones - LlamaIndex or Langchain options are suitable For enterprise grade production use cases - Lyzr's SOTA RAG architecture comes in handy
- vikbehal 3y ago[dead]