9 ms·
How do domain-specific chatbots work? A retrieval augmented generation overview
- czue 3y agoHello, author here. I wrote this guide up after spending a long time trying to wrap my head around LangChain and the associated ecosystem, and hope others find it useful. Would be happy to answer any questions or feedback anyone has.
- sixhobbits 3y agohey I have wanted to write a post like this for ages but of course I never got around to it, and now I can just link people to this one instead! My only suggestions are: - Coining (? at least I think you did) the new term "Embedding machine" I think loses some clarity versus using "vectorizer", or other existing terminology. - I know that researchers were surprised by how well word embeddings work, but I think you could have built a bit more intuition here instead of just saying "black box" (but maybe article would have gotten too long) - maybe using some of the intuitions from word2vec, even though that is dated now, the idea that 'similar tokens appear in similar contexts' is useful I think.
- czue 3y agoThanks for the input and will very happily accept you sending links to this post. :) My use of "embedding machine" is just revealing my own ignorance. Never heard the term "vectorizer" before, but will switch it (or find a better word). I'll also have to do some reading to educate myself on word2vec. That's this? https://en.wikipedia.org/wiki/Word2vec https://en.wikipedia.org/wiki/Word2vec
- sixhobbits 3y agoyes, the most-quoted 'breakthrough' was figuring out analogies like this https://www.technologyreview.com/2015/09/17/166211/king-man-woman-queen-the-marvelous-mathematics-of-computational-linguistics/ https://www.technologyreview.com/2015/09/17/166211/king-man-...
- sharemywin 3y agoyou might want to check these out too: https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models/ https://www.sbert.net/docs/pretrained_models.html#sentence-e... that way you don't need to pay openai for the embeddings
- spankalee 3y agoRelatedly, to have a useful chatbot you need to track chat history in a way very similar to augmenting with document retrieval, but you may need to generate embeddings and summaries as you go. A friend of mine is working on an OSS memory system for chat apps that helps store, retrieve, summarize chat history, and documents to now, I believe, on top of LangChain: https://www.getzep.com/ https://www.getzep.com/
- andrenotgiant 3y agoGreat explanation. I understand the process a lot better after reading. I have questions now: It seems like everything hinges on the search step, where domain-specific content is gathered to serve as prompt input for the LLM. 1. In an area like technical docs, if I can't find what I need it's often because I don't know the right terms - e.g.: I am looking for "inline subquery" but they're really called "lateral joins" - Would the search step have any likelihood of finding the right context to feed the LLM here? 2. How much value is added by feeding the search results through the LLM with the user's prompt, vs just returning the results directly to the user? 3. Are there good techniques being developed for handling citations in the LLM output? IIRC Google had this in Bard
- sharemywin 3y ago1. it's using "semantic" search. so if the embeddings understand that "inline subquery" and "lateral joins" are more closely related than say cross joins or something then that doc segment will get returned. 2. the use cases for this would be you want shorter answers then a collection of document snippets and or you had follow up questions that might not be in the same document. and/or you prefer conversing with a chatobot over reading lots of extra text. 3. you can attach a source column to the embeddings and snippet in the vector database. Here's an explanation of pgvector using supabase https://supabase.com/docs/guides/database/extensions/pgvector https://supabase.com/docs/guides/database/extensions/pgvecto... it's basically just a special kind of index to use on an embeddings column.
- czue 3y agoGreat answers. The only thing I'll add to number 3 is that you can also get the LLM to cite which source it used within the matching snippets. A simple example with OpenAI functions-style declarations looks something like this: https://gist.github.com/czue/81eea58cb718e67be7c57c224800e304 https://gist.github.com/czue/81eea58cb718e67be7c57c224800e30... If you do this you can pare down the citation from everything that returned from embedding search to what the bot actually used to generate its answer.
- andrenotgiant 3y ago
- sharemywin 3y agoSomething I wondered is why there isn't a language model trained on doing just RAG. my suspicion is the language model could be a lot smaller if it's just regurgitating things from the context above.
- isoprophlex 3y agoSome thoughts: - RAG generally gets you to prototype stage with interesting/demo-able results quickly, however, if your users turn out to submit queries that the embedder finds hard to vectorize (meaning you don't retrieve relevant vectorized chunks of source data) then it quickly becomes painful - It's easy to go overboard with prompt chains that do summarizing/reducing of the fetched results, or pre-processing prompts that help vectorize the query better (see pain point above); always invest in some testing framework & sane test data upfront so you can avoid the classical data science "tweak it until the demo looks good" trap - Don't ever use langchain, it's a baroque shitshow of a tool with a cluttered, inconsistent API written with a bad, inefficient coding style - Paying for bespoke vector databases is probably snake oil and besides the weird pricing only causes pain in the long run when you want to store more than just your embeddings (looking at you 70$+ usd/month Pinecone); postgres with pgvector gets you very far until you hit multiple million documents, and you get all the benefits of a mature, scalable rdbms. Keep your embeddings close to the rest of your data.
- sharemywin 3y agowhat do you use to create your vectorized chunks?
- isoprophlex 3y agoOpenAI's stuff, no experience with local models. I'm curious to hear from people that have compared between those.
- kordlessagain 3y agoCheck out Instructor Large.
- J_Shelby_J 3y agoIt's pretty straight forward. Pre-process -> chunk -> openai embedding endpoint I found langchain not great, so ended up rewritting most of it for my chat bot. https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8299dd0c03d1c5576cdc84fa1377b475ed6ff/app/services/index_service.py#L348 https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8...
- ofermend 3y agoretrieval augmented generation (we call it "grounded generation" at Vectara) is a great way to build GenAI apps with your data. This blog post can be useful: https://vectara.com/a-reference-architecture-for-grounded-generation/ https://vectara.com/a-reference-architecture-for-grounded-ge.... The long and short of it is: building RAG applications seems easy at the start but gets complicated as you go from toy application to scalable enterprise deployment.
- ingridpan 3y agoRAG is great for pulling some additional knowledge, but if you combine it with fine-tuning (i.e., the LLM 'understands' the domain-specific terminology better) it becomes a lot more effective
- monkeydust 3y agoLooking exactly into this, any research on this topic?
- ingridpan 3y agoResearch: https://arxiv.org/abs/2005.11401 https://arxiv.org/abs/2005.11401 An implementation on Hugging Face: https://huggingface.co/docs/transformers/model_doc/rag https://huggingface.co/docs/transformers/model_doc/rag
- monkeydust 3y agoTxs but I meant fine tuning and RAG combined
- Animats 3y agoOK, so this is one of those things where the query first goes to some kind of lookup system, and you get back something which is fed into the LLM as part of the prompt. Is the lookup running locally (not on OpenAI)? Can you look at the output of the lookup to see what hopefully relevant info it is throwing into the LLM? Can you use this with some local open-source LLM system?
- ryoshu 3y agoYou can run it locally depending on the machine (13b models will run on a 32GB Apple Silicon laptop). You can also spin up one of the larger models on Huggingface or similar. If you want to productize it you'll likely need devops/mlops and money.
- goldemerald 3y agoACL had a recent tutorial about state of the art for this topic. https://acl2023-retrieval-lm.github.io/ https://acl2023-retrieval-lm.github.io/ My favorite takeaway was that purely fine tuning your model on your documents (without extra document context during inference) consistently performs worse than using a context added from a datastore.
- atmb4u 3y agoACL tutorial is a treasure trove. Thank you so much for sharing. Do you have any thoughts on how to handle real-time streaming data refreshes in such a system?
- goldemerald 3y agoHire a grad student for a year to write a research paper on it, haha. I've just gotten into the LLM space for work, so I wouldn't know
- captaindiego 3y agoFor those playing with this - If you attach unique identifiers in a consistent way to your documents - you can prompt to cite sources when generating the answer.
- hammeiam 3y agoOne thing I’ve wondered about is how can I best perform query expansion to return vectors that answer my question rather than just looking like my question itself?