3 ms·

Some thoughts: - RAG generally gets you to prototype stage with interesting/demo-able results quickly, however, if your users turn out to submit queries that t

Some thoughts: - RAG generally gets you to prototype stage with interesting/demo-able results quickly, however, if your users turn out to submit queries that the embedder finds hard to vectorize (meaning you don't retrieve relevant vectorized chunks of source data) then it quickly becomes painful - It's easy to go overboard with prompt chains that do summarizing/reducing of the fetched results, or pre-processing prompts that help vectorize the query better (see pain point above); always invest in some testing framework & sane test data upfront so you can avoid the classical data science "tweak it until the demo looks good" trap - Don't ever use langchain, it's a baroque shitshow of a tool with a cluttered, inconsistent API written with a bad, inefficient coding style - Paying for bespoke vector databases is probably snake oil and besides the weird pricing only causes pain in the long run when you want to store more than just your embeddings (looking at you 70$+ usd/month Pinecone); postgres with pgvector gets you very far until you hit multiple million documents, and you get all the benefits of a mature, scalable rdbms. Keep your embeddings close to the rest of your data.

sharemywin 3y ago
what do you use to create your vectorized chunks?
- isoprophlex 3y ago
  OpenAI's stuff, no experience with local models. I'm curious to hear from people that have compared between those.
- kordlessagain 3y ago
  Check out Instructor Large.
- J_Shelby_J 3y ago
  It's pretty straight forward. Pre-process -> chunk -> openai embedding endpoint I found langchain not great, so ended up rewritting most of it for my chat bot. https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8299dd0c03d1c5576cdc84fa1377b475ed6ff/app/services/index_service.py#L348 https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8...
  - sharemywin 3y ago
    Sorry I meant the chunk part. do you do anything special with trying to break the text into manageable chunks. TextSplitter in Langchain just uses some line breaks stuff I guess. --- The default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""] --- https://python.langchain.com/docs/modules/data_connection/document_transformers/ https://python.langchain.com/docs/modules/data_connection/do...
czue 3y ago
I largely agree with this (and especially your first and last points!) I'm intruiged by your second bullet. What have you seen in the way of test frameworks and test data that's been beneficial? I will admit, a lot of my iteration has fallen into the "tweak till the demos look good" camp... On LangChain, I'd say it slightly differently. It is a fantastic tool for getting up and running with something quickly, and for giving a high-level overview of the pattern and components you will need. That said, it is pretty awful for scaling or customizing. I (and many other people I have talked to) benefited greatly from starting a project with LangChain, but then eventually re-wrote ~everything in the stack piece-by-piece, as needed. As I mentioned in the post, the LangChain loaders are the most valuable piece for me, if not for direct use, at least as a starting point of working code to build various integrations.
- isoprophlex 3y ago
  Sounds legit what you mention wrt. Langchain. I agree that it's nice for grokking the patterns and getting something out there. Maybe I worded my post too harshly... Testing; I haven't really found something solid, something that I'd write up in good conscience as a clear 'best practice'. What I meant to say with testing framework is more along the lines of "take some time to set up a bespoke test harness for your use case". For example, a RAG summarizer chatbot-and-PowerPoint generator that is backed by a data store containing 100s of sales pitches. No low latency reqs, but we need some appreciable accuracy. There's plenty of cool ideas to try: do I generate n expanded versions of my user query to help retrieval, get m records back, summarize those in one go? Or do I go for a single query but do I let the LLM filter out false positives? Etc etc. It helps to invest in putting together an (almost) statistically relevant, diverse set of queries and grepping (or even ranking with a LLM) for expected results. In our case, when summarizing sales in the public sector in the past 5 years, did the thing exclude our Y2K work from 23 years back? Simple, diverse test jobs you can use to give some flavor of scientific hypothesis testing to the weirdness of figuring out how to tweak everything inside the pipeline. This also allows you to register succes criteria upfront, eg. we accept this if we get 19/20 right. Then that becomes 190/200 in the next project phase. Then that becomes "less than 5% thumbs down feedback from users when we go live". It's an exciting field. Makes me think about building my own little convnets before pytorch become so good.
eskibars 3y ago
IMHO I think the best points you make here are the last 2. To contrast the first 2, I have seen that many users can provide really good capabilities with RAG -- especially combined with some kind of keyword/hybrid blending so that hyper-specific terms still come up. Of course, it's difficult for most organizations to actually test their results in a meaningful and cost-effective way, which is why we see so much of this "deploy, cross fingers, wait for a call from the CEO" behavior.

sharemywin 3y ago

what do you use to create your vectorized chunks?

isoprophlex 3y ago
OpenAI's stuff, no experience with local models. I'm curious to hear from people that have compared between those.
kordlessagain 3y ago
Check out Instructor Large.
J_Shelby_J 3y ago
It's pretty straight forward. Pre-process -> chunk -> openai embedding endpoint I found langchain not great, so ended up rewritting most of it for my chat bot. https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8299dd0c03d1c5576cdc84fa1377b475ed6ff/app/services/index_service.py#L348 https://github.com/shelby-as-a/shelby-as-a-service/blob/50d8...
- sharemywin 3y ago
  Sorry I meant the chunk part. do you do anything special with trying to break the text into manageable chunks. TextSplitter in Langchain just uses some line breaks stuff I guess. --- The default recommended text splitter is the RecursiveCharacterTextSplitter. This text splitter takes a list of characters. It tries to create chunks based on splitting on the first character, but if any chunks are too large it then moves onto the next character, and so forth. By default the characters it tries to split on are ["\n\n", "\n", " ", ""] --- https://python.langchain.com/docs/modules/data_connection/document_transformers/ https://python.langchain.com/docs/modules/data_connection/do...

czue 3y ago

I largely agree with this (and especially your first and last points!) I'm intruiged by your second bullet. What have you seen in the way of test frameworks and test data that's been beneficial? I will admit, a lot of my iteration has fallen into the "tweak till the demos look good" camp... On LangChain, I'd say it slightly differently. It is a fantastic tool for getting up and running with something quickly, and for giving a high-level overview of the pattern and components you will need. That said, it is pretty awful for scaling or customizing. I (and many other people I have talked to) benefited greatly from starting a project with LangChain, but then eventually re-wrote ~everything in the stack piece-by-piece, as needed. As I mentioned in the post, the LangChain loaders are the most valuable piece for me, if not for direct use, at least as a starting point of working code to build various integrations.

isoprophlex 3y ago
Sounds legit what you mention wrt. Langchain. I agree that it's nice for grokking the patterns and getting something out there. Maybe I worded my post too harshly... Testing; I haven't really found something solid, something that I'd write up in good conscience as a clear 'best practice'. What I meant to say with testing framework is more along the lines of "take some time to set up a bespoke test harness for your use case". For example, a RAG summarizer chatbot-and-PowerPoint generator that is backed by a data store containing 100s of sales pitches. No low latency reqs, but we need some appreciable accuracy. There's plenty of cool ideas to try: do I generate n expanded versions of my user query to help retrieval, get m records back, summarize those in one go? Or do I go for a single query but do I let the LLM filter out false positives? Etc etc. It helps to invest in putting together an (almost) statistically relevant, diverse set of queries and grepping (or even ranking with a LLM) for expected results. In our case, when summarizing sales in the public sector in the past 5 years, did the thing exclude our Y2K work from 23 years back? Simple, diverse test jobs you can use to give some flavor of scientific hypothesis testing to the weirdness of figuring out how to tweak everything inside the pipeline. This also allows you to register succes criteria upfront, eg. we accept this if we get 19/20 right. Then that becomes 190/200 in the next project phase. Then that becomes "less than 5% thumbs down feedback from users when we go live". It's an exciting field. Makes me think about building my own little convnets before pytorch become so good.

eskibars 3y ago

IMHO I think the best points you make here are the last 2. To contrast the first 2, I have seen that many users can provide really good capabilities with RAG -- especially combined with some kind of keyword/hybrid blending so that hyper-specific terms still come up. Of course, it's difficult for most organizations to actually test their results in a meaningful and cost-effective way, which is why we see so much of this "deploy, cross fingers, wait for a call from the CEO" behavior.