3 ms·

Is anyone having a really good experience using embedding based semantic retrieval in combination with a downstream LLM ? I am working quite a bit with normal

Is anyone having a really good experience using embedding based semantic retrieval in combination with a downstream LLM ? I am working quite a bit with normal and chained LLMs but so far haven't explored the retrieval route.

itake 3y ago
can you provide an example?
- raoufchebri 3y ago
  Hey @lysecret, what's your use-case?
azmodeus 3y ago
I have enjoyed using haystack for multi doc qa
jonathan-adly 3y ago
Yeb - we use pgvector in production[0]. Great results. There are some secret sauce stuff, but basically - depending on your source material and your user use-case you have to do more than just call cosine similarity and call it a day. (Most people don't even know what's happening, just using langchain which has different implementations, but all essentially call cosine similarity on raw text). In my opinion, until LLMs learn to say I don't know, this is the way to do it in accuracy-first tasks. 0. https://galenai.co https://galenai.co
- raoufchebri 3y ago
  @jonathan-adly do you use ivfflat with pgvector?
- chrisrickard 3y ago
  Out of curiosity, how big are you embedding chunks?
vikp 3y ago
Yes, I use retrieval for Endless Academy [1] , and it works well. Some tips: - Most vector search is basically kNN under the hood, with some kind of compression. If you have too many embeddings in your DB, this starts to pull up irrelevant text very quickly. The key is to segment the DB using other data before doing the embedding search. Postgres extensions are good for this. - The quality of the data you put into your embedding DB matters a lot. - How you chunk text matters. Chunking by paragraph is much better than naive chunking, for example. - This is a good benchmark for embedding models [2] [1] https://www.endless.academy https://www.endless.academy [2] https://huggingface.co/blog/mteb https://huggingface.co/blog/mteb
- raoufchebri 3y ago
  Adding https://ann-benchmarks.com/ https://ann-benchmarks.com/
- chrisrickard 3y ago
  > The key is to segment the DB using other data before doing the embedding search By segmenting data, is this as simple as adding a further condition to the SQL? E.g. if your embeddings belonged to a "client" you would start with WHERE client_id - x? > How you chunk text matters. Chunking by paragraph is much better than naive chunking Would love some more info on this. SO if you had a 400 word slab of text, it would be better to create m3 or 4 embeddings from that?
alach11 3y ago
Yes. I work in the oil and gas industry. When our employees do their pre-job safety meeting, we have an app that makes tailored recommendations about job hazards. The steps are: 1. User types in the planned work (e.g., "replace discharge line on water pump") 2. The app embeds that and look for similar historical work in a vector database 3. We retrieve the most similar historical work and the safety observations that were made on those jobs 4. We summarize the major hazards or mistakes from those previous jobs using gpt-3.5-turbo and deliver that to the user
victorbjorklund 3y ago
I used pg_vector + Open AI api. Worked great!

itake 3y ago

can you provide an example?

raoufchebri 3y ago
Hey @lysecret, what's your use-case?

azmodeus 3y ago

I have enjoyed using haystack for multi doc qa

jonathan-adly 3y ago

Yeb - we use pgvector in production[0]. Great results. There are some secret sauce stuff, but basically - depending on your source material and your user use-case you have to do more than just call cosine similarity and call it a day. (Most people don't even know what's happening, just using langchain which has different implementations, but all essentially call cosine similarity on raw text). In my opinion, until LLMs learn to say I don't know, this is the way to do it in accuracy-first tasks. 0. https://galenai.co https://galenai.co

raoufchebri 3y ago
@jonathan-adly do you use ivfflat with pgvector?
chrisrickard 3y ago
Out of curiosity, how big are you embedding chunks?

vikp 3y ago

Yes, I use retrieval for Endless Academy [1] , and it works well. Some tips: - Most vector search is basically kNN under the hood, with some kind of compression. If you have too many embeddings in your DB, this starts to pull up irrelevant text very quickly. The key is to segment the DB using other data before doing the embedding search. Postgres extensions are good for this. - The quality of the data you put into your embedding DB matters a lot. - How you chunk text matters. Chunking by paragraph is much better than naive chunking, for example. - This is a good benchmark for embedding models [2] [1] https://www.endless.academy https://www.endless.academy [2] https://huggingface.co/blog/mteb https://huggingface.co/blog/mteb

raoufchebri 3y ago
Adding https://ann-benchmarks.com/ https://ann-benchmarks.com/
chrisrickard 3y ago
> The key is to segment the DB using other data before doing the embedding search By segmenting data, is this as simple as adding a further condition to the SQL? E.g. if your embeddings belonged to a "client" you would start with WHERE client_id - x? > How you chunk text matters. Chunking by paragraph is much better than naive chunking Would love some more info on this. SO if you had a 400 word slab of text, it would be better to create m3 or 4 embeddings from that?

alach11 3y ago

Yes. I work in the oil and gas industry. When our employees do their pre-job safety meeting, we have an app that makes tailored recommendations about job hazards. The steps are: 1. User types in the planned work (e.g., "replace discharge line on water pump") 2. The app embeds that and look for similar historical work in a vector database 3. We retrieve the most similar historical work and the safety observations that were made on those jobs 4. We summarize the major hazards or mistakes from those previous jobs using gpt-3.5-turbo and deliver that to the user

victorbjorklund 3y ago

I used pg_vector + Open AI api. Worked great!