4 ms·

Great explanation. I understand the process a lot better after reading. I have questions now: It seems like everything hinges on the search step, where domain-

Great explanation. I understand the process a lot better after reading. I have questions now: It seems like everything hinges on the search step, where domain-specific content is gathered to serve as prompt input for the LLM. 1. In an area like technical docs, if I can't find what I need it's often because I don't know the right terms - e.g.: I am looking for "inline subquery" but they're really called "lateral joins" - Would the search step have any likelihood of finding the right context to feed the LLM here? 2. How much value is added by feeding the search results through the LLM with the user's prompt, vs just returning the results directly to the user? 3. Are there good techniques being developed for handling citations in the LLM output? IIRC Google had this in Bard

sharemywin 3y ago
1. it's using "semantic" search. so if the embeddings understand that "inline subquery" and "lateral joins" are more closely related than say cross joins or something then that doc segment will get returned. 2. the use cases for this would be you want shorter answers then a collection of document snippets and or you had follow up questions that might not be in the same document. and/or you prefer conversing with a chatobot over reading lots of extra text. 3. you can attach a source column to the embeddings and snippet in the vector database. Here's an explanation of pgvector using supabase https://supabase.com/docs/guides/database/extensions/pgvector https://supabase.com/docs/guides/database/extensions/pgvecto... it's basically just a special kind of index to use on an embeddings column.
- czue 3y ago
  Great answers. The only thing I'll add to number 3 is that you can also get the LLM to cite which source it used within the matching snippets. A simple example with OpenAI functions-style declarations looks something like this: https://gist.github.com/czue/81eea58cb718e67be7c57c224800e304 https://gist.github.com/czue/81eea58cb718e67be7c57c224800e30... If you do this you can pare down the citation from everything that returned from embedding search to what the bot actually used to generate its answer.
- andrenotgiant 3y ago
  Is there something (besides the convenience of being already in the proper vector formats) unique about the combination of semantic search and LLM's that makes it a better fit? I guess I'm surprised that the approach is to use semantic search here, where in regular search, completely different algorithms (Lucene?) have "won".
  - lmeyerov 3y ago
    Keyword search won for commodity bc semantic search was weaker and more complicated. More sophisticated orgs like Google switched to semantic/hybrid for years as both problems got addressed. Now that opensearch/elasticsearch have vector indexes, and high quality text embedding is a lot easier...
- petra 3y ago
  Is the search in such systems simple and intuitive, or are there tricks to get better results from such systems ,as a user?
vharuck 3y ago
>2. How much value is added by feeding the search results through the LLM with the user's prompt, vs just returning the results directly to the user? These systems usually retrieve the "closest N" snippets. Which might include irrelevant ones if there aren't enough close matches. The LLM will favor the relevant info when composing an answer. >3. Are there good techniques being developed for handling citations in the LLM output? IIRC Google had this in Bard Like sharemywin said, you can include citation info with the snippets when making the prompt. In a proof-of-concept I did at work, I was having trouble forcing the gpt-3.5-turbo model to include URL citations. What eventually worked was adding an extra snippet to the list of relevant ones: a few sentences from the Wikipedia page for Lorem Ipsum plus its URL (arbitrary choice, just had to not overlap with any topic from my documents). Then the prompt included an example question about Lorem Ipsum, with an example answer citing the URL.

sharemywin 3y ago

1. it's using "semantic" search. so if the embeddings understand that "inline subquery" and "lateral joins" are more closely related than say cross joins or something then that doc segment will get returned. 2. the use cases for this would be you want shorter answers then a collection of document snippets and or you had follow up questions that might not be in the same document. and/or you prefer conversing with a chatobot over reading lots of extra text. 3. you can attach a source column to the embeddings and snippet in the vector database. Here's an explanation of pgvector using supabase https://supabase.com/docs/guides/database/extensions/pgvector https://supabase.com/docs/guides/database/extensions/pgvecto... it's basically just a special kind of index to use on an embeddings column.

czue 3y ago
Great answers. The only thing I'll add to number 3 is that you can also get the LLM to cite which source it used within the matching snippets. A simple example with OpenAI functions-style declarations looks something like this: https://gist.github.com/czue/81eea58cb718e67be7c57c224800e304 https://gist.github.com/czue/81eea58cb718e67be7c57c224800e30... If you do this you can pare down the citation from everything that returned from embedding search to what the bot actually used to generate its answer.
andrenotgiant 3y ago
Is there something (besides the convenience of being already in the proper vector formats) unique about the combination of semantic search and LLM's that makes it a better fit? I guess I'm surprised that the approach is to use semantic search here, where in regular search, completely different algorithms (Lucene?) have "won".
- lmeyerov 3y ago
  Keyword search won for commodity bc semantic search was weaker and more complicated. More sophisticated orgs like Google switched to semantic/hybrid for years as both problems got addressed. Now that opensearch/elasticsearch have vector indexes, and high quality text embedding is a lot easier...
petra 3y ago
Is the search in such systems simple and intuitive, or are there tricks to get better results from such systems ,as a user?

vharuck 3y ago

>2. How much value is added by feeding the search results through the LLM with the user's prompt, vs just returning the results directly to the user? These systems usually retrieve the "closest N" snippets. Which might include irrelevant ones if there aren't enough close matches. The LLM will favor the relevant info when composing an answer. >3. Are there good techniques being developed for handling citations in the LLM output? IIRC Google had this in Bard Like sharemywin said, you can include citation info with the snippets when making the prompt. In a proof-of-concept I did at work, I was having trouble forcing the gpt-3.5-turbo model to include URL citations. What eventually worked was adding an extra snippet to the list of relevant ones: a few sentences from the Wikipedia page for Lorem Ipsum plus its URL (arbitrary choice, just had to not overlap with any topic from my documents). Then the prompt included an example question about Lorem Ipsum, with an example answer citing the URL.