4 ms·

This is super helpful. I'm building a document question-answering service over a custom data corpus (related to Saivism, a sect of Hinduism). So far the first p

This is super helpful. I'm building a document question-answering service over a custom data corpus (related to Saivism, a sect of Hinduism). So far the first pass has been to manually chunk the text (based on headings, chapters etc.) and then I've used OpenAI's embedding service and storing the embeddings in Pinecone. All stiched together using LangChain. To ask a question, the question is again embedded, then searched against the vector store, then the related documents are provided as context to the LLM along with the question. So far it was really easy to set up the prototype, but the results weren't as great as I had hoped, so I'm excited to see how I could improve it. Edit: wow, I didn't see this before. LangChain implements one of the featured article's suggestions (HyDE) - https://python.langchain.com/en/latest/modules/chains/index_examples/hyde.html https://python.langchain.com/en/latest/modules/chains/index_...

kaycebasques 4y ago
This is the same approach that Supabase Clippy took: https://supabase.com/blog/chatgpt-supabase-docs https://supabase.com/blog/chatgpt-supabase-docs They called it "context injection" but the OpenAI community appears to call it "retrieval-augmented generation". Edit: I see now that Matt's post is talking all about these ideas. I'm happy that the terminology around this space is solidifying. (Tangent) I will go to the grave continuing to call it Supabase Clippy even though presumably this prediction from the Supabase blog post became true: > Today, we're doing our part to support the momentum by releasing “Supabase Clippy” for our docs (and we don't expect this name to last long before the lawyers catch on).
- kiwicopple 4y ago
  The lawyers haven’t reached out yet! Too distracted with all the LLM-generated emails they are getting sent no doubt
  - triyambakam 4y ago
    Why call it Clippy at all if you anticipate a conflict? It feels immature.
    - kiwicopple 4y ago
      yes, it's immature, however in January not many companies had created something like this and it was the most analogous way of explaining the functionality. it's also memorable
freezed88 4y ago
depending on what questions you're asking, you could check out LlamaIndex query capabilities - can define different index structures for different queries + can plugin to your langchain workflow: https://gpt-index.readthedocs.io/en/latest/use_cases/queries.html https://gpt-index.readthedocs.io/en/latest/use_cases/queries...

kaycebasques 4y ago

This is the same approach that Supabase Clippy took: https://supabase.com/blog/chatgpt-supabase-docs https://supabase.com/blog/chatgpt-supabase-docs They called it "context injection" but the OpenAI community appears to call it "retrieval-augmented generation". Edit: I see now that Matt's post is talking all about these ideas. I'm happy that the terminology around this space is solidifying. (Tangent) I will go to the grave continuing to call it Supabase Clippy even though presumably this prediction from the Supabase blog post became true: > Today, we're doing our part to support the momentum by releasing “Supabase Clippy” for our docs (and we don't expect this name to last long before the lawyers catch on).

kiwicopple 4y ago
The lawyers haven’t reached out yet! Too distracted with all the LLM-generated emails they are getting sent no doubt
- triyambakam 4y ago
  Why call it Clippy at all if you anticipate a conflict? It feels immature.
  - kiwicopple 4y ago
    yes, it's immature, however in January not many companies had created something like this and it was the most analogous way of explaining the functionality. it's also memorable

freezed88 4y ago

depending on what questions you're asking, you could check out LlamaIndex query capabilities - can define different index structures for different queries + can plugin to your langchain workflow: https://gpt-index.readthedocs.io/en/latest/use_cases/queries.html https://gpt-index.readthedocs.io/en/latest/use_cases/queries...