3 ms·

I’ve been playing around with this and have gotten decent results simply using openAI embeddings. I’ll probably be making some kind of post showing results soon

simonw 4y ago
Yeah I'm happy with the results I got from embeddings so far. The areas I want to explore there are: 1. What's the ideal size of text to embed? I'm doing whole blog entries right now but I'm confident I can get better results if I divide them up into smaller chunks first - I'm just not sure how best to do that. 2. There's a trick called Hypothetical Document Embeddings (HyDE) where you ask GPT-3 to invent an answer to the user's question, embed THAT fictional answer, then use that embedding to find relevant documents in your corpus. https://arxiv.org/abs/2212.10496 https://arxiv.org/abs/2212.10496
- jerpint 4y ago
  Wow thanks for sharing that’s such a clever hack
- james-revisoai 4y ago
  For 1. I think you may find work with semantic text chunking interesting, and the possibility of overlaying embeddings with one another (e.g. powering fragen.co.uk youtube search, the transcriptions whenever you click the "play" icon are semantically chunked...). Audacity has a good Autochapter API for this too. I found after good OCR postprocessing, semantic splitting is most important. For 2 - have you used this or found it to work? I found asymmetric embeddings like MS MARCO trained models much superior when you have a reasonably bespoke corpus (like the demos of yours I've seen) - because, TLDR, it had a negative impact on recall, even as it improved Precision - if your GPT hypothetical answer is a misconception or joke answer, you'll be semantic searching joke answers, etc - same with numbers, GPT-3 might suggest short or plurality-default answer, like US phone numbers for "what is head office of marketings number?" when you have UK numbers in your corpus you would prefer to be surfaced.
- minxomat 4y ago
  I’ve done some experiments and the ideal seems to be semantic chunks of max 175 tokens. But only if those chunks are already very dense (use summarisation to get there)

simonw 4y ago

Yeah I'm happy with the results I got from embeddings so far. The areas I want to explore there are: 1. What's the ideal size of text to embed? I'm doing whole blog entries right now but I'm confident I can get better results if I divide them up into smaller chunks first - I'm just not sure how best to do that. 2. There's a trick called Hypothetical Document Embeddings (HyDE) where you ask GPT-3 to invent an answer to the user's question, embed THAT fictional answer, then use that embedding to find relevant documents in your corpus. https://arxiv.org/abs/2212.10496 https://arxiv.org/abs/2212.10496

jerpint 4y ago
Wow thanks for sharing that’s such a clever hack
james-revisoai 4y ago
For 1. I think you may find work with semantic text chunking interesting, and the possibility of overlaying embeddings with one another (e.g. powering fragen.co.uk youtube search, the transcriptions whenever you click the "play" icon are semantically chunked...). Audacity has a good Autochapter API for this too. I found after good OCR postprocessing, semantic splitting is most important. For 2 - have you used this or found it to work? I found asymmetric embeddings like MS MARCO trained models much superior when you have a reasonably bespoke corpus (like the demos of yours I've seen) - because, TLDR, it had a negative impact on recall, even as it improved Precision - if your GPT hypothetical answer is a misconception or joke answer, you'll be semantic searching joke answers, etc - same with numbers, GPT-3 might suggest short or plurality-default answer, like US phone numbers for "what is head office of marketings number?" when you have UK numbers in your corpus you would prefer to be surfaced.
minxomat 4y ago
I’ve done some experiments and the ideal seems to be semantic chunks of max 175 tokens. But only if those chunks are already very dense (use summarisation to get there)