4 ms·

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way one might phrase a question might be very different content wise from how the document describes the answer. Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I haven't found much info on that yet. Another idea I've had is to "overfit" a generative model like GPT on a dataset but pay more attention to how url and the like are tokenised

nunodonato 4y ago
embeddings are really good at that, you dont need to use similar words at all.
IanCal 4y ago
> Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I haven't found much info on that yet. Here you go https://twitter.com/theseamouse/status/1614453236349693953 https://twitter.com/theseamouse/status/1614453236349693953
summarity 4y ago
> might phrase a question might be very different content wise from how the document describes the answer That's what hypothetical embeddings solve: https://summarity.com/hyde https://summarity.com/hyde There are also encoding schemes for question-answer retrieval (e.g. ColBERT)
sharemywin 4y ago
is there an example what what your talking about? Also would you just return a list of likely candidates and loop over the result set to see if any info is relevant to the question and then have the the final pass try to answer the question.
visarga 4y ago
> The way one might phrase a question might be very different content wise from how the document describes the answer. If the embeddings are worth their salt, then they should not be influenced by paraphrasing with different words. Try the OpenAI embeddings or sbert.net embedding models.
jncraton 4y ago
Embeddings can be trained specifically to cause questions and content including their answers to have similar representations in latent space. This has been used this to create QA retrieval systems. Here's one commonly used example: https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1 https://huggingface.co/sentence-transformers/multi-qa-MiniLM...
amitport 4y ago
"The way one might phrase a question might be very different content wise from how the document describes the answer." You have late-interaction models, which replace the dot product with a few transformer layers and are able to learn complex semantics. Of course this would adversely affect latency and embedding size, so you might want to compress and cache the answers, hence (shameless plug): https://aclanthology.org/2022.acl-long.457/ https://aclanthology.org/2022.acl-long.457/
pstorm 4y ago
In your first paragraph, you are describing Hypothetical Document Embeddings (HyDE) [0]. I've tested it out, and in certain cases, it works amazingly well to get more complete answers. [0] https://python.langchain.com/en/latest/modules/chains/index_examples/hyde.html https://python.langchain.com/en/latest/modules/chains/index_...
yadaeno 4y ago
How many embeddings can fit into a single input?

nunodonato 4y ago

embeddings are really good at that, you dont need to use similar words at all.

IanCal 4y ago

> Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I haven't found much info on that yet. Here you go https://twitter.com/theseamouse/status/1614453236349693953 https://twitter.com/theseamouse/status/1614453236349693953

summarity 4y ago

> might phrase a question might be very different content wise from how the document describes the answer That's what hypothetical embeddings solve: https://summarity.com/hyde https://summarity.com/hyde There are also encoding schemes for question-answer retrieval (e.g. ColBERT)

sharemywin 4y ago

is there an example what what your talking about? Also would you just return a list of likely candidates and loop over the result set to see if any info is relevant to the question and then have the the final pass try to answer the question.

visarga 4y ago

> The way one might phrase a question might be very different content wise from how the document describes the answer. If the embeddings are worth their salt, then they should not be influenced by paraphrasing with different words. Try the OpenAI embeddings or sbert.net embedding models.

jncraton 4y ago

Embeddings can be trained specifically to cause questions and content including their answers to have similar representations in latent space. This has been used this to create QA retrieval systems. Here's one commonly used example: https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1 https://huggingface.co/sentence-transformers/multi-qa-MiniLM...

amitport 4y ago

"The way one might phrase a question might be very different content wise from how the document describes the answer." You have late-interaction models, which replace the dot product with a few transformer layers and are able to learn complex semantics. Of course this would adversely affect latency and embedding size, so you might want to compress and cache the answers, hence (shameless plug): https://aclanthology.org/2022.acl-long.457/ https://aclanthology.org/2022.acl-long.457/

pstorm 4y ago

In your first paragraph, you are describing Hypothetical Document Embeddings (HyDE) [0]. I've tested it out, and in certain cases, it works amazingly well to get more complete answers. [0] https://python.langchain.com/en/latest/modules/chains/index_examples/hyde.html https://python.langchain.com/en/latest/modules/chains/index_...

yadaeno 4y ago

How many embeddings can fit into a single input?