3 ms·
Embeddings are good at capturing surface level information but can't match implicit/deeper/conclusion level information. Say you have a collection of 100,000 ma
by visarga 3y ago
Embeddings are good at capturing surface level information but can't match implicit/deeper/conclusion level information. Say you have a collection of 100,000 math problems, and you want to embed them to search problems that give result "0". Any number of problems can give this result and it is not explicit in the problem statement. But if you solve the problems you can see the data was in there, just not apparent.
In general you can see the raw text as a simulation premise that will generate inferences when "executed". The inferenced part is like the hidden part of the iceberg, you don't see it but it is there, implicit in the source text. Not just in math, but in all fields.
Embeddings are only good at superficial retrieval. The text needs to be fully analyzed with LLMs before embedding. Thus my conclusion is that we still have a long way to go, we haven't peaked.
- deleted 3y ago[deleted]
- wodenokoto 3y agoWhat do you mean by fully analyzed? It’s the LLM that does the embedding.
- visarga 3y agoOh the embedding LLMs are usually lightweight BERT models with few layers and <<1B weights, while LLMs are easily 10-100x larger. The idea is to ingest the text in a LLM to extract the facets you are going to search and add those extra tokens to the original text. Then you do regular RAG.
- sroussey 3y agoHow?
- wodenokoto 3y agoYou want to enrich the base text before embedding? With what kind of prompt? It’s a pretty simple thing to add to a pipeline. Have you tried?
- sroussey 3y agoWhat are your actionable suggestions? I am currently testing embeddings/RAG and could use some insight on how to make the results better.
- visarga 3y ago> The text needs to be fully analyzed with LLMs before embedding. If you happen to know what kinds of questions you will be asking about your RAG index, you should pre-process the texts to add QA pairs. Otherwise you can prompt the LLM to do chain-of-thought inferences based on the source text and add them to the material.
- sroussey 3y agoI guess you log queries to see what is popular and then reprocess texts based on those?
- sroussey 3y agoAside from a feedback loop from usage, is there a way to guess? I guess you put put a whole doc into the I’ll and ask what questions it answers? And then use those question plus a piece of the text and do an embedding?