5 ms·
Depends on the nature of the content you’re working with, but I’ve had some good results using an LLM during indexing to generate a search document by rephrasin
by macNchz 2y ago
Depends on the nature of the content you’re working with, but I’ve had some good results using an LLM during indexing to generate a search document by rephrasing the original text in a standardized way. Then you can search against the embeddings of that document, and perhaps boost based on keyword similarity to the original text.
- joerick 2y agoNice workaround. I just wish there was a less 'lossy' way to go about it!
- jacobr1 2y agoCould you explicitly train a set of embeddings that performed that step in the process? For example which computing the loss, you compare the difference against the normalized text rather than the original. Or alternatively do this as a fine-tuning. Then you would have embedding that optimized for the characteristics you care about.
- hobs 2y agoNormal full text search stuff helps reduce the search space - eg lemming, stemming, query simplification stuff were all way before LLMs.
- mrshu 2y agoThis is also often referred to as Hypothetical Document Embeddings (https://arxiv.org/abs/2212.10496 https://arxiv.org/abs/2212.10496).
- adamgordonbell 2y agoDo you have examples of this? Please say more!