3 ms·
Speaking of which, could we apply vector embeddings to search engines (where crawled pages get indexed by their vector embeddings rather than raw text) and use
by another-dave 1y ago
Speaking of which, could we apply vector embeddings to search engines (where crawled pages get indexed by their vector embeddings rather than raw text) and use that for better fuzzy search results even without an LLM in the mix?
(Might be a naïve question, I'm at the edge of my understanding)
- esafak 1y agoWhy stop there? The LLM can synthesize the results and spare you the work.
- another-dave 1y agoI'm talking about the scenario the GP referenced — where if you search for say "holiday" but get no results because the pages only use the word "vacation" which AFAIK is still a problem in regular search. LLMs inherently would introduce the possibility of hallucinations, but just using the vectors to match documents wouldn't, right?
- esafak 1y agoNo, llms still use similarity search for candidate generation, unless you don't give them any tools.
- com2kid 1y ago> Speaking of which, could we apply vector embeddings to search engines (where crawled pages get indexed by their vector embeddings rather than raw text) and use that for better fuzzy search results even without an LLM in the mix? Yes, this is how all the new dev documentation sites work now days, with their much improved searches. :-D
- another-dave 1y agoah cool right! I didn't know that. One for me to check out and understand more. Thanks!