8 ms·
If you have access to the query log, aka "who makes which query in what context"), you can use see which queries are "close" to others in context. For example,
by cdavid 2y ago
If you have access to the query log, aka "who makes which query in what context"), you can use see which queries are "close" to others in context.
For example, with session, you can detect manual query rewriting, and use this as a signal to see which queries are close to others in the time context. You can do various fancy things from just that.
Nowadays, a simple way to start would be to use SOTA LLMs to generate synonyms offline, and use this for query expansion at query time. At least in a context where queries are small, that should give decent results. This has however diminishing returns because of cost (the more synonyms the more expensive querying the index), and also you lose precision with diminishing returns on increased recall.
Ofc, for complex search like google, I am sure it is much more complicated
- bms2297 2y agoRe: LLMs, I was trying to better understand how pre-LLM search worked, hence the interest in the topic. Any chance you have any open source links that discuss how you practically operate a system based on the concept you describe (manual query rewrite w/i a session as your data set)? Perhaps it's obvious to an NLP person how to reduce that "idea" to practice, but it is not to me! You're definitely right about the idea though - a former Search engineer obliquely mentioned that this sort of session based manual query rewriting was very core to how the synonym system worked.
- cdavid 2y agoIt is hard to find modern references on this. When I led a search group, coming from an ML but non search background, I found the following most useful 1. Query understanding series: https://queryunderstanding.com/query-understanding-8a2b16024c5b https://queryunderstanding.com/query-understanding-8a2b16024... 2. Deep learning for search in manning. It covers some non DL techniques Then it is mostly papers and talking to people. Berlin buzzword has videos of all talks and is very non academic but technical.