31 ms·
"Search" is too broad to ever be solved. That's like "solving entropy". Google focused on a specific subset — you enter a few keywords or a phrase, and the mac
by Radim 8y ago
"Search" is too broad to ever be solved. That's like "solving entropy".
Google focused on a specific subset — you enter a few keywords or a phrase, and the machine returns the top ~10 links to pre-existing (indexed) web pages. But that's not all there is to search!
Challenges:
1. Intranets: internal documents, typically in different modalities (FAQs, support cases, wikis, public pages) and across diverse storages that evolved throughout the years via acquisitions and osmosis.
2. Clustering: you don't have any keywords, but rather want to find how a particular document (legal template, its clause section) evolved over time. You want to avoid using keywords. Search for similar documents or document sections. Find similarity between two documents that is based on semantics rather than query keywords. Applications: eDiscovery, contract management…
3. SME & Intent: "relevant result" means different things in different domains, or even different aspects of a single domain. Google is doing an amazing job with their "single search box", but there are industries (for example, HR) where search precision matters much more than recall. More elaborate, focused, domain-specific facets or even dialogue systems make sense there.
Commercial plug: we built a search solution focused around semantic search (in the "machine learning and vectors" sense, not "sematic web and RDFs" sense), https://scaletext.ai https://scaletext.ai. It's still early days in that our clients are all over the place, but to say Google/Lucene solved search is patently false.
- halflings 8y agoDefinitely valid point (that there are very different types of "search", not just one universal way to do things), but: 2) Google does not do simple keyword matching, and certainly has a strong sense of "semantics". 3) Try searching "Cheap hotels in San Francisco" or "Plumber jobs in Chicago" in Google. Just because it's a single search box does not mean all results are generated/displayed in the same way.
- Radim 8y agoAbsolutely. Determining query intent is an ancient, well researched and still active domain. And Google publishes their results regularly (kudos to them). My point was more that encoding the relevant signals into a single general-purpose search box (a sort-of natural language) is an inherently noisy, ambiguous process. When you know what kind of search you want, it's better to factor out the relevant parameters and feedback loops explicitly, give them a clear UI and search flow. Rather than have users fumble with double quotes and double-guessing the query parser.
- bobosha 8y ago@Radim - is this something you are doing with your current startup? p.s. kudos from another fan of gensim.
- tephra 8y agoAnother fun one that I encountered in an NLP class to many years ago now is multilingual search. Search in one language, get results in multiple languages. The context in which this came up was in organisations such as the EU or the Nordic Council. Another one that I would like to see improved is better search for scientific articles.
- dev1n 8y agoJust want to say thank you for gensim [1]. [1]: https://radimrehurek.com/gensim/ https://radimrehurek.com/gensim/