2 ms·
Searching the web vs. your own file system in 2022
- PaulHoule 5y agoPeople forget that the search problem depends on the size of the document collection. When Gerard Salton was doing information retrieval in the 1960s and 70s he was storing document vectors on IBM cards and working with 70 document collections. In that case recall is everything; if there is 1 document on the topic and you missed it you're screwed. Often you have to take heroic measures to have any relevant results at all! Contrast that to Google which has "billions and billions" of documents and it hardly matters if they lose 70% of them as long as something high-quality ranks near the top. In the first few years after Google got famous for search quality academic researchers struggled to prove that PageRank helped with web search because it doesn't help if you have 10 million documents or even 100 million documents, it takes "billions and billions". (Combining a document ranking with a query-document ranking is tough, you never want to give somebody an irrelevant document just because it is popular.) The main reason why "search sucks" is that even though there is an accepted methodology for measuring and tuning up the performance of search systems developed for https://en.wikipedia.org/wiki/Text_Retrieval_Conference https://en.wikipedia.org/wiki/Text_Retrieval_Conference hardly anybody does it. There are so many ideas that you think would improve relevance that don't... That conference met for five years before somebody found something that did https://en.wikipedia.org/wiki/Okapi_BM25 https://en.wikipedia.org/wiki/Okapi_BM25 Anyhow I worked for a company where they had me use methods from TREC to tune up a search engine for patents and it was so good that we sold a license to the USPTO for it 72 hours after a demo went live. You can go ahead and be the "enterprise search" company that has "8001 integrations" but your results will suck unless you do what we did. I am still nostalgic for the full-text search for the help of Office '95. As much as people mocked the paper clip, that might be the only application I ever used where the full-text search over the help was useful.