3 ms·
Technically the author's description of how Lucene's ranking works is accurate. But the catch is that Lucene's default "vector space model" treats each documen
by teraflop 5y ago
Technically the author's description of how Lucene's ranking works is accurate.
But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly what you want: documents that match more query words end up getting more non-zero terms in their similarity score, which leads to a higher ranking.
- jhgb 5y agoHmm. I'll have to think about it but I don't quite see yet how this ensures that for example all documents containing all four words of a four-word query will be ranked ahead of documents containing three or fewer words of the query. Mere angle between two vectors would for example mean that a "document" consisting of three words of the query could be ranked higher than a document with thousands of words containing all four words of a query, even if the document vectors had tf-idf weighs as components. That clearly is a viable notion of similarity, but that's also clearly not what the type of search alluded to in the article is attempting to do. Plus, your description definitely isn't something I would have been able to divine from the description from the article. Thanks for rephrasing it in comprehensible English.
- jpgvm 5y agoThat is because it doesn't ensure that all documents containing all query terms are ranked above with those with a subset (though it can be tuned such that it does).
- The_rationalist 5y agoisn't BM25 used by default instead of tf-idf?
- huahaiy 5y agoBM25 is just a variant of tf-idf.