4 ms·
> Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers).
by jhgb 5y ago
> Most search engines use something called a vector space model, where both user queries and documents are reduced to vectors (i.e. a fixed number of numbers). That is to say, the search engines are not looking at the meanings of the queries or the documents. Instead, they turned them both into some numbers. The search problem, is reduced to a problem of finding the similarity between the numbers representing the query and the numbers representing the documents.
I don't see how this is true for concordance-like searches (Trados-like, which seems to be what the article is concerned with - preferring the documents with all query words present in it, otherwise the documents with the most query words present in it, etc.). This sounds more like description of similarity useful for metric space indices or something like that.
- teraflop 5y agoTechnically the author's description of how Lucene's ranking works is accurate. But the catch is that Lucene's default "vector space model" treats each document as a very high-dimensional vector, where the components are the words in the document (scaled using tf-idf). Under this definition, "similarity" reduces to a normalized dot-product of the query vector and the document vector. So in practice, it does roughly what you want: documents that match more query words end up getting more non-zero terms in their similarity score, which leads to a higher ranking.
- jhgb 5y agoHmm. I'll have to think about it but I don't quite see yet how this ensures that for example all documents containing all four words of a four-word query will be ranked ahead of documents containing three or fewer words of the query. Mere angle between two vectors would for example mean that a "document" consisting of three words of the query could be ranked higher than a document with thousands of words containing all four words of a query, even if the document vectors had tf-idf weighs as components. That clearly is a viable notion of similarity, but that's also clearly not what the type of search alluded to in the article is attempting to do. Plus, your description definitely isn't something I would have been able to divine from the description from the article. Thanks for rephrasing it in comprehensible English.
- jpgvm 5y agoThat is because it doesn't ensure that all documents containing all query terms are ranked above with those with a subset (though it can be tuned such that it does).
- The_rationalist 5y agoisn't BM25 used by default instead of tf-idf?
- huahaiy 5y agoBM25 is just a variant of tf-idf.