3 ms·
Whatever you do, whichever ML model you develop, a query-independent ordering of all documents will always be necessary since all distributed IR systems will ov
by utopcell 4y ago
Whatever you do, whichever ML model you develop, a query-independent ordering of all documents will always be necessary since all distributed IR systems will over-retrieve according to some form of term matching and then apply sophisticated scoring to extract the best documents. You can't score trillions of docs for every query.
Google still uses PageRank, but at the risk of stating the obvious, the current PageRank is much more sophisticated than the one found in the bibliography.
- marcosdumay 4y ago> You can't score trillions of docs for every query. That completely depends on how you model the queries. It can be a TB sized relation all the way into needing more bits than there are atoms on the universe.
- utopcell 4y agoThis is not a generic discussion, we are discussing web search IR in particular, but if you want to be pedantic, the TB sized dataset could be the PageRank values and "scoring" could be the ordering of these values.
- marcosdumay 4y agoThe TB dataset is PageRank indexed by search term. You can correlate them more and more and end up with exponentially more data, but even the smallest one (that you'll need a hack of servers to query) is quite useful already.