3 ms·
>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the d
by bogomipz 8y ago
>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data."
Can you elaborate on what is the "simple correlated representation of the data"?
It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.
- asdfasgasdgasdg 8y agoSorry, that's just a complicated way of saying that you can embed data in an inverted index that lets you guess how likely a document is to be found relevant on subsequent passes. Basically, you use various properties (embedded in the index) to filter down the list of documents you want to inspect in more detail, as you perform retrieval. There is some information in [1] about some types of filtering that can be done (e.g. their discussion on tiered retrieval and the notion of a global quality score being used to discard candidates). Lucene calls these bits of index-embedded data "token attributes," [2] but how exactly they are used depends on the scorer implementation. To learn more about how the industry approaches these issues, unfortunately you have to join one of the companies that's on the leading edge of this type of research, since they are loathe to disclose too much. [1]: https://nlp.stanford.edu/IR-book/pdf/07system.pdf https://nlp.stanford.edu/IR-book/pdf/07system.pdf [2]: https://pdfs.semanticscholar.org/2795/d9d165607b5ad6d8b9718373b82e55f41606.pdf https://pdfs.semanticscholar.org/2795/d9d165607b5ad6d8b97183...
- bogomipz 8y agoThanks for the detailed explanation and the links, these are really helpful. Cheers.