3 ms·
I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working
by asdfasgasdgasdg 8y ago
I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues.
The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to implement some sort of multi-level fanout, since just one machine can't send enough HTTP requests -- nor even establish that many connections -- in a reasonable time-frame. There are going to be severe tail latency issues that will prevent you from gathering documents from potentially relevant sites. You will have to make frustrating tradeoffs about when to time-out per domain queries to provide a good user experience. And many more issues besides. All decisions that are unnecessary if you control the index. Also, internet bandwidth isn't cheap, and you're going to need a lot of it just to consume the top ten results from a million sites.
The next technical issue is that the inverted index is only a small part of what goes into information retrieval. Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data. Then they do a second pass on a more comprehensive form (i.e. the whole document, and metadata about the document), but over fewer documents. There can even be third and fourth passes. The data for the first pass is often embedded directly into the index, and it would be challenging to come to any kind of agreement among stakeholders about what data should be embedded. This goes double for the second and subsequent passes. Moreover, those second and subsequent passes often use data about the document, rather than data in the document. Data a site owner would be unable to provide or even incentivized to falsify. Not less than these problems is the issue of where to run the scorer code. If you're running it locally, you're operationally 90% of the way to the complexity of an inverted index. Why not go all the way?
Then of course there are the economic issues, which, roughly, are: "Why should I pay all this money to host an index of my site that nobody uses when Google will do it for free and charge me nothing?"
- bogomipz 8y ago>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data." Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.
- asdfasgasdgasdg 8y agoSorry, that's just a complicated way of saying that you can embed data in an inverted index that lets you guess how likely a document is to be found relevant on subsequent passes. Basically, you use various properties (embedded in the index) to filter down the list of documents you want to inspect in more detail, as you perform retrieval. There is some information in [1] about some types of filtering that can be done (e.g. their discussion on tiered retrieval and the notion of a global quality score being used to discard candidates). Lucene calls these bits of index-embedded data "token attributes," [2] but how exactly they are used depends on the scorer implementation. To learn more about how the industry approaches these issues, unfortunately you have to join one of the companies that's on the leading edge of this type of research, since they are loathe to disclose too much. [1]: https://nlp.stanford.edu/IR-book/pdf/07system.pdf https://nlp.stanford.edu/IR-book/pdf/07system.pdf [2]: https://pdfs.semanticscholar.org/2795/d9d165607b5ad6d8b9718373b82e55f41606.pdf https://pdfs.semanticscholar.org/2795/d9d165607b5ad6d8b97183...
- bogomipz 8y agoThanks for the detailed explanation and the links, these are really helpful. Cheers.