7 ms·

I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of update

I've always thought it would make more sense if each web server could be responsible for indexing the material that it serves (and offer notifications of updates), so instead of having to crawl everything yourself, you could just request the index from each domain, and then merge them.

mh- 7y ago
PubSubHubbub was intended to be something like that. https://en.wikipedia.org/wiki/WebSub https://en.wikipedia.org/wiki/WebSub
peterwwillis 7y ago
Like a sitemap? https://en.m.wikipedia.org/wiki/Sitemaps https://en.m.wikipedia.org/wiki/Sitemaps
- aheilbut 7y ago
  That is a start, but I mean an actual inverted index, and preferably even more structured indices with appropriate metadata. Web servers should also be responsible for archiving themselves and providing a change history.
- dredmorbius 7y ago
  No, an actual search index, offerring word-to-URL mappings, along with metadata: creation and revision dates, authors, titles, file and/or MIME types, URLs within the text, other attributes, etc, from which a search interface could query. Page-ranking would remain an issue, likely outside this scope. I'd like to see some sort of cache-and-forward structure. And you'd be relying on good-faith actors, which means heavily penalising bad actors.
lenzm 7y ago
A significant problem with this is trust. You can't trust websites to reliably or accurately index their sites due to both incompetence and malice. I don't think there's any way around the malicious component. Formal or informal standards may take care of the competence factor with the feature being built into common publishing platforms. XML sitemaps are a microcosm of putting the indexing onus on websites instead of the search engines - they are basically ignored by search engines because they have been abused and are not a useful signal. If pages aren't important enough to be linked to throughout your website then they aren't interpreted as being important enough to return to users. The optimistic case is that sitemaps/indices will send parallel signals to the search engines in which case they are redundant. The pessimistic case is that the sitemaps/indices will send signals orthogonal to the content provided to users in which case the website is either being deceitful or incompetent. In any case, the search engine will not want to use the sitemap/index as a signal as it either doesn't provide value or provides negative value.
- aheilbut 7y ago
  The code for doing the indexing (at least by default) could be built right in to the web server, so it'd just be a matter of enabling an option in Apache or the like. It would be pretty easy to verify whether or not the index is accurate with a small random sample of pages on the site, and then penalize / exclude (or do a de-novo crawl) for those sites not providing a legit index.
- bobajeff 7y ago
  One solution could be to have a verifiable module on servers that creates and serves the sitemap. Something like a signed certificate or DRM.
- oldjokes 7y ago
  Honestly I'd much rather have a bunch of dice rolls on incompetence than the current centralized, single point of control over the entire index. Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using. It's difficult to imagine a worse solution for the web than this current model.
  - bogomipz 7y ago
    "Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using." Wow, interesting, this is the first I've heard of this. Might you have some link or citations about this? Thanks.
    - aytekin 7y ago
      4% of the Google index hit by de-indexing https://searchengineland.com/4-of-the-google-index-hit-by-de-indexing-bug-moz-data-shows-315248 https://searchengineland.com/4-of-the-google-index-hit-by-de...
asdfasgasdgasdg 7y ago
I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues. The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to implement some sort of multi-level fanout, since just one machine can't send enough HTTP requests -- nor even establish that many connections -- in a reasonable time-frame. There are going to be severe tail latency issues that will prevent you from gathering documents from potentially relevant sites. You will have to make frustrating tradeoffs about when to time-out per domain queries to provide a good user experience. And many more issues besides. All decisions that are unnecessary if you control the index. Also, internet bandwidth isn't cheap, and you're going to need a lot of it just to consume the top ten results from a million sites. The next technical issue is that the inverted index is only a small part of what goes into information retrieval. Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data. Then they do a second pass on a more comprehensive form (i.e. the whole document, and metadata about the document), but over fewer documents. There can even be third and fourth passes. The data for the first pass is often embedded directly into the index, and it would be challenging to come to any kind of agreement among stakeholders about what data should be embedded. This goes double for the second and subsequent passes. Moreover, those second and subsequent passes often use data about the document, rather than data in the document. Data a site owner would be unable to provide or even incentivized to falsify. Not less than these problems is the issue of where to run the scorer code. If you're running it locally, you're operationally 90% of the way to the complexity of an inverted index. Why not go all the way? Then of course there are the economic issues, which, roughly, are: "Why should I pay all this money to host an index of my site that nobody uses when Google will do it for free and charge me nothing?"
- bogomipz 7y ago
  >"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data." Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.
deytempo 7y ago
This reminds me of how DNS works. Every domain holder is responsible for their nameserver records but every dns server ultimately communicates with a distributed network for lookups. Blockchain would be a great solution for this.

mh- 7y ago

PubSubHubbub was intended to be something like that. https://en.wikipedia.org/wiki/WebSub https://en.wikipedia.org/wiki/WebSub

peterwwillis 7y ago

Like a sitemap? https://en.m.wikipedia.org/wiki/Sitemaps https://en.m.wikipedia.org/wiki/Sitemaps

aheilbut 7y ago
That is a start, but I mean an actual inverted index, and preferably even more structured indices with appropriate metadata. Web servers should also be responsible for archiving themselves and providing a change history.
dredmorbius 7y ago
No, an actual search index, offerring word-to-URL mappings, along with metadata: creation and revision dates, authors, titles, file and/or MIME types, URLs within the text, other attributes, etc, from which a search interface could query. Page-ranking would remain an issue, likely outside this scope. I'd like to see some sort of cache-and-forward structure. And you'd be relying on good-faith actors, which means heavily penalising bad actors.

lenzm 7y ago

A significant problem with this is trust. You can't trust websites to reliably or accurately index their sites due to both incompetence and malice. I don't think there's any way around the malicious component. Formal or informal standards may take care of the competence factor with the feature being built into common publishing platforms. XML sitemaps are a microcosm of putting the indexing onus on websites instead of the search engines - they are basically ignored by search engines because they have been abused and are not a useful signal. If pages aren't important enough to be linked to throughout your website then they aren't interpreted as being important enough to return to users. The optimistic case is that sitemaps/indices will send parallel signals to the search engines in which case they are redundant. The pessimistic case is that the sitemaps/indices will send signals orthogonal to the content provided to users in which case the website is either being deceitful or incompetent. In any case, the search engine will not want to use the sitemap/index as a signal as it either doesn't provide value or provides negative value.

aheilbut 7y ago
The code for doing the indexing (at least by default) could be built right in to the web server, so it'd just be a matter of enabling an option in Apache or the like. It would be pretty easy to verify whether or not the index is accurate with a small random sample of pages on the site, and then penalize / exclude (or do a de-novo crawl) for those sites not providing a legit index.
bobajeff 7y ago
One solution could be to have a verifiable module on servers that creates and serves the sitemap. Something like a signed certificate or DRM.
oldjokes 7y ago
Honestly I'd much rather have a bunch of dice rolls on incompetence than the current centralized, single point of control over the entire index. Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using. It's difficult to imagine a worse solution for the web than this current model.
- bogomipz 7y ago
  "Google has been purging large swaths of data from the indexes and they won't say how or why or exactly what criteria they are using." Wow, interesting, this is the first I've heard of this. Might you have some link or citations about this? Thanks.
  - aytekin 7y ago
    4% of the Google index hit by de-indexing https://searchengineland.com/4-of-the-google-index-hit-by-de-indexing-bug-moz-data-shows-315248 https://searchengineland.com/4-of-the-google-index-hit-by-de...

asdfasgasdgasdg 7y ago

I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues. The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to implement some sort of multi-level fanout, since just one machine can't send enough HTTP requests -- nor even establish that many connections -- in a reasonable time-frame. There are going to be severe tail latency issues that will prevent you from gathering documents from potentially relevant sites. You will have to make frustrating tradeoffs about when to time-out per domain queries to provide a good user experience. And many more issues besides. All decisions that are unnecessary if you control the index. Also, internet bandwidth isn't cheap, and you're going to need a lot of it just to consume the top ten results from a million sites. The next technical issue is that the inverted index is only a small part of what goes into information retrieval. Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data. Then they do a second pass on a more comprehensive form (i.e. the whole document, and metadata about the document), but over fewer documents. There can even be third and fourth passes. The data for the first pass is often embedded directly into the index, and it would be challenging to come to any kind of agreement among stakeholders about what data should be embedded. This goes double for the second and subsequent passes. Moreover, those second and subsequent passes often use data about the document, rather than data in the document. Data a site owner would be unable to provide or even incentivized to falsify. Not less than these problems is the issue of where to run the scorer code. If you're running it locally, you're operationally 90% of the way to the complexity of an inverted index. Why not go all the way? Then of course there are the economic issues, which, roughly, are: "Why should I pay all this money to host an index of my site that nobody uses when Google will do it for free and charge me nothing?"

bogomipz 7y ago
>"Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data." Can you elaborate on what is the "simple correlated representation of the data"? It sounds like you understand this space pretty well might you have any links or literature on how modern crawling architectures and indexing work? Thanks.

deytempo 7y ago

This reminds me of how DNS works. Every domain holder is responsible for their nameserver records but every dns server ultimately communicates with a distributed network for lookups. Blockchain would be a great solution for this.