30 ms·

Since you appear to have a background that includes search and Google I thought I would ask you or anyone else a tangential question - with more than 30 Trillio

Since you appear to have a background that includes search and Google I thought I would ask you or anyone else a tangential question - with more than 30 Trillion sites[1] on the internet how is the crawling ever completed before it needs to be re-indexed again? I'm imagining that "the frontier" a crawler needs to crawl is actually a distributed queue and that crawlers are massively parallelized. I'm also imagining the frontier is bucketed by its indexing frequency i.e daily, weekly monthly etc. Is that close? Might you have any resources on how this is architected at large search providers? [1] https://venturebeat.com/2013/03/01/how-google-searches-30-trillion-web-pages-100-billion-times-a-month/ https://venturebeat.com/2013/03/01/how-google-searches-30-tr...

jedberg 9y ago
The crawl is never done. It's constantly crawling. And you're right, it's just a queue of websites that's been found, and when a crawling machine has an available slot, it crawls the next site (obviously that's simplified). Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site. If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.
- owenversteeg 9y ago
  That's fascinating. Is there any way to contact Google to get them to change their crawl patterns? Or a more efficient way to directly send them the site updates as they happen? I imagine a "direct line" to Google would save resources both on Google's side and yours, but I've never ran a site big enough for that to be a problem so I wouldn't know.
  - himlion 9y ago
    I think that would lend itself to abuse.
ChuckMcM 9y ago
> "[H]ow is the crawling ever completed before > it needs to be re-indexed again?" Crawls essentially never finish and indexes are never 'done'. If you're building a modern search engine you crawl and index continuously and you use the ranking signals you have developed to steer the crawler to promising new pages. The reality is that there are roughly between 5 to 15 billion pages that are nominally "not spam" and not duplicates. Literally 99.9% of the internet is crap. So finding web pages as long since switched from 'crawling every page that is accessible from the web' to 'only surfacing those pages that have something of value on them.' That was the fundamental thesis for the founding of Blekko and it is still true today. That said, a cluster of ~100,000 threads and a couple of petabytes of storage attached with sufficient bandwidth to keep the threads busy can deal with what is out there. If you can create a hash space for strings that is sufficiently uniform you evenly spread the load of crawling every URI you discover across the cluster. As you crawl you take new pages you discover and apply your ranking algorithm to them where they will score a value between 0 (never index) and 1 (always index). At which point you can dial the 'rankable' value from 0 (index everything) to 1 (rank only must rank pages) to set the size of the index you can tolerate.

jedberg 9y ago

The crawl is never done. It's constantly crawling. And you're right, it's just a queue of websites that's been found, and when a crawling machine has an available slot, it crawls the next site (obviously that's simplified). Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site. If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.

owenversteeg 9y ago
That's fascinating. Is there any way to contact Google to get them to change their crawl patterns? Or a more efficient way to directly send them the site updates as they happen? I imagine a "direct line" to Google would save resources both on Google's side and yours, but I've never ran a site big enough for that to be a problem so I wouldn't know.
- himlion 9y ago
  I think that would lend itself to abuse.

ChuckMcM 9y ago

> "[H]ow is the crawling ever completed before > it needs to be re-indexed again?" Crawls essentially never finish and indexes are never 'done'. If you're building a modern search engine you crawl and index continuously and you use the ranking signals you have developed to steer the crawler to promising new pages. The reality is that there are roughly between 5 to 15 billion pages that are nominally "not spam" and not duplicates. Literally 99.9% of the internet is crap. So finding web pages as long since switched from 'crawling every page that is accessible from the web' to 'only surfacing those pages that have something of value on them.' That was the fundamental thesis for the founding of Blekko and it is still true today. That said, a cluster of ~100,000 threads and a couple of petabytes of storage attached with sufficient bandwidth to keep the threads busy can deal with what is out there. If you can create a hash space for strings that is sufficiently uniform you evenly spread the load of crawling every URI you discover across the cluster. As you crawl you take new pages you discover and apply your ranking algorithm to them where they will score a value between 0 (never index) and 1 (always index). At which point you can dial the 'rankable' value from 0 (index everything) to 1 (rank only must rank pages) to set the size of the index you can tolerate.