6 ms·

My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.

frik 11y ago
Probably also one of the reasons to start the Chrome project, though their headless variant isn't open source. Are there other headless browser beside PhantomJS? PhantomJS is based on webkit (Safari).
- jon-wood 11y ago
  Chromium is Open Source and embeddable, so my guess is a big part of why their headless variant isn't open source is simply that there isn't really much to it.
  - frik 11y ago
    From a naive point of view it's easy. But it has to scale and launching a new process for every request works only for a pet project. So a crawler based on a headless browser that consumes little memory and runs for weeks is a major achivement.
    - manigandham 11y ago
      Why doesn't that scale? You might be underestimating Google's resources.
porker 11y ago
Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.
- kawsper 11y ago
  I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.
  - Aissen 11y ago
    They don't really need to, Verisign (for .com) and ICANN (for gTLDs) provide access to the zone files.
    - e12e 11y ago
      Depends on why they'd want to know about "new" domains. For indexing content, it would certainly make sense to index stuff people (appear to be) read(ing). Just because someone registers a domain, doesn't mean it has any (interesting) content.
- jacquesm 11y ago
  I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.
  - kawsper 11y ago
    Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.
    - jacquesm 11y ago
      No, I haven't. Just chrome for the one and gmail for the other. The way it works is simple, I made a random url, stuck a script in there that sends an email with the url as the subject header. The first script I visited using chrome, the second script I mailed myself a link to from another email account to a gmail account. Both scripts fired when chrome activated the links so I know they work, then I simply let it rest. I guess the gmail one would require re-crawling of all gmail for it to fire, the chrome only one would be dependent on the version of chrome that I ran the test on phoning home with the link, and I did not re-try this for every version of chrome (or on every platform). So it's not a perfect method but it definitely puts the lie to chrome or gmail data being directly used to power google search results that people would expect to remain confidential because only they have the urls. Score one for obscurity and nice of google to ignore these paths. I could set up another url for a test using the DNS but you're free to do so yourself as well of course, it is definitely an interesting idea. And if either of those scripts ever does fire I'll rip google a new one, that would be the sort of abuse of trust that gets my temperature up. But for now I'll take the fact that it hasn't happened as proof that google can be trusted with data to some extent.
txutxu 11y ago
+1 Also, maybe it's complementary to the headed version that lot of persons use and reports back. That millions of persons requesting and rendering html, are some kind of distributed indexers too.

frik 11y ago

Probably also one of the reasons to start the Chrome project, though their headless variant isn't open source. Are there other headless browser beside PhantomJS? PhantomJS is based on webkit (Safari).

jon-wood 11y ago
Chromium is Open Source and embeddable, so my guess is a big part of why their headless variant isn't open source is simply that there isn't really much to it.
- frik 11y ago
  From a naive point of view it's easy. But it has to scale and launching a new process for every request works only for a pet project. So a crawler based on a headless browser that consumes little memory and runs for weeks is a major achivement.
  - manigandham 11y ago
    Why doesn't that scale? You might be underestimating Google's resources.

porker 11y ago

Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.

kawsper 11y ago
I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.
- Aissen 11y ago
  They don't really need to, Verisign (for .com) and ICANN (for gTLDs) provide access to the zone files.
  - e12e 11y ago
    Depends on why they'd want to know about "new" domains. For indexing content, it would certainly make sense to index stuff people (appear to be) read(ing). Just because someone registers a domain, doesn't mean it has any (interesting) content.
jacquesm 11y ago
I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.
- kawsper 11y ago
  Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.
  - jacquesm 11y ago
    No, I haven't. Just chrome for the one and gmail for the other. The way it works is simple, I made a random url, stuck a script in there that sends an email with the url as the subject header. The first script I visited using chrome, the second script I mailed myself a link to from another email account to a gmail account. Both scripts fired when chrome activated the links so I know they work, then I simply let it rest. I guess the gmail one would require re-crawling of all gmail for it to fire, the chrome only one would be dependent on the version of chrome that I ran the test on phoning home with the link, and I did not re-try this for every version of chrome (or on every platform). So it's not a perfect method but it definitely puts the lie to chrome or gmail data being directly used to power google search results that people would expect to remain confidential because only they have the urls. Score one for obscurity and nice of google to ignore these paths. I could set up another url for a test using the DNS but you're free to do so yourself as well of course, it is definitely an interesting idea. And if either of those scripts ever does fire I'll rip google a new one, that would be the sort of abuse of trust that gets my temperature up. But for now I'll take the fact that it hasn't happened as proof that google can be trusted with data to some extent.

txutxu 11y ago

+1 Also, maybe it's complementary to the headed version that lot of persons use and reports back. That millions of persons requesting and rendering html, are some kind of distributed indexers too.