4 ms·

Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very

by porker 11y ago

Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.

kawsper 11y ago
I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.
- Aissen 11y ago
  They don't really need to, Verisign (for .com) and ICANN (for gTLDs) provide access to the zone files.
  - e12e 11y ago
    Depends on why they'd want to know about "new" domains. For indexing content, it would certainly make sense to index stuff people (appear to be) read(ing). Just because someone registers a domain, doesn't mean it has any (interesting) content.
jacquesm 11y ago
I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.
- kawsper 11y ago
  Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.
  - jacquesm 11y ago
    No, I haven't. Just chrome for the one and gmail for the other. The way it works is simple, I made a random url, stuck a script in there that sends an email with the url as the subject header. The first script I visited using chrome, the second script I mailed myself a link to from another email account to a gmail account. Both scripts fired when chrome activated the links so I know they work, then I simply let it rest. I guess the gmail one would require re-crawling of all gmail for it to fire, the chrome only one would be dependent on the version of chrome that I ran the test on phoning home with the link, and I did not re-try this for every version of chrome (or on every platform). So it's not a perfect method but it definitely puts the lie to chrome or gmail data being directly used to power google search results that people would expect to remain confidential because only they have the urls. Score one for obscurity and nice of google to ignore these paths. I could set up another url for a test using the DNS but you're free to do so yourself as well of course, it is definitely an interesting idea. And if either of those scripts ever does fire I'll rip google a new one, that would be the sort of abuse of trust that gets my temperature up. But for now I'll take the fact that it hasn't happened as proof that google can be trusted with data to some extent.
  - uhoreg 11y ago
    DNS only sees the host name, so it can't be used to see what URLs are being accessed.
    - jacquesm 11y ago
      I think that was the test case that he intended. So set up a domain that is otherwise unknown and then use google's DNS and see if the domain is hit by the search engine.
anarchitect 11y ago
We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages). This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.

by porker 11y ago

kawsper 11y ago

I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.

Aissen 11y ago
They don't really need to, Verisign (for .com) and ICANN (for gTLDs) provide access to the zone files.
- e12e 11y ago
  Depends on why they'd want to know about "new" domains. For indexing content, it would certainly make sense to index stuff people (appear to be) read(ing). Just because someone registers a domain, doesn't mean it has any (interesting) content.

jacquesm 11y ago

I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.

kawsper 11y ago
Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.
- jacquesm 11y ago
  No, I haven't. Just chrome for the one and gmail for the other. The way it works is simple, I made a random url, stuck a script in there that sends an email with the url as the subject header. The first script I visited using chrome, the second script I mailed myself a link to from another email account to a gmail account. Both scripts fired when chrome activated the links so I know they work, then I simply let it rest. I guess the gmail one would require re-crawling of all gmail for it to fire, the chrome only one would be dependent on the version of chrome that I ran the test on phoning home with the link, and I did not re-try this for every version of chrome (or on every platform). So it's not a perfect method but it definitely puts the lie to chrome or gmail data being directly used to power google search results that people would expect to remain confidential because only they have the urls. Score one for obscurity and nice of google to ignore these paths. I could set up another url for a test using the DNS but you're free to do so yourself as well of course, it is definitely an interesting idea. And if either of those scripts ever does fire I'll rip google a new one, that would be the sort of abuse of trust that gets my temperature up. But for now I'll take the fact that it hasn't happened as proof that google can be trusted with data to some extent.
- uhoreg 11y ago
  DNS only sees the host name, so it can't be used to see what URLs are being accessed.
  - jacquesm 11y ago
    I think that was the test case that he intended. So set up a domain that is otherwise unknown and then use google's DNS and see if the domain is hit by the search engine.

anarchitect 11y ago

We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages). This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.