5 ms·

I’d assume that they occasionally send requests with different IPs and user-agents to detect cloaking.

• chrisper 9y ago
  Yes, but their IPs are registered to Google Inc.
  • TeMPOraL 9y ago
    Are they?
    • joombaga 9y ago
      Yes. Are you implying they are not?
      • TeMPOraL 9y ago
        Yes? I mean, scanning for cloaking only from their own IP range sounds ridiculous.
• codedokode 9y ago
  I think that cloaking is very difficult to detect automatically without using humans. You have to distinguish for example between a paywall and real cloaking.
• leni536 9y ago
  Maybe they change the behavior of their random() function as well. The deterministic random() function can very well be a red herring.
  • lucideer 9y ago
    I'd guess if they're crawling purely for the purpose of detecting cloaking (which would be a much smaller-scale job than the standard Googlebot indexing), they'd just use Chrome Headless[0][1] [0] https://intoli.com/blog/not-possible-to-block-chrome-headless/ https://intoli.com/blog/not-possible-to-block-chrome-headles... [1] http://antoinevastel.github.io/bot%20detection/2018/01/17/detect-chrome-headless-v2.html http://antoinevastel.github.io/bot%20detection/2018/01/17/de...
    • thecatspaw 9y ago
      I doubt that they use chrome at all. They probably just fetch the page over http and let it run in a JS sandbox. hence the deterministic random function (which I assume is not there in chrome?)
      • DamonHD 9y ago
        2018/01/18: Chrome 41: The Key to Successful Website Rendering: "Google updated their Search Guides and announced that they use Chrome 41 for rendering." https://www.elephate.com/blog/chrome-41-key-to-website-rendering/ https://www.elephate.com/blog/chrome-41-key-to-website-rende...
        • codedokode 9y ago
          It is interesting that when I use Chromium 46 (which is newer than Googlebot) I get warnings from Github and Google Docs about an outdated or unsupported browser, and I get plain HTML Github pages without JS. But Google uses even older browser than I. So even Google cannot cope with browser version race and keep their browsers updated although they require this from the users.
          • lucideer 9y ago
            According to that article, Googlebot is "based on Chromium 41", it doesn't actually use Chromium 41 directly, it's still a distinct, separately developed browser. The warning you're seeing is a) probably shown based on UA, and Googlebot has a very different UA to the Chromium browser b) warning users who will need to browse/actively use the site. Googlebot simply parses content, so has no use for quite a lot of the active functionality on the site. As such, it will generally not need to support all of the features used by that active functionality, just barely enough to be able to read content.
      • lucideer 9y ago
        I'm sure they don't use Chrome for Googlebot, that would be quite inefficient. My comment was about cloaking detection.
  • taneq 9y ago
    Maybe that's just what they want you to think! /tinfoil