I can't share the specific examples, unfortunately. That's right that if you force a googlebot request from the webmaster tools, it's essentially just how the

I can't share the specific examples, unfortunately. That's right that if you force a googlebot request from the webmaster tools, it's essentially just how the basic googlebot would wget a page on your site. We know they're doing more than that though, because it would be trivial to hide cloaking and malware from Google if that's the only tool they used for scraping. What Google actually does for crawling content seems to be a bit more nuanced. If you watch the server-side logs on a site that Google crawls often, you'll sometimes see a pattern of googlebot crawling a page and then an immediate subsequent request from a Google IP from a Chrome UA. I'm sure that's partially just to assess malware/cloaking, but it's probably also related to developments like this one: https://twitter.com/mattcutts/status/131425949597179904 https://twitter.com/mattcutts/status/131425949597179904 They're definitely doing more than just the ?_escaped_fragment_ rigamarole that almost no one uses. Fair point about the AdSense crawler though. It seems to be just about as dumb as a rock in my experience.