2 ms·
Clearly, a bunch of Google's services have been able to behave like a rendering/JS-executing browser for a while. However, that may not mean the initial URL fet
by gojomo 15y ago
Clearly, a bunch of Google's services have been able to behave like a rendering/JS-executing browser for a while. However, that may not mean the initial URL fetching is usually (or even often) by a Chrome-like process.
My guess would be there are a mixture of processes which all collect page captures into a working area. (Which process is used for a particular site/URL might vary over time based on feature-detection – for example an initial old-style survey collection might be followed by a more browser-like collection later. The 'working area' is whatever serves the BigTable role in 2011.) All captures can then be further analyzed (virtually re-crawled from the working area) as necessary, by an even larger collection of analysis processes. That many of these analysis processes share code with other Google projects is to be expected, but to say that they are actually 'GoogleBot' (once they're not doing the initial fetching) or 'Chrome' (when they're processing bulk working-area data after-the-fact) is likely an oversimplification that obscures more than it is illuminates.
The strongest external evidence that Google's main crawling was using a Chrome-like engine would be if the pattern of URL-fetching, as observed in logs, became more like a browser: everything needed to render one page fetched in rapid succession (unless it's already previously cached). Are people seeing that in their logs, for example on Googlebot's first visit to all-new content?