4 ms·
Well, then change your user agent to 'googlebot' when crawling the web, since it now became the synonym for search crawlers?
by snidane 5y ago
Well, then change your user agent to 'googlebot' when crawling the web, since it now became the synonym for search crawlers?
- jeroenhd 5y agoThat trick doesn't work because people also filter out Google's IP addresses to verify that the "googlebot" crawler is actually a real Google bot.
- deleted 5y ago[deleted]
- jasonladuke0311 5y agoCan you run a crawler from GCP or do they explicitly filter for other Google AS?