2 ms·
Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expec
by jsheard 2y ago
Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check.
https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
- wilg 2y agoYou could maybe still only follow robots.txt rules for Googlebot.
- deleted 2y ago[deleted]