4 ms·
If I were making a search engine or AI crawler, I would simply pose as Googlebot
by wilg 2y ago
If I were making a search engine or AI crawler, I would simply pose as Googlebot
- jsheard 2y agoGoogle actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check. https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
- wilg 2y agoYou could maybe still only follow robots.txt rules for Googlebot.
- deleted 2y ago[deleted]