4 ms·
By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run
by intricatedetail 5y ago
By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run reverse DNS as it slows down legitimate clients.
I know you can put a meta tag, but bot still has to make a request to read it.
I would like to completely cut off Google from scrapping.
- jrockway 5y agoYou can buy databases of who owns which IP blocks. If you really care but don't want to spend the money, just block the subnet each time you see a Googlebot request. "whois w.x.y.z" returns an entire CIDR, and it seems unlikely to me that Google is scraping from a bunch of disconnected /24s.
- drivebycomment 5y agoJust put robot.txt and block Googlebot from there. Google obeys robot.txt.
- intricatedetail 5y agoNot all Google crawlers obey it. Also if Google already indexed something, only way is to let it crawl again and see meta noindex. It's a mess.