3 ms·
You can buy databases of who owns which IP blocks. If you really care but don't want to spend the money, just block the subnet each time you see a Googlebot re
by jrockway 5y ago
You can buy databases of who owns which IP blocks.
If you really care but don't want to spend the money, just block the subnet each time you see a Googlebot request. "whois w.x.y.z" returns an entire CIDR, and it seems unlikely to me that Google is scraping from a bunch of disconnected /24s.