3 ms·
The de facto standard robots.txt is pretty likely to be respected by Google, so it's fairly easy to stop their scraping your site. Yes, it is opt-out, bit I'd e
by ezequiel-garzon 13y ago
The de facto standard robots.txt is pretty likely to be respected by Google, so it's fairly easy to stop their scraping your site. Yes, it is opt-out, bit I'd expect it to be.
It may be quite frustrating for an upstart to be denied access while Google is explicitly allowed, but that's another matter.