2 ms·
Because then you end up in an arms race that the little guy usually does not win. There are a significant number of crawlers out there that don't respect robot
by nostrademons 8y ago
Because then you end up in an arms race that the little guy usually does not win.
There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations).