2 ms·
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. That depends what you expect from it. For the purpose of limiting cr
by mschuster91 4y ago
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.
That depends what you expect from it. For the purpose of limiting crawlers, at least the major search engines respect it.