3 ms·
The problem with robots.txt is that it depends on the crawler to respect it (and many do not). It's not an adequate protection against crawlers in general and a
by JohnFen 3y ago
The problem with robots.txt is that it depends on the crawler to respect it (and many do not). It's not an adequate protection against crawlers in general and a specific form just for AI wouldn't be adequate either, for the same reason.