3 ms·
Robots.txt is for crawlers. It's explicitly not meant to say one-off requests from user agents can't access the site, because that would break the open web.
by lolinder 2y ago
Robots.txt is for crawlers. It's explicitly not meant to say one-off requests from user agents can't access the site, because that would break the open web.
- Spivak 2y agoYep, there's really two parts to this. * Some company's crawler they're planning to use for AI training data. * User agents that make web requests on behalf of a person. Blocking the second one because the user's preferred browser is ChatGPT isn't really in keeping with the hacker spirit. The client shouldn't matter, I would hope that the web is made to be consumed by more than just Chrome.