3 ms·
Not that I'm arguing for or against preventing access from AI crawlers, but wouldn't it make more sense to block them at a higher level, e.g. the webserver, and
by rocky_raccoon 3y ago
Not that I'm arguing for or against preventing access from AI crawlers, but wouldn't it make more sense to block them at a higher level, e.g. the webserver, and not even give them the choice to obey/disobey robots.txt?
- rideontime 3y agoHow would you propose doing so?
- adrianN 3y agoWe could repurpose the evil bit.
- rideontime 3y agoOne second, let me google this. e: Okay, this is funny.
- gtirloni 3y agoWeb servers can check the user-agent and block the request. E.g. nginx $http_user_agent
- rocky_raccoon 3y agoOff the top of my head: - Cloudflare - Webserver-level user-agent blocking (Apache, nginx) - Application-level user-agent blocking (`if request.user_agent == 'OpenAI'`) None of them are ideal since you can simply change your user agent, but all of them seem like better options than robots.txt to me.