3 ms·

For anyone reading this, you can skip the robots.txt, as others have pointed out, who knows if they will actually listen to it. Instead, use a redirect or retu

For anyone reading this, you can skip the robots.txt, as others have pointed out, who knows if they will actually listen to it. Instead, use a redirect or return a response code by doing a user agent check in your server config. I posted elsewhere in this thread on the way i did it with nginx

salawat 3y ago
...You realize user agent is operator set, right? If they won't reapect robots.txt, they aren't interested in your consent.
- CableNinja 3y ago
  Respecting robots.txt and setting UA are two different things. And yes, i know UA can be set to anything, however, the UA has been mentioned, and it shouldnt change drastically, under normal circumstances by a lot of these scrapers. Respecting the robots.txt has nothing to do with what the UA is set to. Yes, you can say this UA can do x in the robots.txt, but not respecting it, makes it moot. The method i put in place does not use robots.txt, so theres no need to worry about them not respecting it anymore. As someone else mentioned, like the world of spam and such, its an arms race. The solution may not be perfect, but its functional