4 ms·

I actually didn't know that. Do you operate the same crawlers? I have considered putting a single file that is only accessible via no-follow links and perma-ba

by tomjen3 10y ago

I actually didn't know that. Do you operate the same crawlers? I have considered putting a single file that is only accessible via no-follow links and perma-ban any ip that access the file, as a way to punish bad robots.

bigbugbag 10y ago
FWIW humans happen to be able to choose their user-agent at will. Not so long ago changing your user-agent to one of the search engine bot as a simple workaround for some paywalls that appeared in search results was a thing. It's also part of the techniques used to give extra privacy and messing with fingerprinting. For example random agent spoofer: https://github.com/dillbyrne/random-agent-spoofer https://github.com/dillbyrne/random-agent-spoofer