3 ms·
Yeah, this is why robots.txt is garbage. Too many crawlers completely ignore it, so I stopped bothering with it (I have it set to block all bots, but I don't ex
by JohnFen 3y ago
Yeah, this is why robots.txt is garbage. Too many crawlers completely ignore it, so I stopped bothering with it (I have it set to block all bots, but I don't expect it to be effective.)
Instead, I'd just keep an eye on my access logs and block obvious crawlers when I saw them.