4 ms·
> Broadly speaking, robots.txt files are often ignored. If you wanna go nuclear on people who do that, include an invisible link in your html and forbid access
by chmod775 6y ago
> Broadly speaking, robots.txt files are often ignored.
If you wanna go nuclear on people who do that, include an invisible link in your html and forbid access to that URL in your robots.txt, then block every IP who accesses that URL for X amount of time.
Don't do this if you actually rely on search engine traffic though. Google may get pissed and send you lots of angry mail like "There's a problem with your site".
- jedberg 6y ago> Don't do this if you actually rely on search engine traffic though. Google may get pissed and send you lots of angry mail like "There's a problem with your site". Ah, but of course you would exclude Google's published crawler IPs from this restriction, because that is exactly what they want you to do.
- indymike 6y agoWe would occasionally have customer try doing that. AWS has lots of IP addresses :-).
- dzhiurgis 6y agoNice insight - use different IP address for hidden links!