2 ms·

I'm with you on this, not for huge sites but if even 1% of sites used this behavior malicious bots would be forced to honor robots.txt