3 ms·
And even those are advisory, if you really need to block them, you have to add rules to your webserver of known user agents and IPs (because plenty of indexers
by Cthulhu_ 5y ago
And even those are advisory, if you really need to block them, you have to add rules to your webserver of known user agents and IPs (because plenty of indexers will spoof their user agent as well. Google probably does that so they can penalize those that offer different content to search engines than regular users. Doesn't really work though).