2 ms·
No need, you can block them by user agent, they are consistent - or just disallow them in robots.txt. If you need to make sure it's actually a google bot when
by luckylion 8y ago
No need, you can block them by user agent, they are consistent - or just disallow them in robots.txt.
If you need to make sure it's actually a google bot when a client shows up with the user agent, you can use reverse dns: https://support.google.com/webmasters/answer/80553 https://support.google.com/webmasters/answer/80553