2 ms·

We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a c

by tleb_ 6y ago

We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.

ddorian43 6y ago
https://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
- Xylakant 6y ago
  This helps to verify that a bot that announces itself as google bot is indeed a google bot. It doesn’t help identify a bot that pretends to be a user/browser.