2 ms·
Search engine crawlers do have verifiable signatures, if a client claims to be Googlebot or Bingbot you don't have to take their word for it. https://developer
by jsheard 2y ago
Search engine crawlers do have verifiable signatures, if a client claims to be Googlebot or Bingbot you don't have to take their word for it.
https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
- yazzku 2y agoBut the converse is not true? There is no guarantee the crawler is not amassing data for model training, or that a crawler (AI or otherwise) does not disguise itself as a normal user?
- jsheard 2y agoYeah, but traffic appearing to come from normal users can be throttled and/or CAPTCHA'ed while still allowing Google and Bing to crawl to their hearts content so your SEO isn't affected.