3 ms·
Should we webmasters just start blocking user agents wholesale? I mean except known good actors. I guess known actors would need a verifiable signature
by dartos 2y ago
Should we webmasters just start blocking user agents wholesale?
I mean except known good actors.
I guess known actors would need a verifiable signature
- jsheard 2y agoSearch engine crawlers do have verifiable signatures, if a client claims to be Googlebot or Bingbot you don't have to take their word for it. https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/... https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
- yazzku 2y agoBut the converse is not true? There is no guarantee the crawler is not amassing data for model training, or that a crawler (AI or otherwise) does not disguise itself as a normal user?
- jsheard 2y agoYeah, but traffic appearing to come from normal users can be throttled and/or CAPTCHA'ed while still allowing Google and Bing to crawl to their hearts content so your SEO isn't affected.
- readyplayernull 2y agoGreedy crawlers will use fake user-agent strings.
- rty32 2y agoNot viable. They are going to use user agents that look like those coming from completely normal human users. "Verifiable signature"? That's a dangerous road to go down, and Google actually wanted to do it (Web Integrity API). Nobody supported them and they backed out.
- SoftTalker 2y agoI would think rate-limiting would be good. Crawlers are not patient enough to operate at the speed of a real human user.