4 ms·
You just have a rule that says block everything except crawlers: A, B, C. Also the AppleBot was known about before it appeared in Siri.
by threeseed 2y ago
You just have a rule that says block everything except crawlers: A, B, C.
Also the AppleBot was known about before it appeared in Siri.
- immibis 2y agoSo you expect all websites to block FoobarSearch so it never gets off the ground and becomes a big search engine that people know to unblock. Then FoobarSearch learns to ignore robots.txt wildcards, and we're back at square one. IIRC this happened to DDG or Bing.
- threeseed 2y agoWebsites have always had the ability to precisely control who has access to their content. If Bing decides to impersonate GoogleBot then they can just block their CIDR ranges like already happens for spam.