2 ms·
So you expect all websites to block FoobarSearch so it never gets off the ground and becomes a big search engine that people know to unblock. Then FoobarSearch
by immibis 2y ago
So you expect all websites to block FoobarSearch so it never gets off the ground and becomes a big search engine that people know to unblock.
Then FoobarSearch learns to ignore robots.txt wildcards, and we're back at square one.
IIRC this happened to DDG or Bing.
- threeseed 2y agoWebsites have always had the ability to precisely control who has access to their content. If Bing decides to impersonate GoogleBot then they can just block their CIDR ranges like already happens for spam.