3 ms·
Google is partly winning because many sites are now hostile to crawlers other than Google's. If we want real competition in the search engine space, Google's c
by ericd 6y ago
Google is partly winning because many sites are now hostile to crawlers other than Google's.
If we want real competition in the search engine space, Google's crawl should be opened up as a "common crawl", accessible to any company that wants to pay a nominal sum to access it for building an index.
- deleted 6y ago[deleted]
- MattGaiser 6y agoWhat do you mean by hostile? Do they ban non-Google crawlers?
- threatofrain 6y agoYes because it’s harder to verify the intentionality or harmoniousness behind the other bots.