3 ms·
We run a search engine crawler at mojeek and have no hostility problems
by ColinHayhurst 6y ago
We run a search engine crawler at mojeek and have no hostility problems
- Grimm1 6y agoHey we do too at Whize, surely you've seen sites that give special privilege to Google and or lock the sitemap away. While I don't know if I'd say "hostility" I would say passive aggressive to other crawlers.
- ericd 6y agoOh, good to know, thanks. Are you running relatively complete crawls of larger sites at similar rates to Googlebot? I've definitely seen my share of robots.txt that give special permission to Googlebot, but maybe my corner of the web was unusually aggressive toward crawlers.
- ColinHayhurst 6y agoNo, not complete crawls of any sites, we've aimed to get a wider coverage in place of deeper crawls. This means we have most sites indexed and are now gradually going deeper. We do encounter robots.txt blocks, but we don't see it as a major issue right now. We have an API, in case of interest.