3 ms·
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
by eminence32 6y ago
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
- oefrha 6y agoTraders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.
- drcongo 6y agoA site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there. edit: replaced "scrapes" with "crawls" for accuracy.