3 ms·
Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
by hackits 9y ago
Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
- pravda 9y agoWell, from amazonaws.com, there are so many requests for wp-login.php! And then all the off-brand scraping companies use amazonaws.com.
- afandian 9y agoWhat do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
- pravda 9y agoSome AWS visitors: Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/ http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/ http://www.linkdex.com/bots/ That last one is a "SEO platform".