4 ms·
>What are sites' motivations for blocking AWS IPs? I block AWS. So many crawlers up to so much nonsense! I don't block by IP, but by hostname. $block='.amaz
by pravda 9y ago
>What are sites' motivations for blocking AWS IPs?
I block AWS. So many crawlers up to so much nonsense!
I don't block by IP, but by hostname.
$block='.amazonaws.com';
$ua = @$_SERVER['HTTP_USER_AGENT'];
if (stripos($rh,$block)!==false &&
stripos($ua,'Silk')===false &&
stripos($ua,'Safari')===false){
$block_visitor=true;
$message="Blocked Host:Amazon Web Services";
}
- hackits 9y agoJust curious what have you seen crawlers do to make you conclude they're up to nonsense?
- pravda 9y agoWell, from amazonaws.com, there are so many requests for wp-login.php! And then all the off-brand scraping companies use amazonaws.com.
- afandian 9y agoWhat do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
- pravda 9y agoSome AWS visitors: Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/ http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/ http://www.linkdex.com/bots/ That last one is a "SEO platform".