5 ms·

I assume the number one use of this would be test automation for one's own sites so blocking would not be an issue. What are sites' motivations for blocking AW

I assume the number one use of this would be test automation for one's own sites so blocking would not be an issue. What are sites' motivations for blocking AWS IPs? I bet there are some reasons I would agree with even though the somewhat crude method of blocking ip range would have some unintended consequences (e.g. blocking people running a personal VPN).

pravda 9y ago
>What are sites' motivations for blocking AWS IPs? I block AWS. So many crawlers up to so much nonsense! I don't block by IP, but by hostname. $block='.amazonaws.com'; $ua = @$_SERVER['HTTP_USER_AGENT']; if (stripos($rh,$block)!==false && stripos($ua,'Silk')===false && stripos($ua,'Safari')===false){ $block_visitor=true; $message="Blocked Host:Amazon Web Services"; }
- hackits 9y ago
  Just curious what have you seen crawlers do to make you conclude they're up to nonsense?
  - pravda 9y ago
    Well, from amazonaws.com, there are so many requests for wp-login.php! And then all the off-brand scraping companies use amazonaws.com.
    - afandian 9y ago
      What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?
      - pravda 9y ago
        Some AWS visitors: Cliqzbot, VidibleScraper/1.0, CheckMarkNetwork, CCBot/2.0 (http://commoncrawl.org/faq/ http://commoncrawl.org/faq/), linkdexbot/2.2; +http://www.linkdex.com/bots/ http://www.linkdex.com/bots/ That last one is a "SEO platform".