4 ms·

Perhaps time to start a central community ban pool for IP ranges?

• MartijnBraam 2y ago
  Doesn't really work if crawlers are coming from the IP ranges of AWS and Azure etc...
  • Tostino 2y ago
    Or sometimes they use consumer IP proxies. Makes it even harder because sometimes those IPs get reused for actual users.
    • megadata 2y ago
      Block them for 24 hours.
  • immibis 2y ago
    Traditionally, holders of IP ranges that attack the internet at large get kicked off the internet by having those ranges blacklisted everywhere. This can also get them in serious trouble with the places they got their IP ranges (I assume AWS has them directly from ARIN, so maybe not) and their upstream bandwidth providers and so on, as well as making them less attractive hosts because they are blocked everywhere.
    • mrweasel 2y ago
      Kicking AWS of the internet would effectively break half of everything. So much stuff is relying on services running on AWS that it's not even funny. We've seen random stuff break when AWS has had outages, not because we used AWS ourselves, but because suppliers do.
      • megadata 2y ago
        You don't need to kick anyone off the internet. Just ban them for accessing your resources.
        • mrweasel 2y ago
          No, but the suggestion from the parent comment was to have the holder of the offending IP ranges kicked off the internet. Technically I'm all for kicking AWS off the internet for a day or to, for failing to police their customers, but it would just break a lot.
      • curtisblaine 2y ago
        That's actually an argument in favour of kicking AWS off the Internet. We rely too much on their services, to the point we're afraid of banning their IPs if they do something bad. Better stop this now than being worse off later. The best moment would have been ten years ago, the second best moment is today.
  • blibble 2y ago
    just ban the lot nothing good comes from there unfortunately then they instantly switch to home IPs
  • megadata 2y ago
    AWS and Azure can be blocked like anyone else.
• Galanwe 2y ago
  There's already loads of these. The problem is that most of these IPs are just cloud providers or DC ISPs.
  • tecleandor 2y ago
    Or even worse, lots of them are using barely legal residential proxies so the requests are coming from everywhere. In Drew DeVault's article linked in this post he complained precisely about the residential-looking source IP addresses [0]. And I think I remember something about a Chinese company, some months ago, very aggressively scraping using that method. Companies like DataImpulse [1] or ScraperAPI [2] will happily publicize their services with that specific target. -- 0: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html 1: https://dataimpulse.com/use-cases/ai-proxies/ 2: https://www.scraperapi.com/solutions/ai-data/
    • tremon 2y ago
      Are these "residential proxies" assumed to be infected devices part of a botnet or malicious apps on users' phones?
      • Galanwe 2y ago
        A lot of these residential proxy as a service companies just use regular ISPs to run their automated headless browsers. It doesn't need to be illegal.
      • prmoustache 2y ago
        I guess iot devices and mobile apps. Unethical, definitely. Illegal, no.
  • megadata 2y ago
    > There's already loads of these Examples?