4 ms·

> As long as a given client is not abusing your systems, then why do you care if the client is a human? Well, that's the rub. The bots are abusing the systems.

> As long as a given client is not abusing your systems, then why do you care if the client is a human? Well, that's the rub. The bots are abusing the systems. The bots are accessing the contents at rates thousands of times faster and more often than humans. The bots also have access patterns unlike your expected human audience (downloading gigabytes or terabytes of data multiples times, over and over). And these bots aren't some being with rights. They're tools unleashed by humans. It's humans abusing the systems. These are anti-abuse measures.

bbor 2y ago
Well, that's the meta-rub: if they're abusing, block abuse. Rate limits are far simpler, anyway! In the interest of bringing the AI bickering to HN: I think one could accurately characterize "block bots just in case they choose to request too much data" as discrimination! Robots of course don't have any rights so it's not wrong, but it certainly might be unwise.
- inejge 2y ago
  > Rate limits are far simpler, anyway! Not when the bots are actively programmed to thwart them by using far-flung IP address carousels, request pacing, spoofed user agents and similar techniques. It's open war these days.
  - parineum 2y ago
    Request pacing sounds intentionally unabusive.
    - rollcat 2y ago
      It's called DDoS. DDoS is abusive.
    - j16sdiz 2y ago
      It is not bringing down your server, but they are taking 80%+ of your bandwidth budget. Does this count as abuse?
      - ithkuil 2y ago
        Isn't that what a rate limiter would address?
        mkl 2y ago
        Not when the traffic is coming from 10s of thousands of IP addresses, with very few requests from each one: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
        KronisLV 2y ago
        That very much reads like the rant of someone who is sick and tired of the state of things. I’m afraid that it doesn’t change anything in of itself and any sorts of solutions to only allow the users that you’re okay with are what’s direly needed all across the web. Though reading about the people trying to mine crypto on a CI solution, it feels that sometimes it won’t just be LLM scrapers that you need to protect against but any number of malicious people. At that point, you might as well run an invite only community.
        bayindirh 2y ago
        Source Hut implemented Anubis, and it works so well. I mostly never see the waiting screen. And after it whitelists me for a very long time, so I work without any limitations.
        KronisLV 2y ago
        That’s great to hear and Anubis seems cool! I just worry about the idea of running public/free services on the web, due to the potential for misuse and bad actors, though making things paid also seems sensible, e.g. what was linked: https://man.sr.ht/ops/builds.sr.ht-migration.md https://man.sr.ht/ops/builds.sr.ht-migration.md
immibis 2y ago
Then you look up their IP address's abuse contact, send an email and get them to either stop attacking you or get booted off the internet so they can't attack you. And if that doesn't happen, you go to their ISP's ISP and get their ISP booted off the Internet. Actual ISPs and hosting providers take abuse reports extremely seriously, mostly because they're terrified of getting kicked off by their ISP. And there's no end to that - just a chain of ISPs from them to you and you might end with convincing your ISP or some intermediary to block traffic from them. However, as we've seen recently, rules don't apply if enough money is involved. But I'm not sure if these shitty interim solutions come from ISPs ignoring abuse when money is involved, or from not knowing that abuse reporting is taken seriously to begin with. Anyone know if it's legal to return a never-ending stream of /dev/urandom based on the user-agent?
- sussmannbaka 2y ago
  Please, read literally any article about the ongoing problem. The IPs are basically random, come from residential blocks, requests don’t reuse the same IP more than a bunch of times.
  - immibis 2y ago
    Are you sure that's AI? I get requests that are overtly from AI crawlers, and almost no other requests. Certainly all of the high-volume crawler-like requests overtly say that they're from crawlers. And those residential proxy services cost their customer around $0.50/GB up to $20/GB. Do with that knowledge what you will.
- zinekeller 2y ago
  > Then you look up their IP address's abuse contact, send an email and get them to either stop attacking you or get booted off the internet so they can't attack you. You will be surprised on how many ISPs will not respond. Sure, Hetzner will respond, but these abusers are not using Hetzner at all. If you actually studied the actual problem, these are residential ISPs in various countries (including in US and Europe, mind you). At best the ISP will respond one-by-one to their customers and scan their computers (and at this point the abusers have already switched to another IP block) and at worst the ISP literally has no capability to control this because they cannot trace their CGNATted connections (short of blocking connections to your site, which is definitely nuclear). > And if that doesn't happen, you go to their ISP's ISP and get their ISP booted off the Internet. Again, the IP blocks are rotated, so by the time that they would respond you need to do the whole reporting rigomarole again. Additionally, these ISPs would instead suggest to blackhole these requests or to utilize a commercial solution (aka using Cloudflare or something else), because at the end of the day the residential ISPs are national entites that would quite literally trigger geopolitcal concerns if you disconnected them.

bbor 2y ago

Well, that's the meta-rub: if they're abusing, block abuse. Rate limits are far simpler, anyway! In the interest of bringing the AI bickering to HN: I think one could accurately characterize "block bots just in case they choose to request too much data" as discrimination! Robots of course don't have any rights so it's not wrong, but it certainly might be unwise.

inejge 2y ago
> Rate limits are far simpler, anyway! Not when the bots are actively programmed to thwart them by using far-flung IP address carousels, request pacing, spoofed user agents and similar techniques. It's open war these days.
- parineum 2y ago
  Request pacing sounds intentionally unabusive.
  - rollcat 2y ago
    It's called DDoS. DDoS is abusive.
  - j16sdiz 2y ago
    It is not bringing down your server, but they are taking 80%+ of your bandwidth budget. Does this count as abuse?
    - ithkuil 2y ago
      Isn't that what a rate limiter would address?
      - mkl 2y ago
        Not when the traffic is coming from 10s of thousands of IP addresses, with very few requests from each one: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
        KronisLV 2y ago
        That very much reads like the rant of someone who is sick and tired of the state of things. I’m afraid that it doesn’t change anything in of itself and any sorts of solutions to only allow the users that you’re okay with are what’s direly needed all across the web. Though reading about the people trying to mine crypto on a CI solution, it feels that sometimes it won’t just be LLM scrapers that you need to protect against but any number of malicious people. At that point, you might as well run an invite only community.
        bayindirh 2y ago
        Source Hut implemented Anubis, and it works so well. I mostly never see the waiting screen. And after it whitelists me for a very long time, so I work without any limitations.
        KronisLV 2y ago
        That’s great to hear and Anubis seems cool! I just worry about the idea of running public/free services on the web, due to the potential for misuse and bad actors, though making things paid also seems sensible, e.g. what was linked: https://man.sr.ht/ops/builds.sr.ht-migration.md https://man.sr.ht/ops/builds.sr.ht-migration.md

immibis 2y ago

Then you look up their IP address's abuse contact, send an email and get them to either stop attacking you or get booted off the internet so they can't attack you. And if that doesn't happen, you go to their ISP's ISP and get their ISP booted off the Internet. Actual ISPs and hosting providers take abuse reports extremely seriously, mostly because they're terrified of getting kicked off by their ISP. And there's no end to that - just a chain of ISPs from them to you and you might end with convincing your ISP or some intermediary to block traffic from them. However, as we've seen recently, rules don't apply if enough money is involved. But I'm not sure if these shitty interim solutions come from ISPs ignoring abuse when money is involved, or from not knowing that abuse reporting is taken seriously to begin with. Anyone know if it's legal to return a never-ending stream of /dev/urandom based on the user-agent?

sussmannbaka 2y ago
Please, read literally any article about the ongoing problem. The IPs are basically random, come from residential blocks, requests don’t reuse the same IP more than a bunch of times.
- immibis 2y ago
  Are you sure that's AI? I get requests that are overtly from AI crawlers, and almost no other requests. Certainly all of the high-volume crawler-like requests overtly say that they're from crawlers. And those residential proxy services cost their customer around $0.50/GB up to $20/GB. Do with that knowledge what you will.
zinekeller 2y ago
> Then you look up their IP address's abuse contact, send an email and get them to either stop attacking you or get booted off the internet so they can't attack you. You will be surprised on how many ISPs will not respond. Sure, Hetzner will respond, but these abusers are not using Hetzner at all. If you actually studied the actual problem, these are residential ISPs in various countries (including in US and Europe, mind you). At best the ISP will respond one-by-one to their customers and scan their computers (and at this point the abusers have already switched to another IP block) and at worst the ISP literally has no capability to control this because they cannot trace their CGNATted connections (short of blocking connections to your site, which is definitely nuclear). > And if that doesn't happen, you go to their ISP's ISP and get their ISP booted off the Internet. Again, the IP blocks are rotated, so by the time that they would respond you need to do the whole reporting rigomarole again. Additionally, these ISPs would instead suggest to blackhole these requests or to utilize a commercial solution (aka using Cloudflare or something else), because at the end of the day the residential ISPs are national entites that would quite literally trigger geopolitcal concerns if you disconnected them.