4 ms·

We need some kind of fail2ban for AI scrapers. Fingerprint them, then share the fingerprint databases via torrent or something.

by teeray 2y ago

michaelcampbell 2y ago
For THIS application, would a boring rate-limiter not help? I mean it won't get rid of the DOS part of this right off, but these are not agents/bots MEANING to DOS a site, so if they get a 429 often enough they should back off on their own, no?
- theamk 2y ago
  they explicitly mention this: > each IP stays below the thresholds for our existing circuit breakers, but the overload is overwhelming. We need something like DCC[0] or Spamhaus block list, but for web, not for email. [0] https://en.wikipedia.org/wiki/Distributed_Checksum_Clearinghouse https://en.wikipedia.org/wiki/Distributed_Checksum_Clearingh...
  - halJordan 2y ago
    There are already legal remedies in place. This isnt a technical problem. We cant advocate for three or four behemoths gatekeeping whitelists they're willing to peer with. Which is how spam is currently controlled.
    - theamk 2y ago
      Huh? It's the opposite. Megacorps have no problem with illegal AI scraping - they have departments full of people whose jobs is scraping detection & prevention, expensive layers who can sue scrapers, and plenty of machines to handle any scraping-related load. But small orgs and individuals have none of that (see the OP article) - how would someone like LWN Jonathan Corbet use "legal remedies" against scraper bots which use fake user agent, a wide variety of IP addresses, and don't announce themselves? Suing random ISPs is going to be prohibitively expensive. It really seems to me that it's time for web to get the same distributed protection mechanisms as email did. It is really a technical problem. If you ever run your own mail server (I used to), you know that it would be impossible without relying on non-commercial/non-profit organizations like SpamHaus and DCC. Specifically, I am hoping for "distributed blocklist" - by itself, Fedora infra or LWN don't see enough traffic to detect a distributed scraper traffic. But if enough small websites pool their logs together via some DCC-like technology, then they might be able to detect scraping farms and block them for good. Throw in some poisoner tech, like Nerpenthes, and this might have good enough false-positive/false-negative rate to be usable.

We need some kind of fail2ban for AI scrapers. Fingerprint them, then share the fingerprint databases via torrent or something.

by teeray 2y ago

We need some kind of fail2ban for AI scrapers. Fingerprint them, then share the fingerprint databases via torrent or something.

michaelcampbell 2y ago

For THIS application, would a boring rate-limiter not help? I mean it won't get rid of the DOS part of this right off, but these are not agents/bots MEANING to DOS a site, so if they get a 429 often enough they should back off on their own, no?

theamk 2y ago
they explicitly mention this: > each IP stays below the thresholds for our existing circuit breakers, but the overload is overwhelming. We need something like DCC[0] or Spamhaus block list, but for web, not for email. [0] https://en.wikipedia.org/wiki/Distributed_Checksum_Clearinghouse https://en.wikipedia.org/wiki/Distributed_Checksum_Clearingh...
- halJordan 2y ago
  There are already legal remedies in place. This isnt a technical problem. We cant advocate for three or four behemoths gatekeeping whitelists they're willing to peer with. Which is how spam is currently controlled.
  - theamk 2y ago
    Huh? It's the opposite. Megacorps have no problem with illegal AI scraping - they have departments full of people whose jobs is scraping detection & prevention, expensive layers who can sue scrapers, and plenty of machines to handle any scraping-related load. But small orgs and individuals have none of that (see the OP article) - how would someone like LWN Jonathan Corbet use "legal remedies" against scraper bots which use fake user agent, a wide variety of IP addresses, and don't announce themselves? Suing random ISPs is going to be prohibitively expensive. It really seems to me that it's time for web to get the same distributed protection mechanisms as email did. It is really a technical problem. If you ever run your own mail server (I used to), you know that it would be impossible without relying on non-commercial/non-profit organizations like SpamHaus and DCC. Specifically, I am hoping for "distributed blocklist" - by itself, Fedora infra or LWN don't see enough traffic to detect a distributed scraper traffic. But if enough small websites pool their logs together via some DCC-like technology, then they might be able to detect scraping farms and block them for good. Throw in some poisoner tech, like Nerpenthes, and this might have good enough false-positive/false-negative rate to be usable.