3 ms·

There are already legal remedies in place. This isnt a technical problem. We cant advocate for three or four behemoths gatekeeping whitelists they're willing to

• theamk 2y ago
  Huh? It's the opposite. Megacorps have no problem with illegal AI scraping - they have departments full of people whose jobs is scraping detection & prevention, expensive layers who can sue scrapers, and plenty of machines to handle any scraping-related load. But small orgs and individuals have none of that (see the OP article) - how would someone like LWN Jonathan Corbet use "legal remedies" against scraper bots which use fake user agent, a wide variety of IP addresses, and don't announce themselves? Suing random ISPs is going to be prohibitively expensive. It really seems to me that it's time for web to get the same distributed protection mechanisms as email did. It is really a technical problem. If you ever run your own mail server (I used to), you know that it would be impossible without relying on non-commercial/non-profit organizations like SpamHaus and DCC. Specifically, I am hoping for "distributed blocklist" - by itself, Fedora infra or LWN don't see enough traffic to detect a distributed scraper traffic. But if enough small websites pool their logs together via some DCC-like technology, then they might be able to detect scraping farms and block them for good. Throw in some poisoner tech, like Nerpenthes, and this might have good enough false-positive/false-negative rate to be usable.