3 ms·

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual us

by tptacek 2mo ago

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.

deleted 2mo ago
[deleted]
yorwba 2mo ago
For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
kstrauser 2mo ago
And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing. I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
- akerl_ 2mo ago
  Have you compared against adding a JavaScript requirement with no proof of work?
  - kstrauser 2mo ago
    Yes. That helped, but not nearly so much.
A1kmm 2mo ago
The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass. An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it. PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).

by tptacek 2mo ago

deleted 2mo ago

[deleted]

yorwba 2mo ago

For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.

kstrauser 2mo ago

And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing. I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.

akerl_ 2mo ago
Have you compared against adding a JavaScript requirement with no proof of work?
- kstrauser 2mo ago
  Yes. That helped, but not nearly so much.

A1kmm 2mo ago

The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass. An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it. PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).