4 ms·

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often th

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.

Y_Y 1mo ago
The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.
- wongarsu 1mo ago
  That's still a notable step up in completely and resource investment for the crawler See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
  - inigyou 1mo ago
    That's not a step up, that is how they work by default.
tptacek 1mo ago
It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.
- deleted 1mo ago
  [deleted]
- yorwba 1mo ago
  For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
- kstrauser 1mo ago
  And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing. I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
  - akerl_ 1mo ago
    Have you compared against adding a JavaScript requirement with no proof of work?
    - kstrauser 1mo ago
      Yes. That helped, but not nearly so much.
- A1kmm 1mo ago
  The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass. An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it. PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
lxgr 1mo ago
> 66% of scrapers are turned away directly. Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users. The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while. [1] https://news.ycombinator.com/item?id=49500040 https://news.ycombinator.com/item?id=49500040
- FridgeSeal 1mo ago
  It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection. Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good. The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
  - lxgr 1mo ago
    Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?
    - done_lurking 1mo ago
      As gp said, it does provide some protection. If it didn't, nobody would use it and there would be nothing to discuss. Making snarky comparisons doesn't do anything.
      - lxgr 1mo ago
        People can remain irrationally convinced about the efficacy of non-solutions to their problems for quite a while, in my experience. And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!
- 1mo ago
graemep 1mo ago
How s that measured? How do you count human users who have been turned away?
lxgr 1mo ago
Why would they even run a browser engine on the devices they're hosted on? All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
semiquaver 1mo ago
> meaningful dent Until you actually do the math and realize that it is not meaningful at all. It’s equivalent to the blogs that have a custom “bot protector” that asks you “what’s 2+2” every time you submit a comment. It might work temporarily as an inconvenience, but nothing more.
oasisbob 1mo ago
> Many of these boxes may not even have the spare CPU power ... I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point. Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol. If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.

Y_Y 1mo ago

The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.

wongarsu 1mo ago
That's still a notable step up in completely and resource investment for the crawler See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
- inigyou 1mo ago
  That's not a step up, that is how they work by default.

tptacek 1mo ago

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.

deleted 1mo ago
[deleted]
yorwba 1mo ago
For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
kstrauser 1mo ago
And yet, in practice, it's true. I was getting about 600K requests to my Forgejo server a day from crawlers making idiotic queries: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing. I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
- akerl_ 1mo ago
  Have you compared against adding a JavaScript requirement with no proof of work?
  - kstrauser 1mo ago
    Yes. That helped, but not nearly so much.
A1kmm 1mo ago
The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass. An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it. PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).

lxgr 1mo ago

> 66% of scrapers are turned away directly. Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users. The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while. [1] https://news.ycombinator.com/item?id=49500040 https://news.ycombinator.com/item?id=49500040

FridgeSeal 1mo ago
It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection. Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good. The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
- lxgr 1mo ago
  Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?
  - done_lurking 1mo ago
    As gp said, it does provide some protection. If it didn't, nobody would use it and there would be nothing to discuss. Making snarky comparisons doesn't do anything.
    - lxgr 1mo ago
      People can remain irrationally convinced about the efficacy of non-solutions to their problems for quite a while, in my experience. And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!
1mo ago

graemep 1mo ago

How s that measured? How do you count human users who have been turned away?

lxgr 1mo ago

Why would they even run a browser engine on the devices they're hosted on? All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.

semiquaver 1mo ago

> meaningful dent Until you actually do the math and realize that it is not meaningful at all. It’s equivalent to the blogs that have a custom “bot protector” that asks you “what’s 2+2” every time you submit a comment. It might work temporarily as an inconvenience, but nothing more.

oasisbob 1mo ago

> Many of these boxes may not even have the spare CPU power ... I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point. Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol. If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.