3 ms·

It sounds like you're saying that it's not the proof-of-work that's stopping AI scrapers, but the fact that Anubis imposes an unusual flow to load the site. If

by agwa 1y ago

It sounds like you're saying that it's not the proof-of-work that's stopping AI scrapers, but the fact that Anubis imposes an unusual flow to load the site. If that's true Anubis should just remove the proof-of-work part, so legitimate human visitors don't have to stare at a loading screen for several seconds while their device wastes electricity.

tptacek 1y ago
Exactly this.
kaszanka 1y ago
This is basically what most of the challenge types in go-away (https://git.gammaspectra.live/git/go-away/wiki/Challenges https://git.gammaspectra.live/git/go-away/wiki/Challenges) do.
- Tmpod 1y ago
  +1 for go-away. It's a bit more involved to configure, but worth the effort imo. It can be considerably more transparent to the user, triggering the nuclear PoW check less often, while being just as effective, in my experience.
amarant 1y ago
I feel like the future will have this, plus ads displayed while the work is done, so websites can profit while they profit.
- silversmith 1y ago
  Every now and then I consider stepping away from the computer job, and becoming a lumberjack. This is one of those moments.
  - zxexz 1y ago
    Worth getting to know the in and outs of forest management now. I don’t think AI will take most tech jobs soon, but they sure as hell are already making them boring.
  - jones89176 1y ago
    my family takes care of a large-ish forest, so I have to help since my early teens. Let me tell you: think twice, it's f*ckin dangerous. Chainsaws, winches, heavy trees falling and breaking in unpredictable ways. I had a couple of close calls myself. Recently a guy from a neighbor village was squashed to death by a root plate that tilted. I often think about quitting tech myself, but becoming a full-time lumberjack is certainly not an alternative for me.
    - silversmith 1y ago
      Hah, I know, been around forests since childhood, seen (and done) plenty of sketchy stuff. For me it averages out to couple days of forest work a year. It's backbreaking labour, and then you deal with the weather. But man, if tech goes straight into cyberpunk dystopia but without the cool gadgets, maybe it is the better alternative.
- JimDabell 1y ago
  adCAPTCHA already does this: https://adcaptcha.com https://adcaptcha.com
  - Tmpod 1y ago
    This is a joke, right? The landing page makes it seem so. I tried the captcha in their login page and it made the entire page, including the puzzle piece slider, run at 2 fps. My god, we do really live in 2025.
  - Aachen 1y ago
    Holy shit. Opening the demo from the menu, it's like captchas and youtube ads had a baby
chrismorgan 1y ago
> If that's true Anubis should just remove the proof-of-work part This is my very strong belief. To make it even clearer how absurd the present situation is, every single one of the proof-of-work systems I’ve looked at has been using SHA-256, which is basically the worst choice possible. Proof-of-work is bad rate limiting which depends on a level playing field between real users and attackers. This is already a doomed endeavour. Using SHA-256 just makes it more obvious: there’s an asymmetry factor in the order of tens of thousands between common real-user hardware and software, and pretty easy attacker hardware and software. You cannot bridge such a divide. If you allow the attacker to augment it with a Bitcoin mining rig, the efficiency disparity factor can go up to tens of millions. These proof-of-work systems are only working because attackers haven’t tried yet. And as long as attackers aren’t trying, you can settle for something much simpler and more transparent. If they were serious about the proof-of-work being the defence, they’d at least have started with something like Argon2d.
- username332211 1y ago
  All this is true, but also somewhat irrelevant. In reality the amount of actual hash work is completely negligible. For usability reasons Anubus only requires that you to go trough a the proof of work flow only once in a given period. (I think the default is once per week.) That's just very little work. Detecting you need to occasionally send a request trough a headless browser far more of a hassle than the PoW. If you prefer LLMs rather than normal internet search, it'll probably consume far more compute as well.
  - rendx 1y ago
    > For usability reasons Anubus only requires that you to go trough a the proof of work flow only once in a given period. (I think the default is once per week.) That's just very little work. If you keep cookies. I do not want to keep cookies for otherwise "stateless" sites. I have maybe a dozen sites whitelisted, every other site loses cookies when I close the tab.
    - account42 1y ago
empath75 1y ago
I don't think anything will stop AI companies for long. They can do spot AI agentic checks of workflows that stop working for some reason and the AI can usually figure out what the problem is and then update the workflow to get around it.

by agwa 1y ago

tptacek 1y ago

Exactly this.

kaszanka 1y ago

This is basically what most of the challenge types in go-away (https://git.gammaspectra.live/git/go-away/wiki/Challenges https://git.gammaspectra.live/git/go-away/wiki/Challenges) do.

Tmpod 1y ago
+1 for go-away. It's a bit more involved to configure, but worth the effort imo. It can be considerably more transparent to the user, triggering the nuclear PoW check less often, while being just as effective, in my experience.

amarant 1y ago

I feel like the future will have this, plus ads displayed while the work is done, so websites can profit while they profit.

silversmith 1y ago
Every now and then I consider stepping away from the computer job, and becoming a lumberjack. This is one of those moments.
- zxexz 1y ago
  Worth getting to know the in and outs of forest management now. I don’t think AI will take most tech jobs soon, but they sure as hell are already making them boring.
- jones89176 1y ago
  my family takes care of a large-ish forest, so I have to help since my early teens. Let me tell you: think twice, it's f*ckin dangerous. Chainsaws, winches, heavy trees falling and breaking in unpredictable ways. I had a couple of close calls myself. Recently a guy from a neighbor village was squashed to death by a root plate that tilted. I often think about quitting tech myself, but becoming a full-time lumberjack is certainly not an alternative for me.
  - silversmith 1y ago
    Hah, I know, been around forests since childhood, seen (and done) plenty of sketchy stuff. For me it averages out to couple days of forest work a year. It's backbreaking labour, and then you deal with the weather. But man, if tech goes straight into cyberpunk dystopia but without the cool gadgets, maybe it is the better alternative.
JimDabell 1y ago
adCAPTCHA already does this: https://adcaptcha.com https://adcaptcha.com
- Tmpod 1y ago
  This is a joke, right? The landing page makes it seem so. I tried the captcha in their login page and it made the entire page, including the puzzle piece slider, run at 2 fps. My god, we do really live in 2025.
- Aachen 1y ago
  Holy shit. Opening the demo from the menu, it's like captchas and youtube ads had a baby

chrismorgan 1y ago

> If that's true Anubis should just remove the proof-of-work part This is my very strong belief. To make it even clearer how absurd the present situation is, every single one of the proof-of-work systems I’ve looked at has been using SHA-256, which is basically the worst choice possible. Proof-of-work is bad rate limiting which depends on a level playing field between real users and attackers. This is already a doomed endeavour. Using SHA-256 just makes it more obvious: there’s an asymmetry factor in the order of tens of thousands between common real-user hardware and software, and pretty easy attacker hardware and software. You cannot bridge such a divide. If you allow the attacker to augment it with a Bitcoin mining rig, the efficiency disparity factor can go up to tens of millions. These proof-of-work systems are only working because attackers haven’t tried yet. And as long as attackers aren’t trying, you can settle for something much simpler and more transparent. If they were serious about the proof-of-work being the defence, they’d at least have started with something like Argon2d.

username332211 1y ago
All this is true, but also somewhat irrelevant. In reality the amount of actual hash work is completely negligible. For usability reasons Anubus only requires that you to go trough a the proof of work flow only once in a given period. (I think the default is once per week.) That's just very little work. Detecting you need to occasionally send a request trough a headless browser far more of a hassle than the PoW. If you prefer LLMs rather than normal internet search, it'll probably consume far more compute as well.
- rendx 1y ago
  > For usability reasons Anubus only requires that you to go trough a the proof of work flow only once in a given period. (I think the default is once per week.) That's just very little work. If you keep cookies. I do not want to keep cookies for otherwise "stateless" sites. I have maybe a dozen sites whitelisted, every other site loses cookies when I close the tab.
  - account42 1y ago

empath75 1y ago

I don't think anything will stop AI companies for long. They can do spot AI agentic checks of workflows that stop working for some reason and the AI can usually figure out what the problem is and then update the workflow to get around it.