5 ms·

The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on c

The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.

drum55 28d ago
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take. last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16) This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
- bombela 28d ago
  Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
  - xenafan1 27d ago
    [flagged]
- Jtariiiii 28d ago
  >It’s not stopping anyone. You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
  - gruez 28d ago
    >There are plenty of people demonstrating that Anubis reduces bot traffic. "Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
    - Jtariiiii 28d ago
      "it's not stopping anyone" and "it is stopping some people" are mutually exclusive statements.
      - gruez 28d ago
        You're selectively quoting the comment. It's clearly talking about the PoW aspect not being a serious impediment.
        xboxnolifes 27d ago
        Who cares if its PoW or something else? Does it matter in any meaningful way?
        account42 27d ago
        It matters if the PoW component is negatively impacting real users, which it is.
    - wiredfool 27d ago
      IP blocks don't work, because they're using proxy networks so that you see an ip address 1 or 2 times within 10 minutes. They have effectively infinite ip addresses. (actually, looking at my data from today, I think this relationship holds over ~3 hours, where we're seeing ip address cardinality at about 1/2 of the hits.) * Sometimes there's a pattern to the country. Oftentimes, not. * User-agent, rotated between common, valid, current web browsers. * Other headers, sec-*, accept, etc, generally valid and rotating. * Bots will load the site to saturation in a denial of wallet attack. The only thing that's specific is: * urls have a pattern. * it's obviously invalid traffic. (non-bot traffic on my sites does not go from 0 to 200r/sec on the search interface in seconds. It does not go away that fast either)
  - drum55 28d ago
    That’s because its weird and mostly irrelevant, it’s nothing to do with the proof of work being any sort of economic impediment. It takes Claude a few minutes to write a near instant solver, that’s the maximum proof of work cost this adds.
    - yborg 27d ago
      This strain of nihilism about anti-bot efforts baffles me. Cloudflare built a gigantic business on stopping DDoS attacks. There will never be good faith self-restraint by AI bros, there is too much money at stake and no regulation, doing nothing means your only alternative is disconnect from the Internet. Anubis introduces enough friction that some percentage of the lunacy is throttled and possibly give your server a fighting chance.
gruez 28d ago
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages. There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
- account42 27d ago
  Yes, at the end of the day this is DRM and as always it affects legitimate users more than it does those it is meant to block.
Aachen 28d ago
> requiring 1 second of compute before allowing a page load After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
- jdlshore 28d ago
  The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
  - gruez 28d ago
    >If scrapers reuse a cookie, traditional anti-abuse mechanisms will work. Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
    - embedding-shape 28d ago
      > Scrapers can't change their strategy? Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
      - gruez 28d ago
        That argument also works against anubis. Why bother forcing clients to solve a PoW challenge, when you're fighting a bunch of unsophisticated attackers that will be deterred by a javascript challenge that requires them to solve `result = x + y`?

drum55 28d ago

It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take. last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16) This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.

bombela 28d ago
Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
- xenafan1 27d ago
  [flagged]
Jtariiiii 28d ago
>It’s not stopping anyone. You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
- gruez 28d ago
  >There are plenty of people demonstrating that Anubis reduces bot traffic. "Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
  - Jtariiiii 28d ago
    "it's not stopping anyone" and "it is stopping some people" are mutually exclusive statements.
    - gruez 28d ago
      You're selectively quoting the comment. It's clearly talking about the PoW aspect not being a serious impediment.
      - xboxnolifes 27d ago
        Who cares if its PoW or something else? Does it matter in any meaningful way?
        account42 27d ago
        It matters if the PoW component is negatively impacting real users, which it is.
  - wiredfool 27d ago
    IP blocks don't work, because they're using proxy networks so that you see an ip address 1 or 2 times within 10 minutes. They have effectively infinite ip addresses. (actually, looking at my data from today, I think this relationship holds over ~3 hours, where we're seeing ip address cardinality at about 1/2 of the hits.) * Sometimes there's a pattern to the country. Oftentimes, not. * User-agent, rotated between common, valid, current web browsers. * Other headers, sec-*, accept, etc, generally valid and rotating. * Bots will load the site to saturation in a denial of wallet attack. The only thing that's specific is: * urls have a pattern. * it's obviously invalid traffic. (non-bot traffic on my sites does not go from 0 to 200r/sec on the search interface in seconds. It does not go away that fast either)
- drum55 28d ago
  That’s because its weird and mostly irrelevant, it’s nothing to do with the proof of work being any sort of economic impediment. It takes Claude a few minutes to write a near instant solver, that’s the maximum proof of work cost this adds.
  - yborg 27d ago
    This strain of nihilism about anti-bot efforts baffles me. Cloudflare built a gigantic business on stopping DDoS attacks. There will never be good faith self-restraint by AI bros, there is too much money at stake and no regulation, doing nothing means your only alternative is disconnect from the Internet. Anubis introduces enough friction that some percentage of the lunacy is throttled and possibly give your server a fighting chance.

gruez 28d ago

>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages. There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.

account42 27d ago
Yes, at the end of the day this is DRM and as always it affects legitimate users more than it does those it is meant to block.

Aachen 28d ago

> requiring 1 second of compute before allowing a page load After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing

jdlshore 28d ago
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
- gruez 28d ago
  >If scrapers reuse a cookie, traditional anti-abuse mechanisms will work. Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
  - embedding-shape 28d ago
    > Scrapers can't change their strategy? Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
    - gruez 28d ago
      That argument also works against anubis. Why bother forcing clients to solve a PoW challenge, when you're fighting a bunch of unsophisticated attackers that will be deterred by a javascript challenge that requires them to solve `result = x + y`?