3 ms·

It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.

It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?

fc417fc802 28d ago
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators. If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
- XorNot 28d ago
  Okay but it's like no RAM at all. Thats the problem: you've got stupid bots causing a bunch of issues - these can be stopped. And then regular users: who now can't really browse your site. And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down. The solution basically makes the favored customer well funded AI scrapers, not users.
  - fc417fc802 28d ago
    > And then regular users: who now can't really browse your site. That's entirely up to how the site operator configures the PoW challenge. > anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters? Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
  - jeroenhd 27d ago
    Regular users store the cookie and do the challenge once. Bots do it over and over again throwing out the cookies so they're not rate-limited. Extreme configurations with high difficulty do make for a frustrating experience on very old devices, but in practice it rarely gets in the way. Only the tail end of what one could consider normal users (those running custom browsers or having JS disabled or nonfunctional) run into issues here, but they'll have these issues with any other bot-blocking mechanism that still offers some protection these days. Apple's "ask your vendor for tokens to exchange for access" web standard gets around this, assuming those bespoke and Javascript-less browsers implement PATs, but that standard seems to be completely dead in the water outside of Safari.
pelcg 28d ago
[dead]
solid_fuel 28d ago
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
- drum55 28d ago
  It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take. last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16) This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
  - bombela 28d ago
    Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
    - xenafan1 27d ago
      [flagged]
  - Jtariiiii 28d ago
    >It’s not stopping anyone. You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
    - gruez 28d ago
      >There are plenty of people demonstrating that Anubis reduces bot traffic. "Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
deleted 28d ago
[deleted]

fc417fc802 28d ago

The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators. If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.

XorNot 28d ago
Okay but it's like no RAM at all. Thats the problem: you've got stupid bots causing a bunch of issues - these can be stopped. And then regular users: who now can't really browse your site. And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down. The solution basically makes the favored customer well funded AI scrapers, not users.
- fc417fc802 28d ago
  > And then regular users: who now can't really browse your site. That's entirely up to how the site operator configures the PoW challenge. > anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters? Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
- jeroenhd 27d ago
  Regular users store the cookie and do the challenge once. Bots do it over and over again throwing out the cookies so they're not rate-limited. Extreme configurations with high difficulty do make for a frustrating experience on very old devices, but in practice it rarely gets in the way. Only the tail end of what one could consider normal users (those running custom browsers or having JS disabled or nonfunctional) run into issues here, but they'll have these issues with any other bot-blocking mechanism that still offers some protection these days. Apple's "ask your vendor for tokens to exchange for access" web standard gets around this, assuming those bespoke and Javascript-less browsers implement PATs, but that standard seems to be completely dead in the water outside of Safari.

pelcg 28d ago

[dead]

solid_fuel 28d ago

The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.

drum55 28d ago
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take. last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16) This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
- bombela 28d ago
  Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
  - xenafan1 27d ago
    [flagged]
- Jtariiiii 28d ago
  >It’s not stopping anyone. You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
  - gruez 28d ago
    >There are plenty of people demonstrating that Anubis reduces bot traffic. "Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.

deleted 28d ago

[deleted]