4 ms·

> In retrospect implementing the proof of work challenge may have been a mistake Why? What would the alternative have been?

tux3 1y ago
It does two things: Force everyone (including scrapers) to run a real JS engine, and force everyone to solve the challenge. The first effect is great, because it's a lot more annoying to bring up a full browser environment in your scraper than just run a curl command. But the actual proof of work only takes about 10ms on a server in native code, while it can take multiple seconds on a low-end phone. Given the companies in questions are building entire data centers to house all their GPUs, an extra 10ms per web-page is not a problem for them. They're going to spend orders of magnitude more compute actually training on the content they scraped, than solving the challenge. It's mostly the inconvenience of adapting to Anubis's JS requirements that held them back for a while, but the PoW difficulty mostly slowed down real users.
- mhh__ 1y ago
  I'd have thought that anyone doing serious scraping has been driving a browser for years at this point. you can even get a curl fork that drives a browser under the hood
- o11c 1y ago
  It does 2.5 things actually - it counters the aggressive use of proxies by scrapers.
alright2565 1y ago
In part because this particular proof of work is absolutely trivial at scale, with commercial hardware able to do 390TH/s, while your typical phone would only be able to do a million and still have acceptable latency.
MBCook 1y ago
They also suggest maybe “proof of React” would be better with a link to this rough proof of concept: https://github.com/TecharoHQ/anubis/pull/1038 https://github.com/TecharoHQ/anubis/pull/1038 Could someone explain how this would help stop scrapers? If you’re just running the page JS wouldn’t this run too and let you through?
- fluoridation 1y ago
  Low-effort scrapers don't run JS, they just fetch static content.
  - MBCook 1y ago
    But then they couldn’t get past the current Anubis. Sonia the idea it would just be cheaper for clients?
    - fluoridation 1y ago
      That's the idea. Impose software requirements on the client instead of computational requirements.
- ranger_danger 1y ago
  They admitted that this was a 'shitpost'. > how this would help stop scrapers I think anubis bases its purpose on some flawed assumptions: - that most scrapers aren't headless browsers - that they don't have access to millions of different IPs across the world from big/shady proxy companies - that this can help with a real network-level DDoS - that scrapers will give up if the requests become 'too expensive' - that they aren't contributing to warming the planet I'm sure there does exist some older bots that are not smart and don't use headless browsers, but especially with newer tech/AI crawlers/etc., I don't think this is a realistic majority assumption anymore.
zetanor 1y ago
In practice, any automated work that a real user is willing to wait through will be trivial to accomplish for an organization which scrapes the entire Internet. The real weight behind Anubis is the Javascript gate, not the PoW. It might as well just fetch() into browser.cookies.set().
jsnell 1y ago
An unavoidable aspect of abuse problems is that there is no perfect solution. As the defender, you’re always making a precision vs. recall tradeoff. After you’ve picked off the low hanging fruit, most of the time the only way to increase recall (i.e. catch more abuse) is by reducing the precision (i.e. having more false positives, where a good user is falsely considered an abuser). In an adversarial engineering domain neither the problems or solutions are static. If by some miracle you have a perfect solution at one point in time, the adversaries will quickly adapt, and your solution stops being perfect. So you’ll mostly be playing the game in this shifting gray area of maybe legit, maybe abusive cases. Since you can’t perfectly classify them (if you could, they wouldn’t be in the gray area), the options are basically to either block all of them, allow all of them, or issue them a challenge that the user must pass to be allowed. The first two options tend to be unacceptable in the gray area, so issuing a challenge that the client must pass is usually the preferred option. A good counter-abuse challenge is something that has at least one of the following properties: 1. It costs more to pass than the economic value that the adversary can extract from the service, but not so much that the legitimate users won’t be willing to pay it. 2. It proves control of a scarce resource without necessarily having to spend that resource, but at least in such a way that the same scarce resource can’t be used to pass unlimited challenges. 3. It produces additional signals that can be used to meaningfully improve the precision/recall tradeoff. And proof of work does none of those. The last two by construction, since compute is about the most fungible resource in the world. The last doesn't work since it's impossible to balance the difficulty factor such that it imposes a cost the attacker would notice but would be acceptable to the defender. If you add 10s to the latency for your worst-case real users (already too long), it'll cost about $0.01/1k solves. That's not a deterrent to any kind of abuse. So proof of work just is a really bad fit for this specific use case. The only advantage is that it is easy to implement, but that's a very short term benefit.
tptacek 1y ago
Without getting into the alternatives: scraper defense isn't a viable proof of work setting, because there's no asymmetry to exploit. You're imposing exactly the same cost on legit users as you are on scrapers. Economies of scale mean that the marginal cost for your adversary is actually significantly lower than for your real users. What the Anubis POW system is doing right now is exploiting the fact that there's been no need for crawlers to be anything but naive. But the cost to make them sophisticated enough to defeat the POW system is quite low, and when that happens, the POW will just be annoying legit users for no benefit. I don't know if "mistake" is the word I'd use for it. It's not a whole lot of code! It's a reasonable first step to force crawlers to emulate a tiny fraction of a real browser. But as it evolves, it should evolve away from burning compute, because that's playing to lose.
- reisse 1y ago
  Wait, but there is an asymmetry. Legitimate user spends at least a dozen seconds on a page, they don't care about 10ms overhead. For a scraper, however, 10ms overhead can easily be 10x the time it spends on a page overall - the scraper is now ten times slower. However the exact PoW implementation (hash) chosen by Anubis might significantly reduce this asymmetry, because the calculation speed is highly dependent on hardware.
  - tptacek 1y ago
    No, I don't think this is accurate. You have to look at both the cost and the benefit. If you're an AI scraper, it's literally just "what does the marginal next token of training data cost me" --- the answer is: the same as the marginal next token of content costs a reader. Tavis Ormandy went into more detail on the math here, but it's not great!
    - comex 1y ago
      I don’t understand what you mean. Training an LLM requires orders of magnitude more tokens than any one human will ever read. Perhaps an AI company can amortize across all their users, but it would still represent a substantial cost. And I’m pretty sure the big AI companies don’t rely on abusive scraping (i.e. ignoring robots.txt), so the companies doing the scraping may not have a lot of users anyway.

tux3 1y ago

It does two things: Force everyone (including scrapers) to run a real JS engine, and force everyone to solve the challenge. The first effect is great, because it's a lot more annoying to bring up a full browser environment in your scraper than just run a curl command. But the actual proof of work only takes about 10ms on a server in native code, while it can take multiple seconds on a low-end phone. Given the companies in questions are building entire data centers to house all their GPUs, an extra 10ms per web-page is not a problem for them. They're going to spend orders of magnitude more compute actually training on the content they scraped, than solving the challenge. It's mostly the inconvenience of adapting to Anubis's JS requirements that held them back for a while, but the PoW difficulty mostly slowed down real users.

mhh__ 1y ago
I'd have thought that anyone doing serious scraping has been driving a browser for years at this point. you can even get a curl fork that drives a browser under the hood
o11c 1y ago
It does 2.5 things actually - it counters the aggressive use of proxies by scrapers.

alright2565 1y ago

In part because this particular proof of work is absolutely trivial at scale, with commercial hardware able to do 390TH/s, while your typical phone would only be able to do a million and still have acceptable latency.

MBCook 1y ago

They also suggest maybe “proof of React” would be better with a link to this rough proof of concept: https://github.com/TecharoHQ/anubis/pull/1038 https://github.com/TecharoHQ/anubis/pull/1038 Could someone explain how this would help stop scrapers? If you’re just running the page JS wouldn’t this run too and let you through?

fluoridation 1y ago
Low-effort scrapers don't run JS, they just fetch static content.
- MBCook 1y ago
  But then they couldn’t get past the current Anubis. Sonia the idea it would just be cheaper for clients?
  - fluoridation 1y ago
    That's the idea. Impose software requirements on the client instead of computational requirements.
ranger_danger 1y ago
They admitted that this was a 'shitpost'. > how this would help stop scrapers I think anubis bases its purpose on some flawed assumptions: - that most scrapers aren't headless browsers - that they don't have access to millions of different IPs across the world from big/shady proxy companies - that this can help with a real network-level DDoS - that scrapers will give up if the requests become 'too expensive' - that they aren't contributing to warming the planet I'm sure there does exist some older bots that are not smart and don't use headless browsers, but especially with newer tech/AI crawlers/etc., I don't think this is a realistic majority assumption anymore.

zetanor 1y ago

In practice, any automated work that a real user is willing to wait through will be trivial to accomplish for an organization which scrapes the entire Internet. The real weight behind Anubis is the Javascript gate, not the PoW. It might as well just fetch() into browser.cookies.set().

jsnell 1y ago

An unavoidable aspect of abuse problems is that there is no perfect solution. As the defender, you’re always making a precision vs. recall tradeoff. After you’ve picked off the low hanging fruit, most of the time the only way to increase recall (i.e. catch more abuse) is by reducing the precision (i.e. having more false positives, where a good user is falsely considered an abuser). In an adversarial engineering domain neither the problems or solutions are static. If by some miracle you have a perfect solution at one point in time, the adversaries will quickly adapt, and your solution stops being perfect. So you’ll mostly be playing the game in this shifting gray area of maybe legit, maybe abusive cases. Since you can’t perfectly classify them (if you could, they wouldn’t be in the gray area), the options are basically to either block all of them, allow all of them, or issue them a challenge that the user must pass to be allowed. The first two options tend to be unacceptable in the gray area, so issuing a challenge that the client must pass is usually the preferred option. A good counter-abuse challenge is something that has at least one of the following properties: 1. It costs more to pass than the economic value that the adversary can extract from the service, but not so much that the legitimate users won’t be willing to pay it. 2. It proves control of a scarce resource without necessarily having to spend that resource, but at least in such a way that the same scarce resource can’t be used to pass unlimited challenges. 3. It produces additional signals that can be used to meaningfully improve the precision/recall tradeoff. And proof of work does none of those. The last two by construction, since compute is about the most fungible resource in the world. The last doesn't work since it's impossible to balance the difficulty factor such that it imposes a cost the attacker would notice but would be acceptable to the defender. If you add 10s to the latency for your worst-case real users (already too long), it'll cost about $0.01/1k solves. That's not a deterrent to any kind of abuse. So proof of work just is a really bad fit for this specific use case. The only advantage is that it is easy to implement, but that's a very short term benefit.

tptacek 1y ago

Without getting into the alternatives: scraper defense isn't a viable proof of work setting, because there's no asymmetry to exploit. You're imposing exactly the same cost on legit users as you are on scrapers. Economies of scale mean that the marginal cost for your adversary is actually significantly lower than for your real users. What the Anubis POW system is doing right now is exploiting the fact that there's been no need for crawlers to be anything but naive. But the cost to make them sophisticated enough to defeat the POW system is quite low, and when that happens, the POW will just be annoying legit users for no benefit. I don't know if "mistake" is the word I'd use for it. It's not a whole lot of code! It's a reasonable first step to force crawlers to emulate a tiny fraction of a real browser. But as it evolves, it should evolve away from burning compute, because that's playing to lose.

reisse 1y ago
Wait, but there is an asymmetry. Legitimate user spends at least a dozen seconds on a page, they don't care about 10ms overhead. For a scraper, however, 10ms overhead can easily be 10x the time it spends on a page overall - the scraper is now ten times slower. However the exact PoW implementation (hash) chosen by Anubis might significantly reduce this asymmetry, because the calculation speed is highly dependent on hardware.
- tptacek 1y ago
  No, I don't think this is accurate. You have to look at both the cost and the benefit. If you're an AI scraper, it's literally just "what does the marginal next token of training data cost me" --- the answer is: the same as the marginal next token of content costs a reader. Tavis Ormandy went into more detail on the math here, but it's not great!
  - comex 1y ago
    I don’t understand what you mean. Training an LLM requires orders of magnitude more tokens than any one human will ever read. Perhaps an AI company can amortize across all their users, but it would still represent a substantial cost. And I’m pretty sure the big AI companies don’t rely on abusive scraping (i.e. ignoring robots.txt), so the companies doing the scraping may not have a lot of users anyway.