3 ms·

Last time I checked, Anubis used SHA256 for PoW. This is very GPU/ASIC friendly, so there's a big disparity between the amount of compute available in a legit b

Last time I checked, Anubis used SHA256 for PoW. This is very GPU/ASIC friendly, so there's a big disparity between the amount of compute available in a legit browser vs a datacentre-scale scraping operation. A more memory-hard "mining" algorithm could help.

jsnell 1y ago
A different algorithm would not help. Here's the basic problem: the fully loaded cost of a server CPU core is ~1 cent/hour. The most latency you can afford to inflict on real users is a couple of seconds. That means the cost of passing a challenge the way the users pass it, with a CPU running Javascript, is about 1/1000th of a cent. And then that single proof of work will let them scrape at a minimum hundreds, but more likely thousands, of pages. So a millionth of a cent per page. How much engineering effort is worth spending on optimizing that? Basically none, certainly not enough to offload to GPUs or ASICs.
- Retr0id 1y ago
  No matter where the bar is there will always be scrapers willing to jump over it, but if you can raise the bar while holding the user-facing cost constant, that's a win.
  - jsnell 1y ago
    No, but what I'm saying is that these scrapers are already not using GPUs or ASICs. It just doesn't make any economical sense to do that in the first place. They are running the same Javascript code on the same commodity CPUs and the same Javascript engine as the real users. So switching to an ASIC-resistant algorithm will not raise the bar. It's just going to be another round of the security theater that proof of work was in the first place.
    - Retr0id 1y ago
      They might not be using GPUs but their servers definitely have finite RAM. Memory-hard PoW reduces the number of concurrent sessions you can maintain per fixed amount of RAM. The more sites get protected by Anubis, the stronger the incentives are for scrapers to actually switch to GPUs etc. It wouldn't take all that much engineering work to hook the webcrypto apis up to a GPU impl (although it would still be fairly inefficient like that). If you're scraping a billion pages then the costs add up.
      - jsnell 1y ago
        The duration you'd need the memory for is a couple of seconds, during which time you're pegging a CPU core on the computation anyway. It is not needed for the entirety of the browsing session. Now, could you construct a challenge that forced the client to keep a ton of data in memory, and then regularly be forced to prove they still have that data during the entire session? I don't think so. The problem is that for that kind of intermittent proof scenario there's no need to actually keep the data in low latency memory. It can just be stored on disk, and paged in when needed (not often). It's a very different access pattern from the cryptocurrency use case.
beefnugs 1y ago
Sorry but it is actually a completely wrong solution. This is not a "make it expensive for spammers" problem. They only need to download the source code once, with a background process that doesn't matter if it takes hours. Besides the "got the source code for training data" , the other access scenario is just downloading to an end users "agent" Which again, the end user is running something in the background, doesn't care how long it takes, how much it costs, its not a volume or spam type problem

jsnell 1y ago

A different algorithm would not help. Here's the basic problem: the fully loaded cost of a server CPU core is ~1 cent/hour. The most latency you can afford to inflict on real users is a couple of seconds. That means the cost of passing a challenge the way the users pass it, with a CPU running Javascript, is about 1/1000th of a cent. And then that single proof of work will let them scrape at a minimum hundreds, but more likely thousands, of pages. So a millionth of a cent per page. How much engineering effort is worth spending on optimizing that? Basically none, certainly not enough to offload to GPUs or ASICs.

Retr0id 1y ago
No matter where the bar is there will always be scrapers willing to jump over it, but if you can raise the bar while holding the user-facing cost constant, that's a win.
- jsnell 1y ago
  No, but what I'm saying is that these scrapers are already not using GPUs or ASICs. It just doesn't make any economical sense to do that in the first place. They are running the same Javascript code on the same commodity CPUs and the same Javascript engine as the real users. So switching to an ASIC-resistant algorithm will not raise the bar. It's just going to be another round of the security theater that proof of work was in the first place.
  - Retr0id 1y ago
    They might not be using GPUs but their servers definitely have finite RAM. Memory-hard PoW reduces the number of concurrent sessions you can maintain per fixed amount of RAM. The more sites get protected by Anubis, the stronger the incentives are for scrapers to actually switch to GPUs etc. It wouldn't take all that much engineering work to hook the webcrypto apis up to a GPU impl (although it would still be fairly inefficient like that). If you're scraping a billion pages then the costs add up.
    - jsnell 1y ago
      The duration you'd need the memory for is a couple of seconds, during which time you're pegging a CPU core on the computation anyway. It is not needed for the entirety of the browsing session. Now, could you construct a challenge that forced the client to keep a ton of data in memory, and then regularly be forced to prove they still have that data during the entire session? I don't think so. The problem is that for that kind of intermittent proof scenario there's no need to actually keep the data in low latency memory. It can just be stored on disk, and paged in when needed (not often). It's a very different access pattern from the cryptocurrency use case.

beefnugs 1y ago

Sorry but it is actually a completely wrong solution. This is not a "make it expensive for spammers" problem. They only need to download the source code once, with a background process that doesn't matter if it takes hours. Besides the "got the source code for training data" , the other access scenario is just downloading to an end users "agent" Which again, the end user is running something in the background, doesn't care how long it takes, how much it costs, its not a volume or spam type problem