9 ms·

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purp

Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely. Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors. (That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)

inigyou 2mo ago
It does it by blocking the scrapers because they are really stupid scrapers. go-away blocks them too, by seeing if they load images, a much quicker test.
TomatoCo 2mo ago
There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.
- dullcrisp 2mo ago
  If compute isn’t the bottleneck then it wouldn’t make it 100x more expensive to scrape. Of course if it does, then it’ll be effective.
- VladVladikoff 2mo ago
  In my experience, not really. Each asset is made by a new proxy, eg some TV somewhere, they send every new request via a new IP address, which is a new machine, and it doesn’t matter how long the response takes, they have already rotated to the next ip immediately and sent another request. Most of these providers have millions of IPs, and it generally doesn’t take millions of requests to scrape a website (unless it’s really big!)
  - inigyou 2mo ago
    And each new IP triggers a new Anubis challenge. Isn't it great?
    - microtonal 2mo ago
      So, once the challenges are solved by the residential proxies (with optimized native implementations [1]). Some unsuspecting TV owner is paying with their electricity bill (not the companies doing the scraping) and we all suffer through challenges. Yes, great outcome! </s> [1] https://lock.cmpxchg8b.com/anubis.html https://lock.cmpxchg8b.com/anubis.html
      - inigyou 2mo ago
        Proxy networks just proxy. They don't let their client run compute on the proxy. That isn't even a possibility in socks5 protocol.
ddtaylor 2mo ago
There is no distinction between the two. If one can do it the other can just as easily. I know multiple people right now that can scrape any a site very easily. I do this at scale and it took me very little time to set up and almost no resistance. So anyone who thinks that this is difficult or you're preventing people from doing this at scale, you're wrong.
jsnell 2mo ago
Except it does not actually increase the cost of scraping meaningfully. Compute is really cheap. The compute for minting an Anubis cookie will cost less than a thousandth of a cent even assuming the attacker uses the same JS implementation of proof of work rather than an optimized native implementation. That cookie can then be used for hundreds of requests. How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. Just the bandwidth from a residential proxy will cost the scraper orders of magnitude more. Proof of work just isn't a viable counter-abuse challenge, even for something as low-yield as scraping. (It might be economically viable as a counter against some types of DDoS attacks, since they're even lower yield. But in practice it just moves the attack surface to the proof of work validation service.)
- ragall 2mo ago
  > How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. In practice, this would seem to be false, according to the admins of said sites.
- xboxnolifes 2mo ago
  Your theoretical counterargument falls apart by the reality of just putting up anubis and comparing the before and after. I don't understand why this argument shows up in every thread about anubis. There are plenty of people and orgs who have empirical before and after results. We don't need theoretical arguments when there exists actual data.
  - TylerE 2mo ago
    Captchas worked for a few years too. Then computers got better at solving them than humans, and we're still dealing with the fucking useless things decades later.
  - inigyou 2mo ago
    It's not because of the PoW though, it's just because the scraper doesn't run JavaScript. The alternative package called go-away does these tests without the PoW.
  - 2mo ago

inigyou 2mo ago

It does it by blocking the scrapers because they are really stupid scrapers. go-away blocks them too, by seeing if they load images, a much quicker test.

TomatoCo 2mo ago

There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.

dullcrisp 2mo ago
If compute isn’t the bottleneck then it wouldn’t make it 100x more expensive to scrape. Of course if it does, then it’ll be effective.
VladVladikoff 2mo ago
In my experience, not really. Each asset is made by a new proxy, eg some TV somewhere, they send every new request via a new IP address, which is a new machine, and it doesn’t matter how long the response takes, they have already rotated to the next ip immediately and sent another request. Most of these providers have millions of IPs, and it generally doesn’t take millions of requests to scrape a website (unless it’s really big!)
- inigyou 2mo ago
  And each new IP triggers a new Anubis challenge. Isn't it great?
  - microtonal 2mo ago
    So, once the challenges are solved by the residential proxies (with optimized native implementations [1]). Some unsuspecting TV owner is paying with their electricity bill (not the companies doing the scraping) and we all suffer through challenges. Yes, great outcome! </s> [1] https://lock.cmpxchg8b.com/anubis.html https://lock.cmpxchg8b.com/anubis.html
    - inigyou 2mo ago
      Proxy networks just proxy. They don't let their client run compute on the proxy. That isn't even a possibility in socks5 protocol.

ddtaylor 2mo ago

There is no distinction between the two. If one can do it the other can just as easily. I know multiple people right now that can scrape any a site very easily. I do this at scale and it took me very little time to set up and almost no resistance. So anyone who thinks that this is difficult or you're preventing people from doing this at scale, you're wrong.

jsnell 2mo ago

Except it does not actually increase the cost of scraping meaningfully. Compute is really cheap. The compute for minting an Anubis cookie will cost less than a thousandth of a cent even assuming the attacker uses the same JS implementation of proof of work rather than an optimized native implementation. That cookie can then be used for hundreds of requests. How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. Just the bandwidth from a residential proxy will cost the scraper orders of magnitude more. Proof of work just isn't a viable counter-abuse challenge, even for something as low-yield as scraping. (It might be economically viable as a counter against some types of DDoS attacks, since they're even lower yield. But in practice it just moves the attack surface to the proof of work validation service.)

ragall 2mo ago
> How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. In practice, this would seem to be false, according to the admins of said sites.
xboxnolifes 2mo ago
Your theoretical counterargument falls apart by the reality of just putting up anubis and comparing the before and after. I don't understand why this argument shows up in every thread about anubis. There are plenty of people and orgs who have empirical before and after results. We don't need theoretical arguments when there exists actual data.
- TylerE 2mo ago
  Captchas worked for a few years too. Then computers got better at solving them than humans, and we're still dealing with the fucking useless things decades later.
- inigyou 2mo ago
  It's not because of the PoW though, it's just because the scraper doesn't run JavaScript. The alternative package called go-away does these tests without the PoW.
- 2mo ago