4 ms·

> requiring 1 second of compute before allowing a page load After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike

by Aachen 28d ago

> requiring 1 second of compute before allowing a page load After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing

jdlshore 28d ago
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
- gruez 28d ago
  >If scrapers reuse a cookie, traditional anti-abuse mechanisms will work. Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
  - embedding-shape 28d ago
    > Scrapers can't change their strategy? Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
    - gruez 28d ago
      That argument also works against anubis. Why bother forcing clients to solve a PoW challenge, when you're fighting a bunch of unsophisticated attackers that will be deterred by a javascript challenge that requires them to solve `result = x + y`?
      - mitxela 28d ago
        [flagged]
        myng111 27d ago
        [flagged]
      - xena 28d ago
        This is why there's the meta refresh challenge as an option: https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh https://anubis.techaro.lol/docs/admin/configuration/challeng... Turns out that in practice there's some bots that implement support for this but not JavaScript.
      - embedding-shape 28d ago
        Point is, you get problems because of something like this, you try either approach. And when that stops working (because eventually it will), you add something else. Some work better against some actors, but also feels like it never ends, so slowly you add more things. I don't think Anubis is uniquely able to stop that particular type of annoyance, it's just one of the tools you can see if it helps. If you want to try something else before, makes a lot of sense to do so.
      - well_ackshually 27d ago
        You're not fighting unsophisticated attackers. 50% of them are, maybe. The rest? They're actively hostile, detecting blocking attempts, moving on to botnets, DDoSing you and trying everything they can to bypass you. People have been sent to jail for participating in LOIC attacks which were less destructive that what these bots do on the daily.
  - bulder 28d ago
    Plenty of these abusive scrapers are utilizing retail residential proxies, which will be applying forced rotations to avoid "burning" their compromised and or otherwise surreptitiously utilized IP address.
- Aachen 26d ago
  I don't know what you mean by traditional anti-abuse (typing over two words from an image?), but switching IP addresses rapidly, while staying recognisable as one entity because of the unique cookie, is not normal user behavior so this would be trivial to filter out for something like Anubis and require another solve. Maybe that's what you meant though?
mitxela 28d ago
If you change your IP address, it invalidates the cookie. If you don't change your IP address, you can be blocked by IP address.
__s 28d ago
https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies reports anubis effectiveness falling off
- doc_ick 28d ago
  I would disagree with the Anubis effectiveness falling off, but the llm bros are throwing life itself into training to defeat any “perceived” challenge.

by Aachen 28d ago

jdlshore 28d ago

The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.

gruez 28d ago
>If scrapers reuse a cookie, traditional anti-abuse mechanisms will work. Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
- embedding-shape 28d ago
  > Scrapers can't change their strategy? Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
  - gruez 28d ago
    That argument also works against anubis. Why bother forcing clients to solve a PoW challenge, when you're fighting a bunch of unsophisticated attackers that will be deterred by a javascript challenge that requires them to solve `result = x + y`?
    - mitxela 28d ago
      [flagged]
      - myng111 27d ago
        [flagged]
    - xena 28d ago
      This is why there's the meta refresh challenge as an option: https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh https://anubis.techaro.lol/docs/admin/configuration/challeng... Turns out that in practice there's some bots that implement support for this but not JavaScript.
    - embedding-shape 28d ago
      Point is, you get problems because of something like this, you try either approach. And when that stops working (because eventually it will), you add something else. Some work better against some actors, but also feels like it never ends, so slowly you add more things. I don't think Anubis is uniquely able to stop that particular type of annoyance, it's just one of the tools you can see if it helps. If you want to try something else before, makes a lot of sense to do so.
    - well_ackshually 27d ago
      You're not fighting unsophisticated attackers. 50% of them are, maybe. The rest? They're actively hostile, detecting blocking attempts, moving on to botnets, DDoSing you and trying everything they can to bypass you. People have been sent to jail for participating in LOIC attacks which were less destructive that what these bots do on the daily.
- bulder 28d ago
  Plenty of these abusive scrapers are utilizing retail residential proxies, which will be applying forced rotations to avoid "burning" their compromised and or otherwise surreptitiously utilized IP address.
Aachen 26d ago
I don't know what you mean by traditional anti-abuse (typing over two words from an image?), but switching IP addresses rapidly, while staying recognisable as one entity because of the unique cookie, is not normal user behavior so this would be trivial to filter out for something like Anubis and require another solve. Maybe that's what you meant though?

mitxela 28d ago

If you change your IP address, it invalidates the cookie. If you don't change your IP address, you can be blocked by IP address.

__s 28d ago

https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies reports anubis effectiveness falling off

doc_ick 28d ago
I would disagree with the Anubis effectiveness falling off, but the llm bros are throwing life itself into training to defeat any “perceived” challenge.