7 ms·

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension b

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds. Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task: > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.

bjoli 1mo ago
Does it really take 3 minutes on your iPhone? My pixel 8 does it in slightly less than a minute in Firefox.
- efficax 1mo ago
  iphone 16 and i gave up after about 30 seconds, it was maybe a quarter done
smallerize 1mo ago
But the scraper is making way more requests and is paying for all that compute.
- semiquaver 1mo ago
  no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.
  - NobodyNada 1mo ago
    The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
    - semiquaver 1mo ago
      OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site. a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
      - LatticeAnimal 1mo ago
        Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
        inigyou 1mo ago
        You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.
wongarsu 1mo ago
Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
- semiquaver 1mo ago
  > I really don't see why Safari on a modern iPhone should be so slow at this me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?
  - paytonjjones 1mo ago
    In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
    - pbronez 1mo ago
      It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
      - paytonjjones 1mo ago
        It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
TZubiri 1mo ago
I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental. When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.
- lxgr 1mo ago
  I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero". Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
  - TZubiri 1mo ago
    I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed.
Self-Perfection 1mo ago
The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS? But it would just make Anubis suck a little bit less Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.
- bayindirh 1mo ago
  > I suppose it does not even have ability to bypass hashing screen when server load is low. Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site. It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
  - Self-Perfection 1mo ago
    >It's sole purpose is to deny the bots you don't want on your site. This is fundamentally impossible. In the end only pay per crawl will work. Like x402 but maybe something similar. In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
  - thaumasiotes 1mo ago
    > Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site. No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose. Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".
rzmmm 1mo ago
It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.
- inigyou 1mo ago
  And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.
  - wbl 1mo ago
    Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.
    - applfanboysbgon 1mo ago
      Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with.
      - Terr_ 1mo ago
        Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together. Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
      - wbl 1mo ago
radlad 1mo ago
I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.
- jwilk 1mo ago
  Huh? Are you saying there's an Argon2 implementation in the kernel?
  - minimaltom 1mo ago
    Its not in the kernel but in the userspace tool that goes from password to key (the key is handed to the kernel). You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/crypto_backend/argon2_generic.c https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...
  - radlad 1mo ago
    I got this wrong yesterday. While LUKS2 and GRUB now support Argon2, I configured LUKS1, which uses PBKDF2-SHA512. In any case, it is GRUB which behaves slowly during boot. While I haven't tested Argon2, I believe it is affected in the same way as PBKDF2.
inventor7777 1mo ago
My iPad Pro M2 takes around 30 seconds to solve it in Safari, while my Mac Studio solves it in about 3 seconds using 449% logical CPUs. It really makes no sense. Interesting how much slower it is than native - could it be protections against things such as crypto mining?
karlgkk 1mo ago
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. You yourself describe the solution. Extensions or browser integrations of the Anubis task.
- dpark 1mo ago
  He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.
  - karlgkk 1mo ago
    He did in fact describe it.
    - dpark 1mo ago
      I’m left wondering if we disagree about what the problem is. The problem here is not merely that Anubis is inconvenient to humans. It’s that and also that it’s not very effective for blocking bots. Anything that makes it easier for humans to get past will also make it easier for bots to get past.
    - Dylan16807 1mo ago
      The point of contention is not the word "describe", it's the word "solution".
DenisM 1mo ago
Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once. Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
- brokensegue 1mo ago
  Do you want your phone to allocate 4GB to read a mailing list?
- infp_arborist 1mo ago
  Yes, that seems to work for Altcha: https://altcha.org/docs/integration/proof-of-work-captcha/ https://altcha.org/docs/integration/proof-of-work-captcha/ Edit: this is already in discussion for Anubis: https://github.com/TecharoHQ/anubis/issues/94 https://github.com/TecharoHQ/anubis/issues/94
  - xena 1mo ago
    Good news: https://anubis.techaro.lol/docs/admin/configuration/challenges/wasm https://anubis.techaro.lol/docs/admin/configuration/challeng...
apitman 1mo ago
This comment captures the current AI zeitgeist about as well as anything I've seen.
- titularcomment 1mo ago
  I like that they supplied the prompt instead of the working code :)
  - PlotCitizen 1mo ago
    I don't
    - semiquaver 1mo ago
      Tough. The code isn’t worth enough to publish. It means nothing to me and I’d have to figure out how to do it pseudonymously. I only have the one GitHub account.
xena 1mo ago
The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challenges/wasm https://anubis.techaro.lol/docs/admin/configuration/challeng...
- tptacek 1mo ago
  What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!
  - darkmighty 1mo ago
    > rather than moving to a challenge system that actually impacts scrapers At this point isn't it basically auth-only? Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI) I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
    - GoblinSlayer 1mo ago
      Botnets will just borrow your id.
      - darkmighty 1mo ago
        Maybe, but at least you can more easily assign responsibility in this case. If you soft-block or hard-block someone because they're in a botnet, you can get them to change (expire the old ones) their credentials and start with clean systems. Maybe they'll realize their TV is part of a botnet if they have to keep doing it, or their PC has malware.
samtheDamned 1mo ago
Out of curiosity I tried to load this on my iPhone 12 and it took about 8 and a half minutes and once it finally passed it said my browser didn't allow cookies and to enable them before trying again. I would have to go through the whole proof again. I get what they're doing but there definitely needs to be some work done on getting those times down on iphones.
gizmodo59 1mo ago
Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.
- semiquaver 1mo ago
  I used sol as well. Should have noted that fable is more or less guaranteed to refuse something hacking-adjacent like that.
- latentsea 1mo ago
  It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.
  - Dylan16807 1mo ago
    "It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems." I don't see much connection between that problem and these controls.
    - latentsea 1mo ago
      What's the difference when you prompt it versus when it prompts itself?
      - Dylan16807 1mo ago
        It shouldn't prompt itself to go outside its original scope. That would be a good place for controls. But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.
- nozzlegear 1mo ago
  > Fable just rejected this. All in the name of safety, of course.
zalo 1mo ago
One natural counter to this is to dynamically alter the computing algorithm per request (variants of Anubis?) so that the scrapers would need to rewrite their accelerator each time or just use the unoptimized JS version...
stjo 1mo ago
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
- jcul 1mo ago
  Monero is fascinating. I downloaded the spec and tried to digest it, but the mathematics and cryptography are quite dense, at least for a casual read. I might try to tackle it again when I can give it some more attention.
- chr15m 1mo ago
  Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
  - eru 1mo ago
    > [...] which is not possible if somebody has a farm of devices that costs them basically zero. Opportunity costs are real.
  - margalabargala 1mo ago
    One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part. Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
    - arc-in-space 1mo ago
      Disappointed at the lack of replies here, I've had a similar thought before and I'd be curious to hear about the technical viability of this
teravor 1mo ago
anubis was a joke on arrival. PoW for this purpose is plain stupid. the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.
- crote 1mo ago
  Yes, but it actually works. As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic! It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.
  - GoblinSlayer 1mo ago
    That's because those 67% of bots are the most stupid and don't run any javascript at all.
  - teravor 1mo ago
    that means all you had to do was gate the content behind a javascript snippet with a timer.
FergusArgyll 1mo ago
Woah, went from > 1 min on git.ffmpeg (difficulty 6) to instant. Thanks! Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds
miki123211 1mo ago
And if you're a bot operating at scale, you can use different devices for crawling and challenge solving. Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving. If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.
- philippta 1mo ago
  I think TVs are just used as a residential proxies. No remote controlled chrome is running there.
imtringued 1mo ago
That's actually weaponizable. You can deliberately insist on an inefficient algorithm to find bots that use a better algorithm to bypass the PoW. Legitimate users have zero reason to redo the challenge over and over again, meanwhile bots want to constantly redo the challenge because they don't want you to throttle the bot based on the reused session cookie. You can then crank up the difficulty up over and over again.
vagab0nd 1mo ago
Sounds like we just need better proof of work infra? Obviously the bots are using the optimized hashing code while the users suffered.

bjoli 1mo ago

Does it really take 3 minutes on your iPhone? My pixel 8 does it in slightly less than a minute in Firefox.

efficax 1mo ago
iphone 16 and i gave up after about 30 seconds, it was maybe a quarter done

smallerize 1mo ago

But the scraper is making way more requests and is paying for all that compute.

semiquaver 1mo ago
no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.
- NobodyNada 1mo ago
  The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
  - semiquaver 1mo ago
    OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site. a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
    - LatticeAnimal 1mo ago
      Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
      - inigyou 1mo ago
        You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.

wongarsu 1mo ago

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this

semiquaver 1mo ago
> I really don't see why Safari on a modern iPhone should be so slow at this me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?
- paytonjjones 1mo ago
  In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
  - pbronez 1mo ago
    It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
    - paytonjjones 1mo ago
      It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.

TZubiri 1mo ago

I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental. When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.

lxgr 1mo ago
I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero". Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
- TZubiri 1mo ago
  I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed.

Self-Perfection 1mo ago

The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS? But it would just make Anubis suck a little bit less Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.

bayindirh 1mo ago
> I suppose it does not even have ability to bypass hashing screen when server load is low. Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site. It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
- Self-Perfection 1mo ago
  >It's sole purpose is to deny the bots you don't want on your site. This is fundamentally impossible. In the end only pay per crawl will work. Like x402 but maybe something similar. In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
- thaumasiotes 1mo ago
  > Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site. No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose. Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".

rzmmm 1mo ago

It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.

inigyou 1mo ago
And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.
- wbl 1mo ago
  Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.
  - applfanboysbgon 1mo ago
    Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with.
    - Terr_ 1mo ago
      Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together. Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
    - wbl 1mo ago

radlad 1mo ago

I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.

jwilk 1mo ago
Huh? Are you saying there's an Argon2 implementation in the kernel?
- minimaltom 1mo ago
  Its not in the kernel but in the userspace tool that goes from password to key (the key is handed to the kernel). You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/crypto_backend/argon2_generic.c https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...
- radlad 1mo ago
  I got this wrong yesterday. While LUKS2 and GRUB now support Argon2, I configured LUKS1, which uses PBKDF2-SHA512. In any case, it is GRUB which behaves slowly during boot. While I haven't tested Argon2, I believe it is affected in the same way as PBKDF2.

inventor7777 1mo ago

My iPad Pro M2 takes around 30 seconds to solve it in Safari, while my Mac Studio solves it in about 3 seconds using 449% logical CPUs. It really makes no sense. Interesting how much slower it is than native - could it be protections against things such as crypto mining?

karlgkk 1mo ago

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. You yourself describe the solution. Extensions or browser integrations of the Anubis task.

dpark 1mo ago
He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.
- karlgkk 1mo ago
  He did in fact describe it.
  - dpark 1mo ago
    I’m left wondering if we disagree about what the problem is. The problem here is not merely that Anubis is inconvenient to humans. It’s that and also that it’s not very effective for blocking bots. Anything that makes it easier for humans to get past will also make it easier for bots to get past.
  - Dylan16807 1mo ago
    The point of contention is not the word "describe", it's the word "solution".

DenisM 1mo ago

Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once. Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.

brokensegue 1mo ago
Do you want your phone to allocate 4GB to read a mailing list?
infp_arborist 1mo ago
Yes, that seems to work for Altcha: https://altcha.org/docs/integration/proof-of-work-captcha/ https://altcha.org/docs/integration/proof-of-work-captcha/ Edit: this is already in discussion for Anubis: https://github.com/TecharoHQ/anubis/issues/94 https://github.com/TecharoHQ/anubis/issues/94
- xena 1mo ago
  Good news: https://anubis.techaro.lol/docs/admin/configuration/challenges/wasm https://anubis.techaro.lol/docs/admin/configuration/challeng...

apitman 1mo ago

This comment captures the current AI zeitgeist about as well as anything I've seen.

titularcomment 1mo ago
I like that they supplied the prompt instead of the working code :)
- PlotCitizen 1mo ago
  I don't
  - semiquaver 1mo ago
    Tough. The code isn’t worth enough to publish. It means nothing to me and I’d have to figure out how to do it pseudonymously. I only have the one GitHub account.

xena 1mo ago

The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challenges/wasm https://anubis.techaro.lol/docs/admin/configuration/challeng...

tptacek 1mo ago
What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!
- darkmighty 1mo ago
  > rather than moving to a challenge system that actually impacts scrapers At this point isn't it basically auth-only? Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI) I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
  - GoblinSlayer 1mo ago
    Botnets will just borrow your id.
    - darkmighty 1mo ago
      Maybe, but at least you can more easily assign responsibility in this case. If you soft-block or hard-block someone because they're in a botnet, you can get them to change (expire the old ones) their credentials and start with clean systems. Maybe they'll realize their TV is part of a botnet if they have to keep doing it, or their PC has malware.

samtheDamned 1mo ago

Out of curiosity I tried to load this on my iPhone 12 and it took about 8 and a half minutes and once it finally passed it said my browser didn't allow cookies and to enable them before trying again. I would have to go through the whole proof again. I get what they're doing but there definitely needs to be some work done on getting those times down on iphones.

gizmodo59 1mo ago

Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.

semiquaver 1mo ago
I used sol as well. Should have noted that fable is more or less guaranteed to refuse something hacking-adjacent like that.
latentsea 1mo ago
It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.
- Dylan16807 1mo ago
  "It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems." I don't see much connection between that problem and these controls.
  - latentsea 1mo ago
    What's the difference when you prompt it versus when it prompts itself?
    - Dylan16807 1mo ago
      It shouldn't prompt itself to go outside its original scope. That would be a good place for controls. But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.
nozzlegear 1mo ago
> Fable just rejected this. All in the name of safety, of course.

zalo 1mo ago

One natural counter to this is to dynamically alter the computing algorithm per request (variants of Anubis?) so that the scrapers would need to rewrite their accelerator each time or just use the unoptimized JS version...

stjo 1mo ago

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)

jcul 1mo ago
Monero is fascinating. I downloaded the spec and tried to digest it, but the mathematics and cryptography are quite dense, at least for a casual read. I might try to tackle it again when I can give it some more attention.
chr15m 1mo ago
Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
- eru 1mo ago
  > [...] which is not possible if somebody has a farm of devices that costs them basically zero. Opportunity costs are real.
- margalabargala 1mo ago
  One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part. Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
  - arc-in-space 1mo ago
    Disappointed at the lack of replies here, I've had a similar thought before and I'd be curious to hear about the technical viability of this

teravor 1mo ago

anubis was a joke on arrival. PoW for this purpose is plain stupid. the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.

crote 1mo ago
Yes, but it actually works. As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic! It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.
- GoblinSlayer 1mo ago
  That's because those 67% of bots are the most stupid and don't run any javascript at all.
- teravor 1mo ago
  that means all you had to do was gate the content behind a javascript snippet with a timer.

FergusArgyll 1mo ago

Woah, went from > 1 min on git.ffmpeg (difficulty 6) to instant. Thanks! Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds

miki123211 1mo ago

And if you're a bot operating at scale, you can use different devices for crawling and challenge solving. Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving. If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.

philippta 1mo ago
I think TVs are just used as a residential proxies. No remote controlled chrome is running there.

imtringued 1mo ago

That's actually weaponizable. You can deliberately insist on an inefficient algorithm to find bots that use a better algorithm to bypass the PoW. Legitimate users have zero reason to redo the challenge over and over again, meanwhile bots want to constantly redo the challenge because they don't want you to throttle the bot based on the reused session cookie. You can then crank up the difficulty up over and over again.

vagab0nd 1mo ago

Sounds like we just need better proof of work infra? Obviously the bots are using the optimized hashing code while the users suffered.