8 ms·

You Don't Need Anubis

177 points by flexagoon 1y ago

tptacek 1y ago
This came up before (and this post links to the Tavis Ormandy post that kicked up the last firestorm about Anubis) and without myself shading the intent or the execution on Anubis, just from a CS perspective, I want to say again that the PoW thing Anubis uses doesn't make sense. Work functions make sense in password hashes because they exploit an asymmetry: attackers will guess millions of invalid passwords for every validated guess, so the attacker bears most (really almost all) of the cost. Work functions make sense in antispam systems for the same reason: spam "attacks" rely on the cost of an attempt being so low that it's efficient to target millions of victims in the expectation of just one hit. Work functions make sense in Bitcoin because they function as a synchronization mechanism. There's nothing actually valorous about solving a SHA2 puzzle, but the puzzles give the whole protocol a clock. Work functions don't make sense as a token tax; there's actually the opposite of the antispam asymmetry there. Every bot request to a web page yields tokens to the AI company. Legitimate users, who far outnumber the bots, are actually paying more of a cost. None of this is to say that a serious anti-scraping firewall can't be built! I'm fond of pointing to how Youtube addressed this very similar problem, with a content protection system built in Javascript that was deliberately expensive to reverse engineer and which could surreptitiously probe the precise browser configuration a request to create a new Youtube account was using. The next thing Anubis builds should be that, and when they do that, they should chuck the proof of work thing.
- gucci-on-fleek 1y ago
  > Work functions don't make sense as a token tax; there's actually the opposite of the antispam asymmetry there. Every bot request to a web page yields tokens to the AI company. Legitimate users, who far outnumber the bots, are actually paying more of a cost. Agreed, residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs. So I'm not really sure why Anubis works—my best guess is that the bots have some sort of time limit for each page, and they haven't bothered to increase it for pages that use Anubis. > with a content protection system built in Javascript that was deliberately expensive to reverse engineer and which could surreptitiously probe the precise browser configuration a request to create a new Youtube account was using. > The next thing Anubis builds should be that, and when they do that, they should chuck the proof of work thing. They did [0], but it doesn't work [1]. Of course, the Anubis implementation is much simpler than YouTube's, but (1) Anubis doesn't have dozens of employees who can test hundreds of browser/OS/version combinations to make sure that it doesn't inadvertently block human users, and (2) it's much trickier to design an open-source program that resists reverse-engineering than a closed-source program, and I wouldn't want to use Anubis if it went closed-source. [0]: https://anubis.techaro.lol/docs/admin/configuration/challenges/preact https://anubis.techaro.lol/docs/admin/configuration/challeng... [1]: https://github.com/TecharoHQ/anubis/issues/1121 https://github.com/TecharoHQ/anubis/issues/1121
  - tptacek 1y ago
    Google's content-protection system didn't simply make sure you could run client-side Javascript. It implemented an obfuscating virtual machine that, if I'm remembering right (I may be getting some of the detailed blurred with Blu Ray's BD+ scheme) built up a hash input of runtime artifacts. As I understand it, it was one person's work, not the work of a big team. The "source code" we're talking about here is clientside Javascript. Either way: what Anubis does now --- just from a CS perspective, that's all --- doesn't make sense.
- mariusor 1y ago
  With all due respect, but almost all I see in this thread is people looking down their nose at a proven solution, and giving advice instead of doing the work. I can see how you are a _very important person_ with bills to pay and money to make, but at least have the humility of understanding that the solution we got is better than the solution that could be better if only there was someone else to think of it and build it.
  - yumechii 1y ago
    [dead]
    - mariusor 1y ago
      It's weird that you get offended by something which was not directed at you. "The work" is providing those better alternatives to anubis, that everyone in this thread except for Xe seem to know all about. The humility is about accepting the fact that the solution works for some people, the small site operators that get hammered by DDoSes and unethical LLM over crawling, despite not being perfect. And if that inconveniences you as a user of those sites - which I imagine is what you mean by "user backlash", the solution for you is to stop going there, not talk down at them for doing something about an issue that impacts them.
      - yumechii 1y ago
        How am I offended? Did I accuse you of anything? I didn't even accuse Anubis of anything. You asked for the work, I post the work and evidence to ground the discussion in "work", as you demanded.
        mariusor 1y ago
        I repeat "the work" is to make a better thing than Anubis, not provide proof of concept that it can be beaten. :)
        1y ago
- Gander5739 1y ago
  But youtube can still be scraped with yt-dlp, so apparently it wasn't enough.
  - tptacek 1y ago
    Preventing that wasn't the objective of the content-protection system. You'll have to go read up on it.
gucci-on-fleek 1y ago
> But it still works, right? People use Anubis because it actually stops LLM bots from scraping their site, so it must work, right? > Yeah, but only because the LLM bots simply don’t run JavaScript. I don't think that this is the case, because when Anubis itself switched from a proof-of-work to a different JavaScript-based challenge, my server got overloaded, but switching back to the PoW solution fixed it [0]. I also semi-hate Anubis since it required me to add JS to a website that used none before, but (1) it's the only thing that stopped the bot problem for me, (2) it's really easy to deploy, and (3) very few human visitors are incorrectly blocked by it (unlike Captchas or IP/ASN bans that have really high false-positive rates). [0]: https://github.com/TecharoHQ/anubis/issues/1121 https://github.com/TecharoHQ/anubis/issues/1121
uqers 1y ago
> Unfortunately, the price LLM companies would have to pay to scrape every single Anubis deployment out there is approximately $0.00. The math on the site linked here as a source for this claim is incorrect. The author of that site assumes that scrapers will keep track of the access tokens for a week, but most internet-wide scrapers don't do so. The whole purpose of Anubis is to be expensive for bots that repeatedly request the same site multiple times a second.
- tptacek 1y ago
  Right, but that's the point. It's not that the idea is bad. It's that PoW is the wrong fit for it. Internet-wide scrapers don't keep state? Ok, then force clients to do something that requires keeping state. You don't need to grind SHA2 puzzles to do that; you don't need to grind anything at all.
- valicord 1y ago
  The point is that the scrapers can easily bypass this if they cared to do so
  - uqers 1y ago
    How so?
    - tecoholic 1y ago
      Hmm… by setting the verified=1 cookie on every request to the website? Am I missing something here? All this does is set an unencrypted cookie and reload the page right?
      - notpushkin 1y ago
        They could, but if this is slightly different from site to site, they’ll have to either do this for every site (annoying but possible if your site is important enough), or go ahead and run JS (which... I thought they do already, with plenty of sites still being SPAs?)
indrora 1y ago
The problem is that increasingly, they are running JS. In the ongoing arms race, we're likely to see simple things like this sort of check result in a handful of detection systems that look for "set a cookie" or at least "open the page in headless chrome and measure the cookies."
- utopiah 1y ago
  > increasingly, they are running JS. I mean they have access to a mind-blowing amount of computing resources so to they using a fraction of that to improve the quality of the data because they have this fundamental belief (because it's convenient for their situation) that scale is everything, why not use JS too. Heck if they have to run on a container full a browser, not even headless, they will.
  - typpilol 1y ago
    Chrome even released a dev tools mcp they gives any LLM full tool access to do anything in the browser. Navigate, screenshots, etc. it has like 30 tools in it alone. Now we can just run real browsers with LLMs attached. Idk how you even think about defeating that.
- moebrowne 1y ago
  > increasingly, they are running JS. Does anyone have any proof of this?
  - xena 1y ago
    I'm seeing more big botnets hosted on Alibaba Cloud, Huawei Cloud, and one on Tencent Cloud that run Headless Chrome. IP space blocks have been the solution there. I currently have a thread open with Tencent Cloud abuse where they've been begging me to not block them by default.
    - ranger_danger 1y ago
      I don't consider cloud IP blocks a solution. We use Amazon WorkSpaces, and many sites often block or restrict access just because our IPs appear to be from Amazon. There are also a good number of legitimate VPN users that are on cloud IPs.
echelon 1y ago
This whole thing is pointless. OpenAI Atlas defeats all of this by being a user's web browser. They got between you and the user you're trying to serve content, and they slurp up everything the user browses to return it back for training. The firewall is now moot. The bigger AI company, Google, has already been doing this for decades. They were the middlemen between your reader and you, and that position is unassailable. Without them, you don't have readers. At this point, the only people you're keeping out with LLM firewalls are the smaller players, which further entrenches the leaders. OpenAI and Google want you to block everybody else.
- happyopossum 1y ago
  > Google, has already been doing this for decades Do you have any proof, or even circumstantial evidence to point to this being the case? If chrome actually scraped every site ever you visited and sent it off to Google, it’d be trivially simple to find some indication of that in network traffic, or heck - even chromium code.
  - echelon 1y ago
    Sorry, I mean they're between the customer relationship. Who would dare block Google Search from indexing their site? The relationship is adversarial, but necessary.
    - deleted 1y ago
      [deleted]
    - ranger_danger 1y ago
      > Who would dare block Google Search from indexing their site? People who don't want to be indexed. Or found at all.
- Dylan16807 1y ago
  Is it confirmed that site loads go into the training database? But for anyone whose main concern is their server staying up, Atlas isn't a problem. It's not doing a million extra loads.
yellow_lead 1y ago
Anubis should be something that doesn't inconvenience all the real humans that visit your site. I work with ffmpeg so I have to access their bugtracker and mailing list site sometimes. Every few days, I'm hit with the Anubis block. And 1/3 - 1/5 of the time, it fails completely. The other times, it delays me by a few seconds. Over time, this has turned me sour on the Anubis project, which was initially something I supported.
- throwaway290 1y ago
  I understand why ffmpeg does it. No one is expected to pay for it. Until this age of LLMs when bot traffic became dominant on the web ffmpeg site was probably acceptable expense. But they probably don't want to be unpaid data provider for big LLM operators who get to extract a few bucks from their users. It's like airplane checkin. Are we inconvenienced? Yes. Who is there to blame? Probably not the airline or the company who provides the services. Probably people who want to fly without a ticket or bring in explosives. As long as Anubis project and people on it don't try to play both sides and don't make the LLM situation worse (mafia racket style), I think if it works it works.
  - TJSomething 1y ago
    I know it's beside the point, but I think a chunk of the reason for many of the security measures in airports is because creating the appearance of security increases people's willingness to fly.
    - fragmede 1y ago
      Like some sort of theater, you say?
    - majewsky 1y ago
      Yes, this is where the term "security theater" comes from: https://en.wikipedia.org/wiki/Security_theater https://en.wikipedia.org/wiki/Security_theater
- mariusor 1y ago
praptak 1y ago
There are reasons to choose the slightly annoying solution on purpose though. I'm thinking of a political statement along the lines "We have a problem with asshole AI companies and here's how they make everyone's life slightly worse."
GauntletWizard 1y ago
Anubis's design is copied from a great botnet protection mechanism - You serve the Javascript cheaply from memory, and then the client is forced to do expensive compute in order to use your expensive compute. This works great at keeping attackers from attempting to waste your time; It turns a 1:1000 amplification in compute costs into a 1000:1. It is a shitty, and obviously bad solution for preventing scraping traffic. The goal of scraping traffic isn't to overwhelm your site, it's to read it once. If you make it prohibitively expensive to read your site even once, nobody comes to it. If you make it only mildly expensive, nobody scraping cares. Anubis is specifically DDOS protection, not generally anti-bot, aside from defeating basic bots that don't emulate a full browser. It's been cargo-culted in front of a bunch of websites because of the latter, but it was obviously not going to work for long.
- viraptor 1y ago
  > The goal of scraping traffic isn't to overwhelm your site, it's to read it once. If the authors of the scrapers actually cared about it, we wouldn't have this problem in the first place. But today the more appropriate description is: the goal is to scrape as much data as possible as quickly as possible, preferably before your site falls over. They really don't care and side effects beyond that. Search engines have an incentive to leave your site running. AI companies don't. (Maybe apart from perplexity)
- purple_turtle 1y ago
  Some people deployed Anubis not to stop scraping, but to stop scraping the same page multiple times per second.
- reppap 1y ago
  First of all Anubis isn't meant to protect simple websites that gets read once. It's meant for things like a gitlabs instance where AI bots are indexing every single commit of every single file. Resulting in thousands of not millions of reads. And reading an Anubis page once isn't expensive either. So I don't really understand what point you are trying to make as the premise seems completely wrong.
- ranger_danger 1y ago
Razengan 1y ago
How else would I inter my dead and make sure they get to the afterlife?
notpushkin 1y ago
My favourite thing about Anubis is that (in default configuration) it completely bypasses the actual challenge altogether if you set User-Agent header to curl. E.g. if you open this in browser, you’ll get the challenge: https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/13ce36fef98a3f4e6d8360c24d6b8434cbb8869b https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/13ce36fef98a3f4... But if you run this, you get the page content straight away: curl https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/13ce36fef98a3f4e6d8360c24d6b8434cbb8869b I’m pretty sure this gets abused by AI scrapers a lot. If you’re running Anubis, take a moment to configure it properly, or better put together something that’s less annoying for your visitors like the OP.
- rezonant 1y ago
  It only challenges user agents with Mozilla in their name by design, because user agents that do otherwise are already identifiable. If Anubis makes the bots change their user agents, it has done its job, as that traffic can now be addressed directly.
  - hshdhdhehd 1y ago
    What if everyone requests from the bot has a different UA?
    - skylurk 1y ago
      Success. The goal is to differentiate users and bots who are pretending to be users.
    - trenchpilgrim 1y ago
      Then you can tell the bots apart from legitimate users through normal WAF rules, because browsers froze the UA a while back.
  - hsbauauvhabzb 1y ago
    Can you explain what you mean by this? Why Mozilla specifically and not WebKit or similar?
jchw 1y ago
I was briefly messing around with Pangolin, which is supposed to be a self-hosted Cloudflare Tunnels sort of thing. Pretty cool. One thing I noticed though was that the Digital Ocean Marketplace image asks you if you want to install something called Crowdsec, which is described as a "multiplayer firewall", and while it is a paid service, it appears there is a community offering that is well-liked enough. I actually was really wondering what downsides it has (except for the obvious, which is that you are definitely trading some user privacy in service of security) but at least in principle the idea seems kind of a nice middleground between Cloudflare and nothing if it works and the business model holds up.
- bootsmann 1y ago
  Not sure crowdsec is fit for this purpose. Its more a fail2ban replacement than a ddos challenge.
  - jchw 1y ago
    One of the main ways that Cloudflare is able to avoid presenting CAPTCHAs to a lot of people while still filtering tons of non-human traffic is exactly that, though: just having a boatload of data across the Internet.
herpessimplex10 1y ago
[flagged]
- viraptor 1y ago
  Or you know, they may be serious and grown up enough to not be bothered by an image of an anime cat girl. There's really nothing there to be offended by.
  - herpessimplex10 1y ago
    [flagged]
    - petesergeant 1y ago
      > an anime cat-girl > an animated cock and balls You don't see a difference between these things?
      - GaryBluto 1y ago
        While it's a bit of an extreme comparison, they're both weird, unprofessional imagery associated with things you wouldn't wan to associate with a software project.
        lakecresva 1y ago
        If it were just the imagery I don't think this would be such a huge flashpoint relative to something like tux or octocat (the github mascot).
        squigz 1y ago
        Because it's really about the type of people (they think) watch anime, and their inability to separate this preconception from reality.
        petesergeant 1y ago
defraudbah 1y ago
[flagged]
- viaoktavia 1y ago
  [dead]
- m4rtink 1y ago
  Working as intended! ;-)
geokon 1y ago
Big picture, why does everyone scrape the web? Why doesn't one company do it and then resell the data? Is it a legal/liability issue? If you scrape it's a legal grey area, but if you sell what you scrap it's clearly copyright infringement?
- utopiah 1y ago
  My bet is that they believe https://commoncrawl.org https://commoncrawl.org isn't good enough and, precisely as you are suggesting, the "rest" is where is their competitive advantage might stem from.
  - Jackson__ 1y ago
    Thinking that there is anything worth scraping past the llm-apocalypse is pure hubris imo. It is slop city out there, and unless you have an impossibly perfect classifier to detect it, 99.9% of all the great new "content" you scrape will be AI written. E: In fact this whole idea is so stupid that I am forced to consider if it is just a DDoS in the original sense. Scrape everything so hard it goes down, just so that your competitors can't.
  - ccgreg 1y ago
    Most academic AI research and AI startups find Common Crawl adequate for what they're doing. Common Crawl also has a lot of not-AI usage.
  - fragmede 1y ago
    I think that there are lots of people who are working from "first principles" and haven't even heard of common crawl or know how to use it.
GaryBluto 1y ago
[flagged]
utopiah 1y ago
"Yes, it works, and does so as effectively as Anubis, while not bothering your visitors with a 10-second page load time." Cool... but I guess now we need a benchmark for such solutions. I don't know the author, I roughly know the problem (as I self host and most of my traffic now comes from AI scrapper bots, not the usual indexing bots or, mind you, humans) but when they are numerous solutions to a multi-dimensional problem I need a common way to compare them. Yet another solution is always welcomed but without being able to efficiently compare it doesn't help me to pick the right one for me.
hubraumhugo 1y ago
What's the endgame of this increasing arms race? A gated web where you need to log in everywhere? Even more captchas and Cloudflare becoming the gateway to the internet? There must be a better way. We're somehow still stuck with CAPTCHAs (and other challenges), a 25 years old concept that wastes millions of human hours and billions in infra costs [0]. [0] https://arxiv.org/abs/2311.10911 https://arxiv.org/abs/2311.10911
- DecoySalamander 1y ago
  Maybe a web where you provide your credit card number upfront and pay for each outgoing request.
paweladamczuk 1y ago
Internet in its current form, where I can theoretically ping any web server on earth from my bedroom, doesn't seem sustainable. I think it will have to end at some point. I can't fully articulate it but I feel like there is some game theory aspect of the current design that's just not compatible with the reality.
- noAnswer 1y ago
  Years ago, wasn't there a proposal from google or the likes to have push notifications for search engines? Instead of the bots checking offer and offer again if there is something new, you would inform them about it. I think that would be a fair middle ground. You don't ddos us and in exchange we inform you timely if there is something new. (Bot would need a way to subscript themselves.) I have a personal website that sometimes doesn't get an update for a year. Still the bots are in the majority of visitors. (Not so much that I would need counter measures but still.) Most bot visits could be avoided with such a scheme.
  - redwall_hp 1y ago
    Ah, so blog pingbacks are new again. https://en.wikipedia.org/wiki/Pingback https://en.wikipedia.org/wiki/Pingback That's how Technorati worked.
  - ranger_danger 1y ago
    The problem I see with this approach is that it enables website operators to stop alerting bots completely, and then the bots' customers will complain that sites aren't updated, and don't care that the site owner is blocking them.
weinzierl 1y ago
"Unfortunately, Cloudflare is pretty much the only reliable way to protect against bots." With footnote: "I don’t know if they have any good competition, but “Cloudflare” here refers to all similar bot protection services." That's the crux. Cloudflare is the default, no one seems to bother to take the risk with a competitor for some reason. They seem to exist but when asked people can't even name them. (For what it's worth I've been using AWS Cloudfront but I had to think a moment to remember its name.)
- Avamander 1y ago
  It's actually not that reliable either given a bit of effort. Only their paid offerings actually give you tools to properly defend against intentional attacks.
- fragmede 1y ago
  AWS Shield (and GCP Cloud Armor) is the ddos protection product you're thinking of. Cloudfront's a CDN.
Borg3 1y ago
It seems that people do NOT understand its already game over.. Lost.. When stuff was small, and we had abusive actors, nobody cared.. oh just few bad actors, nothing to worry about, they will get bored and go away. No, they wont, they will grow and grow and now most even good guys turned bad because there is no punishment for it.. So as I said, game over. Its time to start do own walled gardens, build overlay VPN networks for humans. Put services there, if someone misbehave? BAN his IP. Came back? BAN again. Came back? wtf? BAN VPN provider.. Just clean the mess.. different networks can peer and exchange. Look, Internet is just network of networks, its not that hard.
- timeon 1y ago
  Good idea. Another solutions is to move our things to p2p. These corporations need expensive servers to run huge models on or just collect data. Sometimes winning move is not play the game: true server-less.
gbuk2013 1y ago
The Caddy config in the parent article uses status code 418. This is cute, but wouldn’t this break search engine indexing? Why not use 307 code?
- flexagoon 1y ago
  I use this for a personal Redlib instance, so search indexing is not important. I don't know if this will allow indexing even with a 307 status code - maybe you just need to add an exception for Googlebot.
agnishom 1y ago
Exactly. I don't understand what computation you can afford to do in 10 seconds on a small number of cores that bots running on large data centers cannot
- juliangmp 1y ago
  The point of anubis isn't to make the scraping impossible, but make it more expensive.
  - agnishom 1y ago
    by how much? I don't understand the cost model here at all.
    - eqvinox 1y ago
      AIUI the idea is to ratelimit each "solution". A normal human's browser only needs to "solve" once. A LLM crawler either needs to slow down (= objective achieved) or solve the puzzle n times to get n × the request rate.
      - agnishom 1y ago
        lets say that that adding Anubis does the job of adding 10 seconds of extra compute for the bot when it tries to access my website. Will this be enough to deter the bot/scraper?
        GabrielTFS 11mo ago
        Empirical evidence appears to show that it is ¯\_(ツ)_/¯
andersmurphy 1y ago
So I don't use cloudflare. But only serve clients that support brotli and have a valid cookie. All the actual content comes down an SSE connection. Haven't had any problems with bots on my 5$ VPS. What I realised recently is for non user browsers my demos are effectively zip bombs. Why? Because I stream each frame and each frame is around 180kb uncompressed (compressed frames can be as small as 13bytes). This is fine as the users browser doesn't hold onto the frames. But, a crawler will hold onto those frames. Very quickly this ends up being a very bad time for them. Of course there's nothing of value to scrape so mostly pointless. But, I found it entertaining that some scummy crawler is getting nuked by checkboxes [1]. - https://checkboxes.andersmurphy.com https://checkboxes.andersmurphy.com
greatgib 1y ago
Just a personal fact, when I want to see a page and instead I have to face a 3s stupid nagscreens like the one of anubis, I'm very pissed off and pushed even more to bypass the website when possible to get the info I want directly from llm or search engine. It's kind of a self fulfilling prophecy, you make it the visitor experience worse, giving a self justification why llm giving the content is wanted and needed. All of that because in the current lambda/cloud computing word, it became very expensive to process only a few requests.
- robinsonb5 1y ago
  Unfortunately the choice isn't between sites with something like Anubis and sites with free and unencumbered access. The choice is between putting up with Anubis and the sites simply going away. A web forum I read regularly has been playing whack-a-mole with LLM scrapers for much of this year, with multiple weeks-long periods where the swarm-of-locusts would make the site inaccessible to actual users. The admins tried all manner of blocks, including ultimately banning entire countries' IP ranges, all to no avail. The forum's continued existence depends on being able to hold off abusive crawlers. Having to see half-a-second of the Anubis splashscreen occasionally is a small price to pay for keeping it alive.
  - greatgib 1y ago
    [flagged]
    - xena 1y ago
      I challenge you to take a critical look at the performance of things like PHPBB and see how even naive scraping brings commonly deployed server CPUs to their knees.
    - pushcx 1y ago
      The scrapers will not attempt to discover and use an efficient representation. They will attempt to hit every URL they can discover on a site, and they'll do it at a rate of hundreds of hits per second, from enough IPs that each only requests at a rate of 1/minute. It's rude to talk down to people for not implementing a technique that you can't get scrapers to adopt, and for matching their investment in performance to their needs instead of accurately predicting years beforehand that traffic would dramatically change.
deleted 1y ago
[deleted]
yumechii 1y ago
Here are some benchmarks, TLDR is Anubis is not as performant as an optimized client prover running on the same HEDT CPU. So the "PoW tax" essentially only applies to low volume requester who have no incentive to optimize or bespoke solution too diverse to optimize at scale. https://yumechi.jp/en/blog/2025/proof-of-mutex-outspeeding-anubis-with-valid-pow/ https://yumechi.jp/en/blog/2025/proof-of-mutex-outspeeding-a... https://github.com/eternal-flame-AD/pow-buster https://github.com/eternal-flame-AD/pow-buster The problem was "fixed" but then reverted because the fix has deadlock bug. (Changelog entry: "Remove bbolt actorify implementation due to causing production issues.")
katdork 1y ago
I don't like this solution because it is hostile to those who use solutions such as UMatrix / NoScript in their browser, who use TUI browsers (e.g. chawan, lynx, w3m, ...) or who have disabled Javascript outright. Admittedly, this is no different than the kinds of ways Anubis is hostile to those same users, truly a tragedy of the commons.
- 1vuio0pswjnm7 1y ago
  Whether intentional or not, there is an obvious benefit to the website operator in forcing users to expose themselves to images and Javascript by requiring the use of particular software. e.g., a popular graphical browser from a company providing advertising services (Google, Apple, etc.) or partnering with one (Mozilla): It makes (a) visual advertising and (b) tracking viable I read the www text-only, no auto-loading of resources (images, etc.), and I see no ads
iamnothere 1y ago
All the critics here miss the point. Anubis has worked to stop DDoS-level scraping against a number of production sites, especially self-hosted source repos and forums. If it stops working, then either Anubis contributors will come up with a fix, site devs will find their own fix, or the sites under attack will be shut down. It’s an arms race in which there is no permanent solution, each escalation will of course be easily bypassed (in theory) until the majority of the attackers find that further adaptations are not worth the additional revenue or there is no further defense possible. Anubis isn’t some conspiracy to show you pictures of anime catgirls, it’s a desperate attempt to stave off bot-driven downtime. Many admins who install it do so reluctantly, because obviously it is annoying to have a delay when you access a website. Nobody is doing that for fun. (There are probably a few people who install it not to protect against scraper DDoS, but due to ideological opposition to AI scrapers. IMHO this is fruitless, as the more intelligent scrapers will find ways around it without calling attention to themselves. Anubis makes almost no sense on a static personal blog.)
1vuio0pswjnm7 1y ago
Looks Anubis allows Internet Archive's bot sometimes Allowed https://web.archive.org/web/20250419222331if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250419222331if_/https://anubis... https://web.archive.org/web/20250419222331if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250419222331if_/https://anubis... https://web.archive.org/web/20250420152651if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250420152651if_/https://anubis... https://web.archive.org/web/20250420152651if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250420152651if_/https://anubis... Blocked https://web.archive.org/web/20250424235436if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250424235436if_/https://anubis... https://web.archive.org/web/20250510230703if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250510230703if_/https://anubis... https://web.archive.org/web/20250511110518if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250511110518if_/https://anubis... https://web.archive.org/web/20250630101240if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250630101240if_/https://anubis... https://web.archive.org/web/20250808051637if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250808051637if_/https://anubis... https://web.archive.org/web/20250909160601if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250909160601if_/https://anubis... Allowed https://web.archive.org/web/20250921062513if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250921062513if_/https://anubis...
aorth 1y ago
There was an interesting comment in the Lobsters thread about this article https://lobste.rs/s/gig2wt/you_don_t_need_anubis https://lobste.rs/s/gig2wt/you_don_t_need_anubis. Basically, Sec-Fetch-* headers are widely available on browsers https://caniuse.com/?search=sec-fetch-dest https://caniuse.com/?search=sec-fetch-dest, so you can detect if a client that says they are Chrome, Firefox, or Safari are really Chrome, Firefox, or Safari. This seems to work in Caddy, using a CEL expression: @unrealistic-browsers <<CEL {header.User-Agent}.matches("(Chrome|Firefox|Safari)") && ! ({header.Sec-Fetch-Dest}.matches("^.+$") && {header.Sec-Fetch-Mode}.matches("^.+$") && {header.Sec-Fetch-Site}.matches("^.+$")) CEL handle @unrealistic-browsers { abort } Maybe there is a better way. And maybe this stops working when all low-effort bots add these headers to their crawlers. BTW if anyone has an invite on Lobsters I would appreciate it. :)

177 points by flexagoon 1y ago

This came up before (and this post links to the Tavis Ormandy post that kicked up the last firestorm about Anubis) and without myself shading the intent or the execution on Anubis, just from a CS perspective, I want to say again that the PoW thing Anubis uses doesn't make sense. Work functions make sense in password hashes because they exploit an asymmetry: attackers will guess millions of invalid passwords for every validated guess, so the attacker bears most (really almost all) of the cost. Work functions make sense in antispam systems for the same reason: spam "attacks" rely on the cost of an attempt being so low that it's efficient to target millions of victims in the expectation of just one hit. Work functions make sense in Bitcoin because they function as a synchronization mechanism. There's nothing actually valorous about solving a SHA2 puzzle, but the puzzles give the whole protocol a clock. Work functions don't make sense as a token tax; there's actually the opposite of the antispam asymmetry there. Every bot request to a web page yields tokens to the AI company. Legitimate users, who far outnumber the bots, are actually paying more of a cost. None of this is to say that a serious anti-scraping firewall can't be built! I'm fond of pointing to how Youtube addressed this very similar problem, with a content protection system built in Javascript that was deliberately expensive to reverse engineer and which could surreptitiously probe the precise browser configuration a request to create a new Youtube account was using. The next thing Anubis builds should be that, and when they do that, they should chuck the proof of work thing.

gucci-on-fleek 1y ago
> Work functions don't make sense as a token tax; there's actually the opposite of the antispam asymmetry there. Every bot request to a web page yields tokens to the AI company. Legitimate users, who far outnumber the bots, are actually paying more of a cost. Agreed, residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs. So I'm not really sure why Anubis works—my best guess is that the bots have some sort of time limit for each page, and they haven't bothered to increase it for pages that use Anubis. > with a content protection system built in Javascript that was deliberately expensive to reverse engineer and which could surreptitiously probe the precise browser configuration a request to create a new Youtube account was using. > The next thing Anubis builds should be that, and when they do that, they should chuck the proof of work thing. They did [0], but it doesn't work [1]. Of course, the Anubis implementation is much simpler than YouTube's, but (1) Anubis doesn't have dozens of employees who can test hundreds of browser/OS/version combinations to make sure that it doesn't inadvertently block human users, and (2) it's much trickier to design an open-source program that resists reverse-engineering than a closed-source program, and I wouldn't want to use Anubis if it went closed-source. [0]: https://anubis.techaro.lol/docs/admin/configuration/challenges/preact https://anubis.techaro.lol/docs/admin/configuration/challeng... [1]: https://github.com/TecharoHQ/anubis/issues/1121 https://github.com/TecharoHQ/anubis/issues/1121
- tptacek 1y ago
  Google's content-protection system didn't simply make sure you could run client-side Javascript. It implemented an obfuscating virtual machine that, if I'm remembering right (I may be getting some of the detailed blurred with Blu Ray's BD+ scheme) built up a hash input of runtime artifacts. As I understand it, it was one person's work, not the work of a big team. The "source code" we're talking about here is clientside Javascript. Either way: what Anubis does now --- just from a CS perspective, that's all --- doesn't make sense.
mariusor 1y ago
With all due respect, but almost all I see in this thread is people looking down their nose at a proven solution, and giving advice instead of doing the work. I can see how you are a _very important person_ with bills to pay and money to make, but at least have the humility of understanding that the solution we got is better than the solution that could be better if only there was someone else to think of it and build it.
- yumechii 1y ago
  [dead]
  - mariusor 1y ago
    It's weird that you get offended by something which was not directed at you. "The work" is providing those better alternatives to anubis, that everyone in this thread except for Xe seem to know all about. The humility is about accepting the fact that the solution works for some people, the small site operators that get hammered by DDoSes and unethical LLM over crawling, despite not being perfect. And if that inconveniences you as a user of those sites - which I imagine is what you mean by "user backlash", the solution for you is to stop going there, not talk down at them for doing something about an issue that impacts them.
    - yumechii 1y ago
      How am I offended? Did I accuse you of anything? I didn't even accuse Anubis of anything. You asked for the work, I post the work and evidence to ground the discussion in "work", as you demanded.
      - mariusor 1y ago
        I repeat "the work" is to make a better thing than Anubis, not provide proof of concept that it can be beaten. :)
        1y ago
Gander5739 1y ago
But youtube can still be scraped with yt-dlp, so apparently it wasn't enough.
- tptacek 1y ago
  Preventing that wasn't the objective of the content-protection system. You'll have to go read up on it.

gucci-on-fleek 1y ago

> But it still works, right? People use Anubis because it actually stops LLM bots from scraping their site, so it must work, right? > Yeah, but only because the LLM bots simply don’t run JavaScript. I don't think that this is the case, because when Anubis itself switched from a proof-of-work to a different JavaScript-based challenge, my server got overloaded, but switching back to the PoW solution fixed it [0]. I also semi-hate Anubis since it required me to add JS to a website that used none before, but (1) it's the only thing that stopped the bot problem for me, (2) it's really easy to deploy, and (3) very few human visitors are incorrectly blocked by it (unlike Captchas or IP/ASN bans that have really high false-positive rates). [0]: https://github.com/TecharoHQ/anubis/issues/1121 https://github.com/TecharoHQ/anubis/issues/1121

uqers 1y ago

> Unfortunately, the price LLM companies would have to pay to scrape every single Anubis deployment out there is approximately $0.00. The math on the site linked here as a source for this claim is incorrect. The author of that site assumes that scrapers will keep track of the access tokens for a week, but most internet-wide scrapers don't do so. The whole purpose of Anubis is to be expensive for bots that repeatedly request the same site multiple times a second.

tptacek 1y ago
Right, but that's the point. It's not that the idea is bad. It's that PoW is the wrong fit for it. Internet-wide scrapers don't keep state? Ok, then force clients to do something that requires keeping state. You don't need to grind SHA2 puzzles to do that; you don't need to grind anything at all.
valicord 1y ago
The point is that the scrapers can easily bypass this if they cared to do so
- uqers 1y ago
  How so?
  - tecoholic 1y ago
    Hmm… by setting the verified=1 cookie on every request to the website? Am I missing something here? All this does is set an unencrypted cookie and reload the page right?
    - notpushkin 1y ago
      They could, but if this is slightly different from site to site, they’ll have to either do this for every site (annoying but possible if your site is important enough), or go ahead and run JS (which... I thought they do already, with plenty of sites still being SPAs?)

indrora 1y ago

The problem is that increasingly, they are running JS. In the ongoing arms race, we're likely to see simple things like this sort of check result in a handful of detection systems that look for "set a cookie" or at least "open the page in headless chrome and measure the cookies."

utopiah 1y ago
> increasingly, they are running JS. I mean they have access to a mind-blowing amount of computing resources so to they using a fraction of that to improve the quality of the data because they have this fundamental belief (because it's convenient for their situation) that scale is everything, why not use JS too. Heck if they have to run on a container full a browser, not even headless, they will.
- typpilol 1y ago
  Chrome even released a dev tools mcp they gives any LLM full tool access to do anything in the browser. Navigate, screenshots, etc. it has like 30 tools in it alone. Now we can just run real browsers with LLMs attached. Idk how you even think about defeating that.
moebrowne 1y ago
> increasingly, they are running JS. Does anyone have any proof of this?
- xena 1y ago
  I'm seeing more big botnets hosted on Alibaba Cloud, Huawei Cloud, and one on Tencent Cloud that run Headless Chrome. IP space blocks have been the solution there. I currently have a thread open with Tencent Cloud abuse where they've been begging me to not block them by default.
  - ranger_danger 1y ago
    I don't consider cloud IP blocks a solution. We use Amazon WorkSpaces, and many sites often block or restrict access just because our IPs appear to be from Amazon. There are also a good number of legitimate VPN users that are on cloud IPs.

echelon 1y ago

This whole thing is pointless. OpenAI Atlas defeats all of this by being a user's web browser. They got between you and the user you're trying to serve content, and they slurp up everything the user browses to return it back for training. The firewall is now moot. The bigger AI company, Google, has already been doing this for decades. They were the middlemen between your reader and you, and that position is unassailable. Without them, you don't have readers. At this point, the only people you're keeping out with LLM firewalls are the smaller players, which further entrenches the leaders. OpenAI and Google want you to block everybody else.

happyopossum 1y ago
> Google, has already been doing this for decades Do you have any proof, or even circumstantial evidence to point to this being the case? If chrome actually scraped every site ever you visited and sent it off to Google, it’d be trivially simple to find some indication of that in network traffic, or heck - even chromium code.
- echelon 1y ago
  Sorry, I mean they're between the customer relationship. Who would dare block Google Search from indexing their site? The relationship is adversarial, but necessary.
  - deleted 1y ago
    [deleted]
  - ranger_danger 1y ago
    > Who would dare block Google Search from indexing their site? People who don't want to be indexed. Or found at all.
Dylan16807 1y ago
Is it confirmed that site loads go into the training database? But for anyone whose main concern is their server staying up, Atlas isn't a problem. It's not doing a million extra loads.

yellow_lead 1y ago

Anubis should be something that doesn't inconvenience all the real humans that visit your site. I work with ffmpeg so I have to access their bugtracker and mailing list site sometimes. Every few days, I'm hit with the Anubis block. And 1/3 - 1/5 of the time, it fails completely. The other times, it delays me by a few seconds. Over time, this has turned me sour on the Anubis project, which was initially something I supported.

throwaway290 1y ago
I understand why ffmpeg does it. No one is expected to pay for it. Until this age of LLMs when bot traffic became dominant on the web ffmpeg site was probably acceptable expense. But they probably don't want to be unpaid data provider for big LLM operators who get to extract a few bucks from their users. It's like airplane checkin. Are we inconvenienced? Yes. Who is there to blame? Probably not the airline or the company who provides the services. Probably people who want to fly without a ticket or bring in explosives. As long as Anubis project and people on it don't try to play both sides and don't make the LLM situation worse (mafia racket style), I think if it works it works.
- TJSomething 1y ago
  I know it's beside the point, but I think a chunk of the reason for many of the security measures in airports is because creating the appearance of security increases people's willingness to fly.
  - fragmede 1y ago
    Like some sort of theater, you say?
  - majewsky 1y ago
    Yes, this is where the term "security theater" comes from: https://en.wikipedia.org/wiki/Security_theater https://en.wikipedia.org/wiki/Security_theater
mariusor 1y ago

praptak 1y ago

There are reasons to choose the slightly annoying solution on purpose though. I'm thinking of a political statement along the lines "We have a problem with asshole AI companies and here's how they make everyone's life slightly worse."

GauntletWizard 1y ago

Anubis's design is copied from a great botnet protection mechanism - You serve the Javascript cheaply from memory, and then the client is forced to do expensive compute in order to use your expensive compute. This works great at keeping attackers from attempting to waste your time; It turns a 1:1000 amplification in compute costs into a 1000:1. It is a shitty, and obviously bad solution for preventing scraping traffic. The goal of scraping traffic isn't to overwhelm your site, it's to read it once. If you make it prohibitively expensive to read your site even once, nobody comes to it. If you make it only mildly expensive, nobody scraping cares. Anubis is specifically DDOS protection, not generally anti-bot, aside from defeating basic bots that don't emulate a full browser. It's been cargo-culted in front of a bunch of websites because of the latter, but it was obviously not going to work for long.

viraptor 1y ago
> The goal of scraping traffic isn't to overwhelm your site, it's to read it once. If the authors of the scrapers actually cared about it, we wouldn't have this problem in the first place. But today the more appropriate description is: the goal is to scrape as much data as possible as quickly as possible, preferably before your site falls over. They really don't care and side effects beyond that. Search engines have an incentive to leave your site running. AI companies don't. (Maybe apart from perplexity)
purple_turtle 1y ago
Some people deployed Anubis not to stop scraping, but to stop scraping the same page multiple times per second.
reppap 1y ago
First of all Anubis isn't meant to protect simple websites that gets read once. It's meant for things like a gitlabs instance where AI bots are indexing every single commit of every single file. Resulting in thousands of not millions of reads. And reading an Anubis page once isn't expensive either. So I don't really understand what point you are trying to make as the premise seems completely wrong.
ranger_danger 1y ago

Razengan 1y ago

How else would I inter my dead and make sure they get to the afterlife?

notpushkin 1y ago

My favourite thing about Anubis is that (in default configuration) it completely bypasses the actual challenge altogether if you set User-Agent header to curl. E.g. if you open this in browser, you’ll get the challenge: https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/13ce36fef98a3f4e6d8360c24d6b8434cbb8869b https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/13ce36fef98a3f4... But if you run this, you get the page content straight away: curl https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/13ce36fef98a3f4e6d8360c24d6b8434cbb8869b I’m pretty sure this gets abused by AI scrapers a lot. If you’re running Anubis, take a moment to configure it properly, or better put together something that’s less annoying for your visitors like the OP.

rezonant 1y ago
It only challenges user agents with Mozilla in their name by design, because user agents that do otherwise are already identifiable. If Anubis makes the bots change their user agents, it has done its job, as that traffic can now be addressed directly.
- hshdhdhehd 1y ago
  What if everyone requests from the bot has a different UA?
  - skylurk 1y ago
    Success. The goal is to differentiate users and bots who are pretending to be users.
  - trenchpilgrim 1y ago
    Then you can tell the bots apart from legitimate users through normal WAF rules, because browsers froze the UA a while back.
- hsbauauvhabzb 1y ago
  Can you explain what you mean by this? Why Mozilla specifically and not WebKit or similar?

jchw 1y ago

I was briefly messing around with Pangolin, which is supposed to be a self-hosted Cloudflare Tunnels sort of thing. Pretty cool. One thing I noticed though was that the Digital Ocean Marketplace image asks you if you want to install something called Crowdsec, which is described as a "multiplayer firewall", and while it is a paid service, it appears there is a community offering that is well-liked enough. I actually was really wondering what downsides it has (except for the obvious, which is that you are definitely trading some user privacy in service of security) but at least in principle the idea seems kind of a nice middleground between Cloudflare and nothing if it works and the business model holds up.

bootsmann 1y ago
Not sure crowdsec is fit for this purpose. Its more a fail2ban replacement than a ddos challenge.
- jchw 1y ago
  One of the main ways that Cloudflare is able to avoid presenting CAPTCHAs to a lot of people while still filtering tons of non-human traffic is exactly that, though: just having a boatload of data across the Internet.

herpessimplex10 1y ago

[flagged]

viraptor 1y ago
Or you know, they may be serious and grown up enough to not be bothered by an image of an anime cat girl. There's really nothing there to be offended by.
- herpessimplex10 1y ago
  [flagged]
  - petesergeant 1y ago
    > an anime cat-girl > an animated cock and balls You don't see a difference between these things?
    - GaryBluto 1y ago
      While it's a bit of an extreme comparison, they're both weird, unprofessional imagery associated with things you wouldn't wan to associate with a software project.
      - lakecresva 1y ago
        If it were just the imagery I don't think this would be such a huge flashpoint relative to something like tux or octocat (the github mascot).
        squigz 1y ago
        Because it's really about the type of people (they think) watch anime, and their inability to separate this preconception from reality.
      - petesergeant 1y ago

defraudbah 1y ago

[flagged]

viaoktavia 1y ago
[dead]
m4rtink 1y ago
Working as intended! ;-)

geokon 1y ago

Big picture, why does everyone scrape the web? Why doesn't one company do it and then resell the data? Is it a legal/liability issue? If you scrape it's a legal grey area, but if you sell what you scrap it's clearly copyright infringement?

utopiah 1y ago
My bet is that they believe https://commoncrawl.org https://commoncrawl.org isn't good enough and, precisely as you are suggesting, the "rest" is where is their competitive advantage might stem from.
- Jackson__ 1y ago
  Thinking that there is anything worth scraping past the llm-apocalypse is pure hubris imo. It is slop city out there, and unless you have an impossibly perfect classifier to detect it, 99.9% of all the great new "content" you scrape will be AI written. E: In fact this whole idea is so stupid that I am forced to consider if it is just a DDoS in the original sense. Scrape everything so hard it goes down, just so that your competitors can't.
- ccgreg 1y ago
  Most academic AI research and AI startups find Common Crawl adequate for what they're doing. Common Crawl also has a lot of not-AI usage.
- fragmede 1y ago
  I think that there are lots of people who are working from "first principles" and haven't even heard of common crawl or know how to use it.

GaryBluto 1y ago

[flagged]

utopiah 1y ago

"Yes, it works, and does so as effectively as Anubis, while not bothering your visitors with a 10-second page load time." Cool... but I guess now we need a benchmark for such solutions. I don't know the author, I roughly know the problem (as I self host and most of my traffic now comes from AI scrapper bots, not the usual indexing bots or, mind you, humans) but when they are numerous solutions to a multi-dimensional problem I need a common way to compare them. Yet another solution is always welcomed but without being able to efficiently compare it doesn't help me to pick the right one for me.

hubraumhugo 1y ago

What's the endgame of this increasing arms race? A gated web where you need to log in everywhere? Even more captchas and Cloudflare becoming the gateway to the internet? There must be a better way. We're somehow still stuck with CAPTCHAs (and other challenges), a 25 years old concept that wastes millions of human hours and billions in infra costs [0]. [0] https://arxiv.org/abs/2311.10911 https://arxiv.org/abs/2311.10911

DecoySalamander 1y ago
Maybe a web where you provide your credit card number upfront and pay for each outgoing request.

paweladamczuk 1y ago

Internet in its current form, where I can theoretically ping any web server on earth from my bedroom, doesn't seem sustainable. I think it will have to end at some point. I can't fully articulate it but I feel like there is some game theory aspect of the current design that's just not compatible with the reality.

noAnswer 1y ago
Years ago, wasn't there a proposal from google or the likes to have push notifications for search engines? Instead of the bots checking offer and offer again if there is something new, you would inform them about it. I think that would be a fair middle ground. You don't ddos us and in exchange we inform you timely if there is something new. (Bot would need a way to subscript themselves.) I have a personal website that sometimes doesn't get an update for a year. Still the bots are in the majority of visitors. (Not so much that I would need counter measures but still.) Most bot visits could be avoided with such a scheme.
- redwall_hp 1y ago
  Ah, so blog pingbacks are new again. https://en.wikipedia.org/wiki/Pingback https://en.wikipedia.org/wiki/Pingback That's how Technorati worked.
- ranger_danger 1y ago
  The problem I see with this approach is that it enables website operators to stop alerting bots completely, and then the bots' customers will complain that sites aren't updated, and don't care that the site owner is blocking them.

weinzierl 1y ago

"Unfortunately, Cloudflare is pretty much the only reliable way to protect against bots." With footnote: "I don’t know if they have any good competition, but “Cloudflare” here refers to all similar bot protection services." That's the crux. Cloudflare is the default, no one seems to bother to take the risk with a competitor for some reason. They seem to exist but when asked people can't even name them. (For what it's worth I've been using AWS Cloudfront but I had to think a moment to remember its name.)

Avamander 1y ago
It's actually not that reliable either given a bit of effort. Only their paid offerings actually give you tools to properly defend against intentional attacks.
fragmede 1y ago
AWS Shield (and GCP Cloud Armor) is the ddos protection product you're thinking of. Cloudfront's a CDN.

Borg3 1y ago

It seems that people do NOT understand its already game over.. Lost.. When stuff was small, and we had abusive actors, nobody cared.. oh just few bad actors, nothing to worry about, they will get bored and go away. No, they wont, they will grow and grow and now most even good guys turned bad because there is no punishment for it.. So as I said, game over. Its time to start do own walled gardens, build overlay VPN networks for humans. Put services there, if someone misbehave? BAN his IP. Came back? BAN again. Came back? wtf? BAN VPN provider.. Just clean the mess.. different networks can peer and exchange. Look, Internet is just network of networks, its not that hard.

timeon 1y ago
Good idea. Another solutions is to move our things to p2p. These corporations need expensive servers to run huge models on or just collect data. Sometimes winning move is not play the game: true server-less.

gbuk2013 1y ago

The Caddy config in the parent article uses status code 418. This is cute, but wouldn’t this break search engine indexing? Why not use 307 code?

flexagoon 1y ago
I use this for a personal Redlib instance, so search indexing is not important. I don't know if this will allow indexing even with a 307 status code - maybe you just need to add an exception for Googlebot.

agnishom 1y ago

Exactly. I don't understand what computation you can afford to do in 10 seconds on a small number of cores that bots running on large data centers cannot

juliangmp 1y ago
The point of anubis isn't to make the scraping impossible, but make it more expensive.
- agnishom 1y ago
  by how much? I don't understand the cost model here at all.
  - eqvinox 1y ago
    AIUI the idea is to ratelimit each "solution". A normal human's browser only needs to "solve" once. A LLM crawler either needs to slow down (= objective achieved) or solve the puzzle n times to get n × the request rate.
    - agnishom 1y ago
      lets say that that adding Anubis does the job of adding 10 seconds of extra compute for the bot when it tries to access my website. Will this be enough to deter the bot/scraper?
      - GabrielTFS 11mo ago
        Empirical evidence appears to show that it is ¯\_(ツ)_/¯

andersmurphy 1y ago

So I don't use cloudflare. But only serve clients that support brotli and have a valid cookie. All the actual content comes down an SSE connection. Haven't had any problems with bots on my 5$ VPS. What I realised recently is for non user browsers my demos are effectively zip bombs. Why? Because I stream each frame and each frame is around 180kb uncompressed (compressed frames can be as small as 13bytes). This is fine as the users browser doesn't hold onto the frames. But, a crawler will hold onto those frames. Very quickly this ends up being a very bad time for them. Of course there's nothing of value to scrape so mostly pointless. But, I found it entertaining that some scummy crawler is getting nuked by checkboxes [1]. - https://checkboxes.andersmurphy.com https://checkboxes.andersmurphy.com

greatgib 1y ago

Just a personal fact, when I want to see a page and instead I have to face a 3s stupid nagscreens like the one of anubis, I'm very pissed off and pushed even more to bypass the website when possible to get the info I want directly from llm or search engine. It's kind of a self fulfilling prophecy, you make it the visitor experience worse, giving a self justification why llm giving the content is wanted and needed. All of that because in the current lambda/cloud computing word, it became very expensive to process only a few requests.

robinsonb5 1y ago
Unfortunately the choice isn't between sites with something like Anubis and sites with free and unencumbered access. The choice is between putting up with Anubis and the sites simply going away. A web forum I read regularly has been playing whack-a-mole with LLM scrapers for much of this year, with multiple weeks-long periods where the swarm-of-locusts would make the site inaccessible to actual users. The admins tried all manner of blocks, including ultimately banning entire countries' IP ranges, all to no avail. The forum's continued existence depends on being able to hold off abusive crawlers. Having to see half-a-second of the Anubis splashscreen occasionally is a small price to pay for keeping it alive.
- greatgib 1y ago
  [flagged]
  - xena 1y ago
    I challenge you to take a critical look at the performance of things like PHPBB and see how even naive scraping brings commonly deployed server CPUs to their knees.
  - pushcx 1y ago
    The scrapers will not attempt to discover and use an efficient representation. They will attempt to hit every URL they can discover on a site, and they'll do it at a rate of hundreds of hits per second, from enough IPs that each only requests at a rate of 1/minute. It's rude to talk down to people for not implementing a technique that you can't get scrapers to adopt, and for matching their investment in performance to their needs instead of accurately predicting years beforehand that traffic would dramatically change.

deleted 1y ago

[deleted]

yumechii 1y ago

Here are some benchmarks, TLDR is Anubis is not as performant as an optimized client prover running on the same HEDT CPU. So the "PoW tax" essentially only applies to low volume requester who have no incentive to optimize or bespoke solution too diverse to optimize at scale. https://yumechi.jp/en/blog/2025/proof-of-mutex-outspeeding-anubis-with-valid-pow/ https://yumechi.jp/en/blog/2025/proof-of-mutex-outspeeding-a... https://github.com/eternal-flame-AD/pow-buster https://github.com/eternal-flame-AD/pow-buster The problem was "fixed" but then reverted because the fix has deadlock bug. (Changelog entry: "Remove bbolt actorify implementation due to causing production issues.")

katdork 1y ago

I don't like this solution because it is hostile to those who use solutions such as UMatrix / NoScript in their browser, who use TUI browsers (e.g. chawan, lynx, w3m, ...) or who have disabled Javascript outright. Admittedly, this is no different than the kinds of ways Anubis is hostile to those same users, truly a tragedy of the commons.

1vuio0pswjnm7 1y ago
Whether intentional or not, there is an obvious benefit to the website operator in forcing users to expose themselves to images and Javascript by requiring the use of particular software. e.g., a popular graphical browser from a company providing advertising services (Google, Apple, etc.) or partnering with one (Mozilla): It makes (a) visual advertising and (b) tracking viable I read the www text-only, no auto-loading of resources (images, etc.), and I see no ads

iamnothere 1y ago

All the critics here miss the point. Anubis has worked to stop DDoS-level scraping against a number of production sites, especially self-hosted source repos and forums. If it stops working, then either Anubis contributors will come up with a fix, site devs will find their own fix, or the sites under attack will be shut down. It’s an arms race in which there is no permanent solution, each escalation will of course be easily bypassed (in theory) until the majority of the attackers find that further adaptations are not worth the additional revenue or there is no further defense possible. Anubis isn’t some conspiracy to show you pictures of anime catgirls, it’s a desperate attempt to stave off bot-driven downtime. Many admins who install it do so reluctantly, because obviously it is annoying to have a delay when you access a website. Nobody is doing that for fun. (There are probably a few people who install it not to protect against scraper DDoS, but due to ideological opposition to AI scrapers. IMHO this is fruitless, as the more intelligent scrapers will find ways around it without calling attention to themselves. Anubis makes almost no sense on a static personal blog.)

1vuio0pswjnm7 1y ago

Looks Anubis allows Internet Archive's bot sometimes Allowed https://web.archive.org/web/20250419222331if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250419222331if_/https://anubis... https://web.archive.org/web/20250419222331if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250419222331if_/https://anubis... https://web.archive.org/web/20250420152651if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250420152651if_/https://anubis... https://web.archive.org/web/20250420152651if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250420152651if_/https://anubis... Blocked https://web.archive.org/web/20250424235436if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250424235436if_/https://anubis... https://web.archive.org/web/20250510230703if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250510230703if_/https://anubis... https://web.archive.org/web/20250511110518if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250511110518if_/https://anubis... https://web.archive.org/web/20250630101240if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250630101240if_/https://anubis... https://web.archive.org/web/20250808051637if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250808051637if_/https://anubis... https://web.archive.org/web/20250909160601if_/https://anubis.techaro.lol/docs/user/known-instances https://web.archive.org/web/20250909160601if_/https://anubis... Allowed https://web.archive.org/web/20250921062513if_/https://anubis.techaro.lol/docs/user/known-instances/ https://web.archive.org/web/20250921062513if_/https://anubis...

aorth 1y ago

There was an interesting comment in the Lobsters thread about this article https://lobste.rs/s/gig2wt/you_don_t_need_anubis https://lobste.rs/s/gig2wt/you_don_t_need_anubis. Basically, Sec-Fetch-* headers are widely available on browsers https://caniuse.com/?search=sec-fetch-dest https://caniuse.com/?search=sec-fetch-dest, so you can detect if a client that says they are Chrome, Firefox, or Safari are really Chrome, Firefox, or Safari. This seems to work in Caddy, using a CEL expression: @unrealistic-browsers <<CEL {header.User-Agent}.matches("(Chrome|Firefox|Safari)") && ! ({header.Sec-Fetch-Dest}.matches("^.+$") && {header.Sec-Fetch-Mode}.matches("^.+$") && {header.Sec-Fetch-Site}.matches("^.+$")) CEL handle @unrealistic-browsers { abort } Maybe there is a better way. And maybe this stops working when all low-effort bots add these headers to their crawlers. BTW if anyone has an invite on Lobsters I would appreciate it. :)