7 ms·

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html https://honeyp

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t my idea; I link to the inspiration for it. I just included my Caddy-specific instructions for implementing it.)

QuiDortDine 9mo ago
I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.
- wolfi1 9mo ago
  "Turns out careless, unethical vibecoders aren't very competent." well, they rely on AI, don't they? and AI is trained with already existing bad code, so why should the outcome be different?
- tuhgdetzhh 9mo ago
  I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.
  - kstrauser 9mo ago
    It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.
    - raw_anon_1111 9mo ago
      Serious question, in 2026 you can actually have a successful crawler with just curl? I just had to create one for a customer - for their own site - and nothing would have worked without using Chromium.
      - kstrauser 9mo ago
        Probably not for most sites. Example of a site where it'd likely work: a blog made with a static site generator. Example of one where it wouldn't: darn near anything made with React.
        franga2000 9mo ago
        It works for the majority of things a text mining scraper would care to scrape. It's not just static sites but also any CMS like wordpress, as well as many JS apps that have server-side rendering. SPA-only sites aren't that common anymore, especially for things like blogs, news and text-based social media.
  - hxtk 9mo ago
    Even that functions as a sort of proof of work, requiring a commitment of compute resources that is table stakes for individual users but multiplies the cost of making millions of requests.
  - cantalopes 9mo ago
    Well it's a race, just like security. And as long as anubis is in the front, all looks bright
  - gruez 9mo ago
    AFAIK you can bypass it with curl because there's an explicit whitelist for it, no need for a headful browser.
  - solid_fuel 9mo ago
    Cool, if they're running full blown chromium maybe the next step can be mining bitcoin on any pages served to bots.
- Elfener 9mo ago
  > Turns out careless, unethical vibecoders aren't very competent. Well they are scraping web pages from a git forge, where they could just, you know, clone the repo(s) instead.
xorcist 9mo ago
> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page load, and to treat users to opt-in to running code in a document browser with suspicion. Not everyone would like to hand over that power to them on a silver platter with a neat little bow on top.
- kstrauser 9mo ago
  Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of uses, either. Web devs like to write TypeScript and React because that's a very pleasant tech stack for writing web apps, and it's not worth the effort for them to support a deliberately hamstrung browser for < 0.1% of users (according to a recent Google report). See also: feel free to disable PNG rendering, but I'm not going to lift a finger to convert everything to GIFs.
  - shakna 9mo ago
    > JS is as fundamental to current design as CSS. I think this hits the crux of the trend fairly well. And is why I have so many workarounds to shitty JS in my user files. Because I can't see your CSS, either.
    - chipsrafferty 9mo ago
      Yet you use CSS on your own website?
      - shakna 9mo ago
trumpdong 9mo ago
IIRC 2 is included in the "go-away" program which is similar to Anubis but without the PoW
zajio1am 9mo ago
600K request per day is ~ 400/minute. That is very low number. But seems to me that many webapps are so bad that even that small number causes significant load for them.
- testrun 9mo ago
  That is an average. The distribution of requests might be more in bursts.
Akronymus 9mo ago
Couldnt you just set a cookie and send a redirect back? Avoiding the need for js.

QuiDortDine 9mo ago

I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.

wolfi1 9mo ago
"Turns out careless, unethical vibecoders aren't very competent." well, they rely on AI, don't they? and AI is trained with already existing bad code, so why should the outcome be different?
tuhgdetzhh 9mo ago
I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.
- kstrauser 9mo ago
  It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.
  - raw_anon_1111 9mo ago
    Serious question, in 2026 you can actually have a successful crawler with just curl? I just had to create one for a customer - for their own site - and nothing would have worked without using Chromium.
    - kstrauser 9mo ago
      Probably not for most sites. Example of a site where it'd likely work: a blog made with a static site generator. Example of one where it wouldn't: darn near anything made with React.
      - franga2000 9mo ago
        It works for the majority of things a text mining scraper would care to scrape. It's not just static sites but also any CMS like wordpress, as well as many JS apps that have server-side rendering. SPA-only sites aren't that common anymore, especially for things like blogs, news and text-based social media.
- hxtk 9mo ago
  Even that functions as a sort of proof of work, requiring a commitment of compute resources that is table stakes for individual users but multiplies the cost of making millions of requests.
- cantalopes 9mo ago
  Well it's a race, just like security. And as long as anubis is in the front, all looks bright
- gruez 9mo ago
  AFAIK you can bypass it with curl because there's an explicit whitelist for it, no need for a headful browser.
- solid_fuel 9mo ago
  Cool, if they're running full blown chromium maybe the next step can be mining bitcoin on any pages served to bots.
Elfener 9mo ago
> Turns out careless, unethical vibecoders aren't very competent. Well they are scraping web pages from a git forge, where they could just, you know, clone the repo(s) instead.

xorcist 9mo ago

> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page load, and to treat users to opt-in to running code in a document browser with suspicion. Not everyone would like to hand over that power to them on a silver platter with a neat little bow on top.

kstrauser 9mo ago
Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of uses, either. Web devs like to write TypeScript and React because that's a very pleasant tech stack for writing web apps, and it's not worth the effort for them to support a deliberately hamstrung browser for < 0.1% of users (according to a recent Google report). See also: feel free to disable PNG rendering, but I'm not going to lift a finger to convert everything to GIFs.
- shakna 9mo ago
  > JS is as fundamental to current design as CSS. I think this hits the crux of the trend fairly well. And is why I have so many workarounds to shitty JS in my user files. Because I can't see your CSS, either.
  - chipsrafferty 9mo ago
    Yet you use CSS on your own website?
    - shakna 9mo ago

trumpdong 9mo ago

IIRC 2 is included in the "go-away" program which is similar to Anubis but without the PoW

zajio1am 9mo ago

600K request per day is ~ 400/minute. That is very low number. But seems to me that many webapps are so bad that even that small number causes significant load for them.

testrun 9mo ago
That is an average. The distribution of requests might be more in bursts.

Akronymus 9mo ago

Couldnt you just set a cookie and send a redirect back? Avoiding the need for js.