10 ms·

How I protect my Forgejo instance from AI web crawlers

189 points by todsacerdoti 10mo ago

immibis 10mo ago
My issue with Gitea (which Forgejo is a fork of) was that crawlers would hit the "download repository as zip" link over and over. Each access creates a new zip file on disk which is never cleaned up. I disabled that (by setting the temporary zip directory to read-only, so the feature won't work) and haven't had a problem since then. It's easy to assume "I received a lot of requests, therefore the problem is too many requests" but you can successfully handle many requests. This is a clever way of doing a minimally invasive botwall though - I like it.
- bob1029 10mo ago
  > you can successfully handle many requests. There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant. Especially at the scale of a self-hosted forge with a constrained audience. I find this to be a much easier path. I wish we could find a way to not conflate the intellectual property concerns with the technological performance concerns. It seems like this is essential to keeping the AI scraping drama going in many ways. We can definitely make the self hosted git forge so fast that anything short of ~a federal crime would have no meaningful effect.
  - idontsee 10mo ago
    > There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant. It isn't just the volume of requests, but also bandwidth. There have been cases where scraping represents >80% of a forge's bandwidth usage. I wouldn't want that to happen to the one I host at home.
    - immibis 10mo ago
      Sure but how much bandwidth is that actually? Of course if your normal traffic is pretty low, it's easy for bot traffic to multiply that by 5, but it doesn't mean it's actually a problem. The market price for bandwidth in a central location (USA or Europe) is around $1-2 per TB and less if you buy in bulk. I think it's somewhat cheaper in Europe than in the USA due to vastly stronger competition. Hetzner includes 20TB outgoing with every Europe VPS plan, and 1€/TB +VAT overage. Most providers aren't quite so generous but still not that bad. How much are you actually spending?
  - spockz 10mo ago
    Maybe it is fast enough but my objection is mostly due to the gross inefficiency of crawlers. Requesting downloads of whole repositories over and over, leading to storing these archives on disk wasting CPU cycles to create them and storage space to retain them, and bandwidth to sent them over the wire. Add this to the gross power consumption of AI and hogging of physical compute hardware, and it is easy to see “AI” as wasteful.
- userbinator 10mo ago
  Each access creates a new zip file on disk which is never cleaned up. That sounds like a bug.
  - isodev 10mo ago
    I think that’s been fixed in Forgejo a long time ago
- GrayShade 10mo ago
  It used to be like that, but they've changed it to a POST request a while ago.
csilker 10mo ago
Cloudflare has a solution to protect routes from crawlers. https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/
- roywashere 10mo ago
  Sure, but the whole point of self-hosting forgejo is to not use these big cloud solutions. Introducing cloudflare is a step back!
  - csilker 10mo ago
    The way is same with cloudflare. Cloudflare has a documentation and manifest.
    - 63stack 10mo ago
      I'm so fascinated by replies like this, it's too random and nonsensical to be a language barrier issue, but it also does not pattern match into LLM generated text. Reminds me of ~2010 era wordpress comment spam.
- mmarian 10mo ago
  I've written a post on why I don't think it'll succeed: https://developerwithacat.com/blog/202507/cloudflare-pay-per-crawl/ https://developerwithacat.com/blog/202507/cloudflare-pay-per...
reconnecting 10mo ago
tirreno (1) guy here. Our open-source system can block IP addresses based on rules triggered by specific behavior. Can you elaborate on what exact type of crawlers you would like to block? Like, a leaky bucket of a certain number of requests per minute? 1. https://github.com/tirrenotechnologies/tirreno https://github.com/tirrenotechnologies/tirreno
- notachatbot123 10mo ago
  The article is about AI web crawlers. How can your tool help and how would one set it up for this specific context?
  - reconnecting 10mo ago
    I don't see how an AI crawler is different from any others. The simplest approach is to count the UA as risky or flag multiple 404 errors or HEAD requests, and block on that. Those are rules we already have out of the box. It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Plus, we have developed a dashboard for manually choosing UA blocks based on name, but we're still not sure if this is something that would be really helpful for website operators.
    - Roark66 10mo ago
      >It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Depends on the goal. Author wants his instance not to get killed. Request rate limiting may achieve that easily in a way transparent to normal users.
    - mmarian 10mo ago
      > count the UA as risky It's trivial to spoof UAs unfortunately.
      - reconnecting 10mo ago
        It depends. If you want to stop OAI-SearchBot/1.3, UA will be enough.
userbinator 10mo ago
Unfortunately this means, my website could only be seen if you enable javascript in your browser. Or have a web-proxy that matches on the pattern and extracts the cookie automatically. ;-)
apples_oranges 10mo ago
HTTP 412 would be better I guess..
- jsheard 10mo ago
  You shouldn't really serve aggressive scrapers any kind of error or otherwise unusual response, because they'll just take that as a signal to try again with a different IP address or user agent, or a residential proxy, or a headless browser, or whatever else. There's no obligation to be polite to rude guests, give them a 200 OK containing the output of a Markov chain trained on the Bee Movie script instead.
  - loloquwowndueo 10mo ago
    Unless your output is static, you’d then be paying the cost of running the markov generator.
maelito 10mo ago
I'm having lots of connections every day from Singapor. It's now the main country... despite the whole website being French-only. AI crawlers, for sure. Thanks for this tip.
- arjie 10mo ago
  Amazonbot does this despite my efforts in robots.txt to help it out. I look at all the Singapore requests and they’re Amazonbot trying to get various variants of the Special:RecentChanges page. You’re wasting your time, Amazonbot. I’m trying to help you.
  - reconnecting 10mo ago
    Did you check IP address of this UA?
    - arjie 10mo ago
      Yeah, a while ago, they're all Singapore reporting Amazonbot. Here is an example request: "GET /w/A_Wedding_at_City_Hall HTTP/1.1" 200 9677 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36" The actual IP is in X-Forwarded-For and I didn't keep that.
      - reconnecting 10mo ago
        This is just a spoofed UA, because Amazon has a publicly available range of their bots and they are not in Singapore.
        arjie 10mo ago
        That makes sense. I wonder why Amazonbot specifically as a target to spoof. I hoped to get them not stuck using a robots.txt but they refuse to obey it and keep hitting that page with various params. No problem for me, but they are going nowhere.
andai 10mo ago
Can someone help me understand where all this traffic is coming from? Are there thousands of companies all doing it simultaneously? How come even small sites get hammered constantly? At some point haven't you scraped the whole thing?
- reppap 10mo ago
  It's not just companies either, a lot of people run crawlers for their home lab projects too.
- devsda 10mo ago
  May be the teams developing AI crawlers are dogfooding & are using the AI itself(and its small context) to keep track of the sites that are already scraped. /s
- bingo-bongo 10mo ago
  AI companies scrape to: - have data to train on - update the data more or less continuously - answer queries from users on the fly With a lot of AI companies, that generates a lot of scraping. Also, some of them behave terribly when scraping or is just bad at it.
  - adastra22 10mo ago
    Why don’t they scrape once though?
    - blell 10mo ago
      1) It may be out of date 2) Storing it costs money
      - adastra22 10mo ago
        Bandwidth costs more money.
- input_sh 10mo ago
KronisLV 10mo ago
We should just have some standard for crawlable archived versions of pages with no back end or DB interaction behind them etc., for example if there's a reverse proxy, whatever it outputs is archived and it wouldn't actually pass on any call in the archive version. Same for translating the output of any dynamic JS into fully static HTML. Then add some proof-of-work that works without JS and is a web standard (e.g. server sends header, client sends correct response, gets access to archive) and mainstream the culture for low-cost hosting for such archives and you're done, also make sure that this sort of feature is enabled in the most basic configuration for all web servers and such, logged separately. Obviously such a thing will never happen, because the web and culture went in a different direction. But if it were a mainstream thing, you'd get easy to consume archives (also for regular archival and data hoarding) and the "live" versions of sites wouldn't have their logs be bogged down by stupid spam. Or if PoW was a proper web standard with no JS, then ppl who want to tell AI and other crawlers to fuck off, they could at least make it uneconomical to crawl their stuff en masse. In my view, proof of work that would work through headers in the current day world should be as ubiquitous as TLS.
Roark66 10mo ago
I'm glad the author clarified he wants to prevent his instance from crashing not simply "block robots and allow humans". I think the idea that you can block bots and allow humans is fallacious. We should focus on a specific behaviour that causes problems (like making a bajillion requests one for each commit, instead of cloning the repo). To fix this we should block clients that work in such ways. If these bots learn to request at a reasonable pace why cares if they are bots, humans, bots under a control of an individual human, bots owned by a huge company scraping for training data? Once you make your code (or anything else) public, then trying to limit access to only a certain class of consumers is a waste of effort. Also, perhaps I'm biased, because I run a searXNG and Crawl4AI (and few ancillaries like jina rerank etc) in my homelab so I can tell my AI to perform live internet searches as well as it can get any website. For code it has a way to clone stuff, but for things like issues, discussions, PRs it goes mostly to GitHub. I like that my AI can browse almost like me. I think this is the future way to consume a lot of the web (except sites like this one that are an actual pleasure to use). The models sometimes hit sites they can't fetch. For this I use Firecrawl. I use MCP proxy that lets me rewrite the tool descriptions so my models get access to both my local Crawl4ai and hosted (and rather expensive)firecrawl, but they are told to use Firecrawl as last resort. The more people use these kinds of solutions the more incentive there will be for sites not to block users that use automation. Of course they will have to rely on alternative monetisation methods, but I think eventually these stupid capchas will disappear and reasonable rate limiting will prevail.
- asfdasfsd 10mo ago
  And people who block AI crawlers on moral grounds?
- szundi 10mo ago
  [dead]
- popcornricecake 10mo ago
  > I think this is the future way to consume a lot of the web I think I see many prompt injections in your future. Like captchas with a special bypass solution just for AIs that leads to special content.
Simplita 10mo ago
We ran into similar issues with aggressive crawling. What helped was rate limiting combined with making intent explicit at the entry point, instead of letting requests fan out blindly. It reduced both load and unexpected edge cases.
- cheshire_cat 10mo ago
  What do you mean by "making intent explicit at the entry point"?
  - deleted 10mo ago
    [deleted]
BLKNSLVR 10mo ago
I really don't know how effective my little system would be against these scrapers, but I've setup a system that blocks IP addresses if they've attempted to connect to ports on my system(s) behind which there are no services, and therefore their connections must be 'uninvited', which I classify as malicious. Since I do actually host a couple of websites / services behind port 443, it means I can't just block everything that tries to scan my ip address at port 443. However, I've setup Cloudflare in front of those websites, so I do log and block any non-Cloudflare (using Cloudflare's ASN: 13335) traffic coming into port 443. I also log and block IP address attempting to connect on port 80, since that essentially deprecated. This, of course, does not block traffic coming via the DNS names of the sites, since that will be routed through Cloudflare - but as someone mentioned, Cloudflare has its own anti-scraping tools. And then as another person mentioned, this does require the use of Cloudflare, which is a vast centralising force on the Internet and therefore part of a different problem... I don't currently split out a separate list for IP addresses that have connected to HTTP(S) ports, but maybe I'll do that over Christmas. This is my current simple project: https://github.com/UninvitedActivity/UninvitedActivity https://github.com/UninvitedActivity/UninvitedActivity Apologies if the README is a bit rambling. It's evolved over time, and it's mostly for me anyway. P.S. I always thought it was Yog Sothoth (not Sototh). Either way, I'm partial to Nyarlathotep. "The Crawling Chaos" always sounded like the coolest of the elder gods.
- ewpratten 10mo ago
  Regarding the Cloudflare part of this, I’d recommend taking a look at “Authenticated Origin Pulls”. It lets you perform your validation at the TLS layer instead of doing it with IP ACLs if that interests you.
- mmarian 10mo ago
  Some scrapers/scanners use residential IPs. Aren't you worried you'll end up blocking legitimate traffic?
  - BLKNSLVR 10mo ago
    Two points: 1. The websites I run get so little traffic it doesn't matter. They're mostly for my own entertainment / experimentation. 2. If they're allowing their IP address to be used for pricing scrapers then I consider that within the blurry definition of malicious anyway. I don't mind if you disagree with me on point #2, and I grant that if I was running some super popular web service, maybe my tune would change. I have four tiers of scanning paranoia, so I can ramp up and down if need be (I'm not sure if that's documented in GitHub though...)
petee 10mo ago
You can also add a honeypot urls to your robots.txt to trap bots that are using it as an index
s_ting765 10mo ago
I use the same exact trick from the source the article mentions. I call it `temu` anubis. https://github.com/rhee876527/expert-octo-robot/blob/f28e48fc59b2a1b823551dfd1cb5ea6ea8bb7f88/app-charts/caddy-chart/templates/configmap.yaml#L19 https://github.com/rhee876527/expert-octo-robot/blob/f28e48f... Jokes aside, the whole web seems to be trending towards some kind of wall (pay, login, app etc.) and this ultimately sucks for the open internet.
- BLKNSLVR 10mo ago
  You missed the obvious portmanteau: Temubis
pedrozieg 10mo ago
What I like about this approach is that it quietly reframes the problem from “detect AI” to “make abusive access patterns uneconomical”. A simple JS+cookie gate is basically saying: if you want to hammer my instance, you now have to spin up a headless browser and execute JS at scale. That’s cheap for humans, expensive for generic crawlers that are tuned for raw HTTP throughput. The deeper issue is that git forges are pathological for naive crawlers: every commit/file combo is a unique URL, so one medium repo explodes into Wikipedia-scale surface area if you just follow links blindly. A more robust pattern for small instances is to explicitly rate limit the expensive paths (/raw, per-commit views, “download as zip”), and treat “AI” as an implementation detail. Good bots that behave like polite users will still work; the ones that try to BFS your entire history at line rate hit a wall long before they can take your box down.
- nucleardog 10mo ago
  Yeah, this is where I landed a while ago. What problem am I _really_ trying to solve? For some people it's an ideological one--we don't want AI vacuuming up all of our content. For those, "is this an AI user?" is a useful question to answer. However it's a hard one. For many the problem is simply "there are a class of users that are putting way too much load on the system and it's causing problems". Initially I was playing wack-a-mole with this and dealing with alerts firing on a regular basis because of Meta crawling our site very aggressively, not backing off when errors were returned, etc. I looked at rate limiting but the work involved in distributed rate limiting versus the number of offenders involved made the effort look a little silly, so I moved towards a "nuke it from orbit" strategy: Requests are bucketed by class C subnet (31.13.80.36 -> 31.13.80.x) and request rate is tracked over 30 minute windows. If the request rate over that window exceeds a very generous threshold I've only seen a few very obvious and poorly behaved crawlers exceed it fires an alert. The alert kicks off a flow where we look up the ASN covering every IP in that range, look up every range associated with those ASNs, and throw an alert in Slack with a big red "Block" button attached. When approved, the entire ASN is blocked at the edge. It's never triggered on anything we weren't willing to block (e.g., a local consumer ISP). We've dropped a handful of foreign providers, some "budget" VPS providers, some more reputable cloud providers, and Facebook. It didn't take long before the alerts stopped--both for high request rates and our application monitoring seeing excessive loads. If anyone's interested in trying to implement something similar, there's a regularly updated database of ASN <-> IP ranges announced here: https://github.com/ipverse/asn-ip https://github.com/ipverse/asn-ip
mintflow 10mo ago
recently I just noticed github trying(but failed) to charge the self host runners, I find a afternoon to setup a mini PC to install freeBSD and gitaea on it, then setup tailscale to let it only listen on the 100.64.x.x IP address. Since I do not make this node public accessable, so no worry for AI web crawlers:)
frogperson 10mo ago
I think it would be really cool if someone built a reverse proxy just for dealing with these bad actors. I would really like to easily serve some markov chain non-sense to Ai bots.
- jakewil 10mo ago
  perhaps Iocaine [1] is what you're looking for. See the demo page [2] for what it serves to AI crawlers. 1. https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/ 2. https://poison.madhouse-project.org/ https://poison.madhouse-project.org/
  - philipwhiuk 10mo ago
    For images you have stuff like https://nightshade.cs.uchicago.edu/whatis.html https://nightshade.cs.uchicago.edu/whatis.html
  - opem 10mo ago
    This site blocked me right away, seems quite agressive
  - gkbrk 10mo ago
    Seems like a good way to waste tons of your bandwidth. Almost every serious data pipeline has some quality filtering in there (even open-source ones like FineWeb and EduWeb). And the stuff Iocaine generates instantly gets filtered. Feel free to test this with any classifier or cheapo LLM.
flexagoon 10mo ago
Oh hey, I wrote the "you don't need anubis" post you (or the post author, if that's not you) got inspiration from! Glad to hear it helped!
everfrustrated 10mo ago
I think what gets lost in this is that we should expect a lot more traffic from AI if simply for the reason that if I ask AI to answer my question it will do a lot more work and fetch from a lot of websites in generating a reply to me. And yes searching over git repos will absolutely be part of that. This is all "legitimate" traffic in that it isn't about crawling the internet but in service of a real human. Put another way, search is moving from a model of crawl the internet and query on cached data to being able to query on live data.
- hombre_fatal 10mo ago
  In some ways that's true. But when it comes to git repos, an LLM agent like claude code can just clone them for local crawling which is far better than crawling remotely, and it's the "Right Way" for various reasons. Frankly I suspect AI agents will push search in the opposite direction from your comment and move us to distributed cache workflows. These tools just hit the origin because it's the easy solution of today, not because the data needs to be up to date to the millisecond. Imagine a system where all those Fetch(url) invocations interact with a local LRU cache. That'd be really nice, and I think that's where we'd want to go, especially once more and more origin servers try to block automated traffic.
- ethin 10mo ago
  I agree and I think that everyone agreeing or disagreeing with you (and sysadmins everywhere) would be perfectly fine with these AI crawlers (well, mostly...) if these corporations wrote them properly, followed best practices and standards, and didn't effectively DDoS servers or pretend to be what they aren't. Because that is, ultimately, what these AI companies are: very effective, for-sale, legal DDoSers. But they are not written properly, do not follow best practices and standards, and DDoS everything you aim them at, and even go as far as pretending that they're things they aren't, hide behind residential IP addresses (which I'm pretty sure could potentially be illegal because, you know, that risks getting people who have no idea what AI even is in trouble), etc. I don't think AI will replace search now just because so much of the world is blocked from them now, and that is only to increase I'm sure. And, honestly, I doubt there is anything these AI companies could do to make sysadmins actually trust them again anymore.
stronglikedan 10mo ago
> Unfortunately this means, my website could only be seen if you enable javascript in your browser. I feel this is acceptable. I wouldn't be surprised if all this AI stuff was just a global conspiracy to get everyone to turn on JS.
justsomehnguy 10mo ago
A similar approach can be done by writing a cookie by the proxy/webserver itself by visiting some path ie: example.net/sesame/open. For a single user or a small team this could be enough.
agentifysh 10mo ago
never heard of forgejo, should one switch from gitea
- tuananh 10mo ago
  it's a fork of gitea.

189 points by todsacerdoti 10mo ago

My issue with Gitea (which Forgejo is a fork of) was that crawlers would hit the "download repository as zip" link over and over. Each access creates a new zip file on disk which is never cleaned up. I disabled that (by setting the temporary zip directory to read-only, so the feature won't work) and haven't had a problem since then. It's easy to assume "I received a lot of requests, therefore the problem is too many requests" but you can successfully handle many requests. This is a clever way of doing a minimally invasive botwall though - I like it.

bob1029 10mo ago
> you can successfully handle many requests. There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant. Especially at the scale of a self-hosted forge with a constrained audience. I find this to be a much easier path. I wish we could find a way to not conflate the intellectual property concerns with the technological performance concerns. It seems like this is essential to keeping the AI scraping drama going in many ways. We can definitely make the self hosted git forge so fast that anything short of ~a federal crime would have no meaningful effect.
- idontsee 10mo ago
  > There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant. It isn't just the volume of requests, but also bandwidth. There have been cases where scraping represents >80% of a forge's bandwidth usage. I wouldn't want that to happen to the one I host at home.
  - immibis 10mo ago
    Sure but how much bandwidth is that actually? Of course if your normal traffic is pretty low, it's easy for bot traffic to multiply that by 5, but it doesn't mean it's actually a problem. The market price for bandwidth in a central location (USA or Europe) is around $1-2 per TB and less if you buy in bulk. I think it's somewhat cheaper in Europe than in the USA due to vastly stronger competition. Hetzner includes 20TB outgoing with every Europe VPS plan, and 1€/TB +VAT overage. Most providers aren't quite so generous but still not that bad. How much are you actually spending?
- spockz 10mo ago
  Maybe it is fast enough but my objection is mostly due to the gross inefficiency of crawlers. Requesting downloads of whole repositories over and over, leading to storing these archives on disk wasting CPU cycles to create them and storage space to retain them, and bandwidth to sent them over the wire. Add this to the gross power consumption of AI and hogging of physical compute hardware, and it is easy to see “AI” as wasteful.
userbinator 10mo ago
Each access creates a new zip file on disk which is never cleaned up. That sounds like a bug.
- isodev 10mo ago
  I think that’s been fixed in Forgejo a long time ago
GrayShade 10mo ago
It used to be like that, but they've changed it to a POST request a while ago.

csilker 10mo ago

Cloudflare has a solution to protect routes from crawlers. https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/

roywashere 10mo ago
Sure, but the whole point of self-hosting forgejo is to not use these big cloud solutions. Introducing cloudflare is a step back!
- csilker 10mo ago
  The way is same with cloudflare. Cloudflare has a documentation and manifest.
  - 63stack 10mo ago
    I'm so fascinated by replies like this, it's too random and nonsensical to be a language barrier issue, but it also does not pattern match into LLM generated text. Reminds me of ~2010 era wordpress comment spam.
mmarian 10mo ago
I've written a post on why I don't think it'll succeed: https://developerwithacat.com/blog/202507/cloudflare-pay-per-crawl/ https://developerwithacat.com/blog/202507/cloudflare-pay-per...

reconnecting 10mo ago

tirreno (1) guy here. Our open-source system can block IP addresses based on rules triggered by specific behavior. Can you elaborate on what exact type of crawlers you would like to block? Like, a leaky bucket of a certain number of requests per minute? 1. https://github.com/tirrenotechnologies/tirreno https://github.com/tirrenotechnologies/tirreno

notachatbot123 10mo ago
The article is about AI web crawlers. How can your tool help and how would one set it up for this specific context?
- reconnecting 10mo ago
  I don't see how an AI crawler is different from any others. The simplest approach is to count the UA as risky or flag multiple 404 errors or HEAD requests, and block on that. Those are rules we already have out of the box. It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Plus, we have developed a dashboard for manually choosing UA blocks based on name, but we're still not sure if this is something that would be really helpful for website operators.
  - Roark66 10mo ago
    >It's open source, there's no pain in writing specific rules for rate limiting, thus my question. Depends on the goal. Author wants his instance not to get killed. Request rate limiting may achieve that easily in a way transparent to normal users.
  - mmarian 10mo ago
    > count the UA as risky It's trivial to spoof UAs unfortunately.
    - reconnecting 10mo ago
      It depends. If you want to stop OAI-SearchBot/1.3, UA will be enough.

userbinator 10mo ago

Unfortunately this means, my website could only be seen if you enable javascript in your browser. Or have a web-proxy that matches on the pattern and extracts the cookie automatically. ;-)

apples_oranges 10mo ago

HTTP 412 would be better I guess..

jsheard 10mo ago
You shouldn't really serve aggressive scrapers any kind of error or otherwise unusual response, because they'll just take that as a signal to try again with a different IP address or user agent, or a residential proxy, or a headless browser, or whatever else. There's no obligation to be polite to rude guests, give them a 200 OK containing the output of a Markov chain trained on the Bee Movie script instead.
- loloquwowndueo 10mo ago
  Unless your output is static, you’d then be paying the cost of running the markov generator.

maelito 10mo ago

I'm having lots of connections every day from Singapor. It's now the main country... despite the whole website being French-only. AI crawlers, for sure. Thanks for this tip.

arjie 10mo ago
Amazonbot does this despite my efforts in robots.txt to help it out. I look at all the Singapore requests and they’re Amazonbot trying to get various variants of the Special:RecentChanges page. You’re wasting your time, Amazonbot. I’m trying to help you.
- reconnecting 10mo ago
  Did you check IP address of this UA?
  - arjie 10mo ago
    Yeah, a while ago, they're all Singapore reporting Amazonbot. Here is an example request: "GET /w/A_Wedding_at_City_Hall HTTP/1.1" 200 9677 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36" The actual IP is in X-Forwarded-For and I didn't keep that.
    - reconnecting 10mo ago
      This is just a spoofed UA, because Amazon has a publicly available range of their bots and they are not in Singapore.
      - arjie 10mo ago
        That makes sense. I wonder why Amazonbot specifically as a target to spoof. I hoped to get them not stuck using a robots.txt but they refuse to obey it and keep hitting that page with various params. No problem for me, but they are going nowhere.

andai 10mo ago

Can someone help me understand where all this traffic is coming from? Are there thousands of companies all doing it simultaneously? How come even small sites get hammered constantly? At some point haven't you scraped the whole thing?

reppap 10mo ago
It's not just companies either, a lot of people run crawlers for their home lab projects too.
devsda 10mo ago
May be the teams developing AI crawlers are dogfooding & are using the AI itself(and its small context) to keep track of the sites that are already scraped. /s
bingo-bongo 10mo ago
AI companies scrape to: - have data to train on - update the data more or less continuously - answer queries from users on the fly With a lot of AI companies, that generates a lot of scraping. Also, some of them behave terribly when scraping or is just bad at it.
- adastra22 10mo ago
  Why don’t they scrape once though?
  - blell 10mo ago
    1) It may be out of date 2) Storing it costs money
    - adastra22 10mo ago
      Bandwidth costs more money.
input_sh 10mo ago

KronisLV 10mo ago

We should just have some standard for crawlable archived versions of pages with no back end or DB interaction behind them etc., for example if there's a reverse proxy, whatever it outputs is archived and it wouldn't actually pass on any call in the archive version. Same for translating the output of any dynamic JS into fully static HTML. Then add some proof-of-work that works without JS and is a web standard (e.g. server sends header, client sends correct response, gets access to archive) and mainstream the culture for low-cost hosting for such archives and you're done, also make sure that this sort of feature is enabled in the most basic configuration for all web servers and such, logged separately. Obviously such a thing will never happen, because the web and culture went in a different direction. But if it were a mainstream thing, you'd get easy to consume archives (also for regular archival and data hoarding) and the "live" versions of sites wouldn't have their logs be bogged down by stupid spam. Or if PoW was a proper web standard with no JS, then ppl who want to tell AI and other crawlers to fuck off, they could at least make it uneconomical to crawl their stuff en masse. In my view, proof of work that would work through headers in the current day world should be as ubiquitous as TLS.

Roark66 10mo ago

I'm glad the author clarified he wants to prevent his instance from crashing not simply "block robots and allow humans". I think the idea that you can block bots and allow humans is fallacious. We should focus on a specific behaviour that causes problems (like making a bajillion requests one for each commit, instead of cloning the repo). To fix this we should block clients that work in such ways. If these bots learn to request at a reasonable pace why cares if they are bots, humans, bots under a control of an individual human, bots owned by a huge company scraping for training data? Once you make your code (or anything else) public, then trying to limit access to only a certain class of consumers is a waste of effort. Also, perhaps I'm biased, because I run a searXNG and Crawl4AI (and few ancillaries like jina rerank etc) in my homelab so I can tell my AI to perform live internet searches as well as it can get any website. For code it has a way to clone stuff, but for things like issues, discussions, PRs it goes mostly to GitHub. I like that my AI can browse almost like me. I think this is the future way to consume a lot of the web (except sites like this one that are an actual pleasure to use). The models sometimes hit sites they can't fetch. For this I use Firecrawl. I use MCP proxy that lets me rewrite the tool descriptions so my models get access to both my local Crawl4ai and hosted (and rather expensive)firecrawl, but they are told to use Firecrawl as last resort. The more people use these kinds of solutions the more incentive there will be for sites not to block users that use automation. Of course they will have to rely on alternative monetisation methods, but I think eventually these stupid capchas will disappear and reasonable rate limiting will prevail.

asfdasfsd 10mo ago
And people who block AI crawlers on moral grounds?
szundi 10mo ago
[dead]
popcornricecake 10mo ago
> I think this is the future way to consume a lot of the web I think I see many prompt injections in your future. Like captchas with a special bypass solution just for AIs that leads to special content.

Simplita 10mo ago

We ran into similar issues with aggressive crawling. What helped was rate limiting combined with making intent explicit at the entry point, instead of letting requests fan out blindly. It reduced both load and unexpected edge cases.

cheshire_cat 10mo ago
What do you mean by "making intent explicit at the entry point"?
- deleted 10mo ago
  [deleted]

BLKNSLVR 10mo ago

I really don't know how effective my little system would be against these scrapers, but I've setup a system that blocks IP addresses if they've attempted to connect to ports on my system(s) behind which there are no services, and therefore their connections must be 'uninvited', which I classify as malicious. Since I do actually host a couple of websites / services behind port 443, it means I can't just block everything that tries to scan my ip address at port 443. However, I've setup Cloudflare in front of those websites, so I do log and block any non-Cloudflare (using Cloudflare's ASN: 13335) traffic coming into port 443. I also log and block IP address attempting to connect on port 80, since that essentially deprecated. This, of course, does not block traffic coming via the DNS names of the sites, since that will be routed through Cloudflare - but as someone mentioned, Cloudflare has its own anti-scraping tools. And then as another person mentioned, this does require the use of Cloudflare, which is a vast centralising force on the Internet and therefore part of a different problem... I don't currently split out a separate list for IP addresses that have connected to HTTP(S) ports, but maybe I'll do that over Christmas. This is my current simple project: https://github.com/UninvitedActivity/UninvitedActivity https://github.com/UninvitedActivity/UninvitedActivity Apologies if the README is a bit rambling. It's evolved over time, and it's mostly for me anyway. P.S. I always thought it was Yog Sothoth (not Sototh). Either way, I'm partial to Nyarlathotep. "The Crawling Chaos" always sounded like the coolest of the elder gods.

ewpratten 10mo ago
Regarding the Cloudflare part of this, I’d recommend taking a look at “Authenticated Origin Pulls”. It lets you perform your validation at the TLS layer instead of doing it with IP ACLs if that interests you.
mmarian 10mo ago
Some scrapers/scanners use residential IPs. Aren't you worried you'll end up blocking legitimate traffic?
- BLKNSLVR 10mo ago
  Two points: 1. The websites I run get so little traffic it doesn't matter. They're mostly for my own entertainment / experimentation. 2. If they're allowing their IP address to be used for pricing scrapers then I consider that within the blurry definition of malicious anyway. I don't mind if you disagree with me on point #2, and I grant that if I was running some super popular web service, maybe my tune would change. I have four tiers of scanning paranoia, so I can ramp up and down if need be (I'm not sure if that's documented in GitHub though...)

petee 10mo ago

You can also add a honeypot urls to your robots.txt to trap bots that are using it as an index

s_ting765 10mo ago

I use the same exact trick from the source the article mentions. I call it `temu` anubis. https://github.com/rhee876527/expert-octo-robot/blob/f28e48fc59b2a1b823551dfd1cb5ea6ea8bb7f88/app-charts/caddy-chart/templates/configmap.yaml#L19 https://github.com/rhee876527/expert-octo-robot/blob/f28e48f... Jokes aside, the whole web seems to be trending towards some kind of wall (pay, login, app etc.) and this ultimately sucks for the open internet.

BLKNSLVR 10mo ago
You missed the obvious portmanteau: Temubis

pedrozieg 10mo ago

What I like about this approach is that it quietly reframes the problem from “detect AI” to “make abusive access patterns uneconomical”. A simple JS+cookie gate is basically saying: if you want to hammer my instance, you now have to spin up a headless browser and execute JS at scale. That’s cheap for humans, expensive for generic crawlers that are tuned for raw HTTP throughput. The deeper issue is that git forges are pathological for naive crawlers: every commit/file combo is a unique URL, so one medium repo explodes into Wikipedia-scale surface area if you just follow links blindly. A more robust pattern for small instances is to explicitly rate limit the expensive paths (/raw, per-commit views, “download as zip”), and treat “AI” as an implementation detail. Good bots that behave like polite users will still work; the ones that try to BFS your entire history at line rate hit a wall long before they can take your box down.

nucleardog 10mo ago
Yeah, this is where I landed a while ago. What problem am I _really_ trying to solve? For some people it's an ideological one--we don't want AI vacuuming up all of our content. For those, "is this an AI user?" is a useful question to answer. However it's a hard one. For many the problem is simply "there are a class of users that are putting way too much load on the system and it's causing problems". Initially I was playing wack-a-mole with this and dealing with alerts firing on a regular basis because of Meta crawling our site very aggressively, not backing off when errors were returned, etc. I looked at rate limiting but the work involved in distributed rate limiting versus the number of offenders involved made the effort look a little silly, so I moved towards a "nuke it from orbit" strategy: Requests are bucketed by class C subnet (31.13.80.36 -> 31.13.80.x) and request rate is tracked over 30 minute windows. If the request rate over that window exceeds a very generous threshold I've only seen a few very obvious and poorly behaved crawlers exceed it fires an alert. The alert kicks off a flow where we look up the ASN covering every IP in that range, look up every range associated with those ASNs, and throw an alert in Slack with a big red "Block" button attached. When approved, the entire ASN is blocked at the edge. It's never triggered on anything we weren't willing to block (e.g., a local consumer ISP). We've dropped a handful of foreign providers, some "budget" VPS providers, some more reputable cloud providers, and Facebook. It didn't take long before the alerts stopped--both for high request rates and our application monitoring seeing excessive loads. If anyone's interested in trying to implement something similar, there's a regularly updated database of ASN <-> IP ranges announced here: https://github.com/ipverse/asn-ip https://github.com/ipverse/asn-ip

mintflow 10mo ago

recently I just noticed github trying(but failed) to charge the self host runners, I find a afternoon to setup a mini PC to install freeBSD and gitaea on it, then setup tailscale to let it only listen on the 100.64.x.x IP address. Since I do not make this node public accessable, so no worry for AI web crawlers:)

frogperson 10mo ago

I think it would be really cool if someone built a reverse proxy just for dealing with these bad actors. I would really like to easily serve some markov chain non-sense to Ai bots.

jakewil 10mo ago
perhaps Iocaine [1] is what you're looking for. See the demo page [2] for what it serves to AI crawlers. 1. https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/ 2. https://poison.madhouse-project.org/ https://poison.madhouse-project.org/
- philipwhiuk 10mo ago
  For images you have stuff like https://nightshade.cs.uchicago.edu/whatis.html https://nightshade.cs.uchicago.edu/whatis.html
- opem 10mo ago
  This site blocked me right away, seems quite agressive
- gkbrk 10mo ago
  Seems like a good way to waste tons of your bandwidth. Almost every serious data pipeline has some quality filtering in there (even open-source ones like FineWeb and EduWeb). And the stuff Iocaine generates instantly gets filtered. Feel free to test this with any classifier or cheapo LLM.

flexagoon 10mo ago

Oh hey, I wrote the "you don't need anubis" post you (or the post author, if that's not you) got inspiration from! Glad to hear it helped!

everfrustrated 10mo ago

I think what gets lost in this is that we should expect a lot more traffic from AI if simply for the reason that if I ask AI to answer my question it will do a lot more work and fetch from a lot of websites in generating a reply to me. And yes searching over git repos will absolutely be part of that. This is all "legitimate" traffic in that it isn't about crawling the internet but in service of a real human. Put another way, search is moving from a model of crawl the internet and query on cached data to being able to query on live data.

hombre_fatal 10mo ago
In some ways that's true. But when it comes to git repos, an LLM agent like claude code can just clone them for local crawling which is far better than crawling remotely, and it's the "Right Way" for various reasons. Frankly I suspect AI agents will push search in the opposite direction from your comment and move us to distributed cache workflows. These tools just hit the origin because it's the easy solution of today, not because the data needs to be up to date to the millisecond. Imagine a system where all those Fetch(url) invocations interact with a local LRU cache. That'd be really nice, and I think that's where we'd want to go, especially once more and more origin servers try to block automated traffic.
ethin 10mo ago
I agree and I think that everyone agreeing or disagreeing with you (and sysadmins everywhere) would be perfectly fine with these AI crawlers (well, mostly...) if these corporations wrote them properly, followed best practices and standards, and didn't effectively DDoS servers or pretend to be what they aren't. Because that is, ultimately, what these AI companies are: very effective, for-sale, legal DDoSers. But they are not written properly, do not follow best practices and standards, and DDoS everything you aim them at, and even go as far as pretending that they're things they aren't, hide behind residential IP addresses (which I'm pretty sure could potentially be illegal because, you know, that risks getting people who have no idea what AI even is in trouble), etc. I don't think AI will replace search now just because so much of the world is blocked from them now, and that is only to increase I'm sure. And, honestly, I doubt there is anything these AI companies could do to make sysadmins actually trust them again anymore.

stronglikedan 10mo ago

> Unfortunately this means, my website could only be seen if you enable javascript in your browser. I feel this is acceptable. I wouldn't be surprised if all this AI stuff was just a global conspiracy to get everyone to turn on JS.

justsomehnguy 10mo ago

A similar approach can be done by writing a cookie by the proxy/webserver itself by visiting some path ie: example.net/sesame/open. For a single user or a small team this could be enough.

agentifysh 10mo ago

never heard of forgejo, should one switch from gitea

tuananh 10mo ago
it's a fork of gitea.