9 ms·
Using lots of little tools to aggressively reject the bots
- loloquwowndueo 1y agoNice - I like that most of the ai scraper bot blocking was done using Nginx configuration. Still, once fail2ban was added to the mix (meaning: additional service and configuration), I wonder if considering something like Anubis (https://anubis.techaro.lol/ https://anubis.techaro.lol/) would have been more automatic. I’ve seen Anubis verification pages pop up more frequently around the web!
- rovr138 1y agoFWIW, the reason I like their approach is that fail2ban is still lean, works off of the same logs, and doesn't start with the requirement to affect everyone's experience due to bad actors.
- nulbyte 1y agoMaking the visitor put up with your bot because you don't like someone else's is just bizarre to me. I disable js by default, so Ive been seeing this more and more. It's all rubbish these days.
- Proofread0592 1y agoIt is nice that the AI crawler bots honestly fill out the `User-Agent` header, I'm shocked that they were the source of that much traffic though. 99% of all websites do not change often enough to warrant this much traffic, let alone a dev blog.
- grishka 1y agoThey also respect robots.txt. However, I've also seen reports that after getting blocked one way or another, they start crawling with browser user-agents from residential IPs. But it might also be someone else misrepresenting their crawlers as OpenAI/Amazon/Facebook/whatever to begin with.
- rovr138 1y agoWe ended up writing similar rules to the article. It was just based on frequency. While we were rate limiting bots based on UA, we ended up also having to apply wider rules because traffic started spiking from other places. I can't say if it's the traffic shifting, but there's definitely a big amount of automated traffic not identifying itself properly. If you look at all your web properties, look at historic traffic to calculate <hits per IP> in <time period>. Then look at the new data and see how it's shifting. You should be able to identify the real traffic and the automated very quickly.
- deleted 1y ago[deleted]
- cratermoon 1y ago> They also respect robots.txt All the reports I've heard from organizations dealing with AI crawler bots say they are not honest about their user agent and do not respect robots.txt "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/ https://xeiaso.net/notes/2025/amazon-crawler/
- eesmith 1y ago
- reconnecting 1y agoCreator of tirreno [1] here. While our platform is primarily designed for live, logged-in users, it also works well for bot detection and blocking. We anonymize IP addresses by replacing the last octet with an asterisk, effectively grouping the same subnet under a single account. You can then use the built-in rule engine to automatically generate blacklists based on specific conditions, such as excessive 500 or 404 errors, brute-force login attempts, or traffic from data center IPs. Finally, you can integrate tirreno blacklist API into your application logic to redirect unwanted traffic to an error page. Bonus: a dashboard [2] is available to help you monitor activity and fine-tune the blacklist to avoid blocking legitimate users. [1] https://github.com/tirrenotechnologies/tirreno https://github.com/tirrenotechnologies/tirreno [2] https://play.tirreno.com/login https://play.tirreno.com/login (admin/tirreno)
- reconnecting 1y agoWe also have work in progress to block bots based on publicly available IP ranges through the same dashboard. Any suggestions are welcome.
- mindslight 1y ago> We anonymize IP addresses by replacing the last octet with an asterisk, effectively grouping the same subnet under a single account So as a user, not only do I have to suffer your blockwall's false positives based on "data center IPs" (ie most things that aren't naively browsing from the information-leaking address of a last-mile connection like some cyber-bumpkin). But if I do manage to find something that isn't a priori blocked (or manage to click through 87 squares of traffic lights), I still then get lumped in with completely unrelated address-neighbors to assuage your conscience that you're not building a user surveillance system based on nonconsentually processing personal information. Just please make sure you have enough of a feedback process that your customers can see that they are losing real customers with real dollars.
- reconnecting 1y agoYou're right, blanket blocking based on IP ranges (like TOR or DC) often creates false positives and punishes privacy-conscious users. Therefore, unlike the traditional way of blocking an IP just because it is from a data center, tirreno suggests using a risk-based system that takes into account dozens or hundreds of rules. As in my example, if the IP is from a data center and creates a lot of 404 errors, send it to a manual review queue or to automatic blocking (not recommended). Personally, I prefer to manually review even bot activity, and tirreno, even if it's not directly designed for bot management, works great for this, especially in cases when bad bots are hidden behind legitimate bot UA's.
- rovr138 1y agoGreat article and sleuthing to find the information. I know you're processing them dynamically as they come in and break the rules. But if you wanted to supplement the list, might be worth sourcing the ones from https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt at some frequency.
- sneak 1y agoYou don’t have to fend off anything, you just have to fix your server to support this modest amount of traffic. Everyone else is visiting your site for entirely self-serving purposes, too. I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site. If you publish data to the public, expect the public to access it. If you don’t want the public (this includes AI startups) to access it, don’t publish it. Your website is not being misused when the data is being downloaded to train AI. That’s literally what public data is for.
- red369 1y agoIs it because people viewed it as Google scraping the site to make an index so that people could find the site, while the AI scraping is intended so people won’t need to visit the site at all?
- ehutch79 1y agoAlso, google is relatively considerate when crawling.
- DamonHD 1y agoUsually though from time to time over many many years from when G was still wet behind the ears I have had to write specific defences effectively just to deal with it.
- ehutch79 1y agoYeah, “Relatively” is doing a lot of heavy lifting there. I remember people with lax security and a tags that went to deletion endpoints… we all learned a lot back in the day.
- owebmaster 1y agoThe AI apps (namely chatgpt and Claude) are evolving to display external data with widgets that will potentially drive more traffic than google has been doing for a long time. Might be worth changing focus as SEO killed google.
- vachina 1y agoI’ve turned off logging on my servers precisely because it’s growing too quickly due to these bots. They’re that relentless, and would fill every form, even access APIs otherwise accessible only by clicking around the site. Anthropic, openAI and Facebook are still scraping to this day.
- davidmurdoch 1y ago> even access APIs otherwise accessible only by clicking around the site How else?
- eGQjxkKF6fif 1y agoWould you mind sharing information on these crawlers accessing APIs only usable for clicking around on websites? And to clarify, It's a part of the UI or something and only a human should be pressing it, and there's no other way to access that API or something? AI agents exist now, there is virtually no way to distinguish between real user and bot if they mimic human patterns.
- vachina 1y agoThey’re using the sign up and sign in forms, and also the search, and then clicking on those search results. I thought some bad actor is masquerading as AI scrapers to enumerate accounts, but their behavior is consistent with a scraper.
- eGQjxkKF6fif 1y agoRead this: https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/ https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-... Basically everybody's a bot in a hidden botnet now. And we agreed to it. Phones, tablets, Windows appstores add the SDK in, and then drone. One of the big ones is "Infatica" - devs get paid to put this in legitimate 'apps' AI companies, and whoever and whatever else use the reputation-good IPs to hammer sites and well, it's fair game for all malicious people.
- atomman747 1y agoI get that the author “doesn’t care anymore”, but I saw Google, ripe.net, and semrush in the banned IPs. Of these, I certainly wouldn’t ban Google, and probably not the others, if I wanted others to see it and talk about it. Even if your content were being scraped for some rando’s AI bot, why have a public site if you don’t expect your site to be used? Turning the lights off on the motel sign when you want people to find it is not a good way to invite people in.
- DamonHD 1y agoSemrush misbehaved so badly for such a long time with various levels of incompetence that I have special notes in my robots.txt files going back at least 8 years and eventually got someone in legal to pay attention. I see zero value to me or any potential users of mine in letting an 'SEO' firm lumpishly trample all over my site barging out real visitors. And some of Semrush's competitors were just as bad. The current round of AI nonsense also very poor. Again had to send legal notes to investor relations and PR depts in at least one well-known case, as well as all the technical measures, to restore some sort of decorum.
- akudha 1y agoMy experience with them has been the same. On one of my employer's websites, the top three bots were Google, Bytedance and Semrush. It is a small website for a niche audience, not even in English, and changes very infrequently (like once or twice a quarter). That did not stop these three bots from hammering the site, every second
- hinkley 1y agoWe were getting hammered on nofollow links. Someone made a change to stop putting the nofollow in and using JavaScript to traverse on interaction, but that person left sometime after the toggle went in and we discovered it had never been turned on much later, when we were rearranging our ingress load balancer and needed to tweak how we handled bot throttling. That was at least 10% of the requests during peak traffic and extended the duration of the crawls by a lot. So dumb.
- miladyincontrol 1y agoA lot of less scrupulous crawlers just seem to imitate the big ones. I feel a lot of people make assumptions because the user agent has to be true, right? My fave method is still just to have bait info in robots.txt that gzip bombs and autoblocks all further requests from them. Was real easy to configure in Caddy and tends to catch the worst offenders. Not excusing the bot behaviours but if a few bots blindly take down your site, then an intentionally malicious offender would have a field day.
- horsawlarway 1y agoYour last comment feels pretty apt. Maybe I'm just a different generation than the folks writing these blog posts, but I really don't understand the fixation on such low resource usage. It's like watching a grandparent freak out over not turning off an LED light or seeing them drive 15 miles to save 5c/gallon on gas. 20 requests per second is just... Nothing. Even if you're dynamically generating them all (and seriously... Why? Time would have been so much better spent fixing that with some caching than this effort) it's just not much demand. I get the "fuck the bots" style posts are popular in the Zeitgeist at the moment, but this is hardly novel. There are a lot more productive ways to handle this that waste a lot less of your time.
- hananova 1y agoBack in the 90's, me as a dumb kid got a phone call from my ISP telling me that my system had been enlisted in a botnet and that they were turning off my connection until I fixed it. Maybe it's time to once again block entire ASN's of ISP's that tolerate being used as residential proxies, and this shit might end.
- NitpickLawyer 1y agoThe residential proxies are not offered by ISPs but by world-wide users (knowingly or not) installing dubious software in exchange for access / pennies / whatever. There was a pretty good article about it a while ago on hn.
- aorth 1y agoYep! Eye opening report on residential proxies from Trend Micro. https://www.trendmicro.com/vinfo/us/security/news/vulnerabilities-and-exploits/a-closer-exploration-of-residential-proxies-and-captcha-breaking-services https://www.trendmicro.com/vinfo/us/security/news/vulnerabil...
- vhcr 1y agoAnd? If your network allows malware you should be blocked.
- StressedDev 1y agoYou would end up blocking every network then. Almost no one wants malware on their machine or network. Unfortunately, people get hacked and network operators cannot always determine which machines are hacked (hackers are not known for letting people know that they have taken over a machine). Security is hard and there are no easy solutions. People often do not know they have been hacked or even know if a computer on their network has been hacked. Also, it is often not easy to determine if traffic is legitimate, illegal, malicious, or abusive.
- out-of-ideas 1y agohmm, might as well extend this hypothetical a bit more... Epic would argue Apple as a whole is malware, and therefore all networks with apple devices on them are malware, block them all? it does not work that way. the open internet goal is to be open. "AI" is generally meant to be intelligent, indistinguishable to humans; we created the problem for ourselves, dismissing so many (back then, and now) theoretical problems. what i fear are increased operating costs for the end users to "prove they are human". what i hope happens is cheaper infra (bandwidth ect) to be able to support the increased demand. and, a big hope that "bots" which are respectful [but this too wont hold up, people are not 100% respectful, why will their bots be?] so we will be stuck with more captcha related crap, more websites that must use javahelll, even a "trusted browser" (walling out any new browsers, and things like curl), maybe websites that want your ID too (the "think of the children!" folks would be pleased), and ways to "trick" the bots (and not the real users) to ban the bots.... yay
- RamRodification 1y agoI could be wrong, but the comment for the rate limiting (1r/s) doesn't seem to line up with the actual rate limit being set (5r/s). # Define a rate limit of 1 request per second every 1m limit_req_zone $binary_remote_addr zone=krei:10m rate=5r/s;
- curtisszmania 1y ago[dead]
- datavirtue 1y ago[flagged]
- tasuki 1y agoI wonder, why do people maintain personal git forges? Is it so others can submit issues and pull requests? Do others actually bother to create accounts at your personal git forge so they can contribute?
- BlueTemplar 1y ago[flagged]
- kassner 1y agoPersonal choices. I want to selfhost all my stuff, and avoid relying on GitHub, and also having some codebases not being used in AI training. I achieve the latter by not having my forge public, so I don’t have much of a dog in this fight. I could just have a directory in another server, but I like being able to link to a specific file/line/commit, and I prefer viewing the code in the browser in some situations. Forgejo takes the same effort to selfhost as gitweb on my setup, so why not?
- tasuki 1y agoOk, fair. I have several remotes for my projects: one on a machine I control, another on GitHub, so I can send links to people etc.
- degamad 1y agoYep, mine is for my wife and a couple of friends to contribute only, and they're able to do so just fine. There are no public repositories on it, and the url is not published anywhere.
- nullc 1y agoI can't use mouser or digikey anymore thanks to anti-crawler crusading.
- opan 1y agoI have always gotten the short end of the stick with this stuff as well. Captchas are annoying, Cloudflare challenges are usually not passable, etc. For some sites I use regularly, they'll log me out and stop me logging in a few times per year and then I have to tweet at them about it, which is usually followed by something about them having just turned up their bot protection settings, but then often they'll fix whatever they did and I can get back in. Would it be so crazy to just let it happen? When ordinary users are suffering as much or more than the "criminal" targets, maybe it's not worth it. I understand there are cases where bandwidth usage would become too much, but there's gotta be a better way. Maybe let some bots through so that fewer humans are blocked.
- sethops1 1y agoBlame the people shitting in the pool. Not the people closing down the pool because people keep shitting in it.
- ThomW 1y agoIt ticks me off that bots no longer respect robots.txt files at all. The authors of these things are complete assholes. If you’re one of them, gfy.
- cyanydeez 1y ago[flagged]
- whatevermom 1y ago[flagged]
- oblio 1y agoWhy do you do it?
- jeffhuys 1y agoProfit in terms of knowledge or just actual money (guessing)
- whatevermom 1y agoFor Fun & Profit
- thowaway7564902 1y agoYou might as well say gfy to anyone using chat bots, search engines and price comparison tools. They're the one's who financially incentivize the scrapers.
- mandmandam 1y agoThat doesn't logic. Giving someone a "financial incentive" to do something (by gasp using a search engine, or comparing prices) does not make that thing ethical or cool to do in and of itself. I wonder where you ever got the idea that it does.
- aswegs8 1y agoAh, good old botkilling. Gotta love those.
- immibis 1y agoI consider the disk space issue a bug in Gitea. When someone downloads a zip, it should be able to stream the zip to the client, but instead it builds the zip in temporary space, serves it to the client, and doesn't delete it. I solved it by marking that directory read-only. Zip downloads, obviously, won't work. If someone really wants one, they can check out the repository and make it theirself. If I really cared, of course I'd fix the bug or make sure there's a way to disable the feature properly or only enable it for logged-in users. Also I server-side redirect certain user-agents to https://git.immibis.com/gptblock.html https://git.immibis.com/gptblock.html . This isn't because they waste resources any more but just because I don't like them, what they're doing is worthless anyway, and because I can. If they really want the data of the Git repository they can clone the Git repository instead of scraping it in a stupid way. That was always allowed. 8 requests per second isn't that much unless each request triggers intensive processing and indeed it wasn't a load on my little VPS other than the disk space bug. I blocked them because they're stupid, not because they're a DoS.
- dakiol 1y agoOpenAI is the biggest offender according to my server’s logs. Nevertheless we keep praising them over here. We are so stupid.
- DamonHD 1y agoWho's "we"?
- nickpsecurity 1y agoMy sites get 10-20k requests a day. Mostly AI scrapers. One thing I noticed is many look for specific, PHP pages. If you dont use PHP, you might be able to autoblock any IP requesting PHP pages. If you have PHP, block those requesting pages you dont have. Some of us are happy to train AI's but want to block overload. For instance, I'm glad they're scraping pages about the Gospel and Biblical theology. It might help to put anything large that you dont want scraped into specifi directories. Then, upon detecting a bot, block the IP from accessing those. In my case, I also have a baseline strategy to deal with a large number of requests. That's text only, HTML/CSS presentation, other stuff externally hosted, and BunnyCDN with Perma-Cache ($10/mo + 1 penny / GB). The BunnyCDN requests go to $5/mo. VM's on Digital Ocean. I didnt even notice AI scrapers at first since (a) they didn't affect performance and (b) a month of them changed my balance from $30 to $29.99. (Note to DO and Bunny team members that may be here: Thanks for your excellent services.)
- AStonesThrow 1y agoChris Siebenmann has shared many thoughts about crawlers, aka spiders. https://utcc.utoronto.ca/~cks/space/?search=spider https://utcc.utoronto.ca/~cks/space/?search=spider https://utcc.utoronto.ca/~cks/space/?search=crawlers https://utcc.utoronto.ca/~cks/space/?search=crawlers It's interesting how in 2005 he was advocating for playing nicely, because yes, search engines were hypothetically driving traffic to sites they crawled. And there don't seem to be a lot of complaints about performance hits. Though typically, his comments and observations are restricted to his blog's corner of his department.
- fluidcruft 1y agoI've been wondering if there's anything like "page knocking" like you open a series of pages in a certain order and that grants access? For example maybe everything 404s unless you start with a specific unlisted url
- blharr 1y agoThis would be user unfriendly regardless of how you design around it. If I save a bookmark or send a link to a friend, they're going to get a 404.
- fluidcruft 1y agoIt can be less unfriendly than requiring them to create an account and login. You could send 403 instead.
- chongli 1y agoHow does that allow authorized users to get access? Note that they want regular people to be able to find the project from a SERP and start browsing without creating an account or authenticating in any way.
- fluidcruft 1y agoI think it depends on what you mean by authorized users. If you're saying everyone is authorized then relying on public scrapers isn't going to work with unlisted pages. Search engines are bots, too. I'm thinking more if it's people coming from email or QR code or link from other sites that keep bots out. It can also be more about sequence of accessing pages. The idea is having behavior unlock the site. Because it should be pretty easy to cause people and bots to behave differently. For example, you could set it up such that access to gitea requires simply accessing the documentation site first.
- chongli 1y agoThe issue here is with publicly hosted git repositories for open source projects. Everyone is authorized, including anonymous users arriving from Google.
- BLKNSLVR 1y agoShould we be moving to a push / submit methodology for centralised indexing such as AI and search? No scraping, if I want you to read it I'll send it to you.
- deleted 1y ago[deleted]
- xelxebar 1y agoMy little server just hums along, and I haven't checked fail2ban status recently. sudo fail2ban-client status | sed -n '/Jail list:/{s/^.*://; s/,//g; p}' | xargs -n1 sudo fail2ban-client status | awk '/jail:/{i=$5}; /Total failed:/{jail[i]=$5}; END{for(i in jail) printf("%s: %s\n", i, jail[i])}' | column -t sshd-ddos: 0 postfix: 583 dovecot: 9690 postfix-sasl: 4227 nginx-botsearch: 1421 nginx-http-auth: 0 postfix-botnet: 5425 sshd: 202157 Yikes! 220,000 banned IPs...
- reconnecting 1y agoWe have one bot identified as 'DotBot/1.2' that, as of today, has made over 220,000 requests in the past two weeks.It randomly requests filenames and folders from the web server. I'm not banning it out of curiosity, to see if there's a limit to how deep it digs.
- kazinator 1y agoOver the past while, I have settled on a four component approach. I place less emphasis on banning; it has turned out to be a nuisance for legit users. 1. User-Agent pattern for various known bots. These gets sent to a honeypot (currently implementing zip bombs). 2. Arithmetic captcha page (called "captxa") protecting most of my /cgit tree, which is the primary target for scrapers. Solve a formula, get a persistent cookie that grants you access. 3. Static IP-level bans. The bulk of the unwanted traffic has been from Alibaba Cloud LLC addresses, so I got a list of their blocks and made rules against them all. 4. Log-scan-and-ban system: now mainly just targeting excessive activity over a short period. About 2: I define the same cookie value given to everyone right inside the Apache httpd.conf. This is passed via en environment variable to the captcha page, a CGI script. When the captcha is solved, and the client has that cookie, it will be the Apache rules themselves which check it. The RewriteRule which redirects to the captcha page is guarded by a RewriteCond which checks the value of %{HTTP_COOKIE} for the presence of the required cookie. I use the Apache <Macro ...> feature to define the anti-bot rules. Then the rules can be included in any VirtualHost block easily by invoking the macro. The skeleton of the system looks like this: Define captxa_secret SecretGoesHere <Macro nobots> # Push captxa secret into environment, for the CGI script. SetEnv CAPTXA_SECRET ${captxa_secret} # These agents get sent to /honeypot RewriteCond %{HTTP_USER_AGENT} .*(ezoom|spider|crawler|scan|yandex|coccoc|github|python|amazonbot[Pp]etalbot|ahrefsbot|semrush|anthropic|Facebot|meta|openai|GPT|OPR/|Edg/).* [NC] RewriteCond %{REQUEST_URI} !honeypot RewriteRule ^.*$ /honeypot [R,L] # Clients which have the cookie with the captxa_secret get a pass RewriteCond %{HTTP_COOKIE} "^(.*; ?)?captxa=${captxa_secret}(;.*)?$" RewriteRule ^.*$ - [L] # Rules for no-cookie clients: # Root level /cgit site: ok for all RewriteRule ^/cgit/$ - [L] # First level of /cgit: the project pages: OK RewriteRule ^/cgit/([^/]+)/?$ - [L] # Certain project sub-pages like about, spanshot, README: Pass for hassle-free landing. RewriteRule ^/cgit/([^/]+)/(about|snapshot|refs|tree/RELNOTES|tree/README|tree/README\.md)/?$ - [L] # Some distros fetch TXR tarball snapshots from CGIT; make it easy for them. RewriteRule ^/cgit/txr/snapshot/txr-([0-9]+)\.tar\. - [L] # Everyone else off to captxa script: which will install the cookie # for those clients who solve it. $0 is the entire URL; # it gets passed to the script as a parameter, so the # script can redirect to that URL upon a successful solve. RewriteRule ^/cgit/?(.*)$ /captxa/?quiz=$0? [R,L,NE,QSA] </Macro> Inside a VirtualHost I just do RewriteEngine on Use nobots # include above macro
- mleonhard 1y agoThe website blocks the privacy proxy (VPN) service I use.