4 ms·
Over the past while, I have settled on a four component approach. I place less emphasis on banning; it has turned out to be a nuisance for legit users. 1. User
by kazinator 1y ago
Over the past while, I have settled on a four component approach. I place less emphasis on banning; it has turned out to be a nuisance for legit users.
1. User-Agent pattern for various known bots. These gets sent to a honeypot (currently implementing zip bombs).
2. Arithmetic captcha page (called "captxa") protecting most of my /cgit tree, which is the primary target for scrapers. Solve a formula, get a persistent cookie that grants you access.
3. Static IP-level bans. The bulk of the unwanted traffic has been from Alibaba Cloud LLC addresses, so I got a list of their blocks and made rules against them all.
4. Log-scan-and-ban system: now mainly just targeting excessive activity over a short period.
About 2: I define the same cookie value given to everyone right inside the Apache httpd.conf. This is passed via en environment variable to the captcha page, a CGI script. When the captcha is solved, and the client has that cookie, it will be the Apache rules themselves which check it. The RewriteRule which redirects to the captcha page is guarded by a RewriteCond which checks the value of %{HTTP_COOKIE} for the presence of the required cookie.
I use the Apache <Macro ...> feature to define the anti-bot rules. Then the rules can be included in any VirtualHost block easily by invoking the macro.
The skeleton of the system looks like this:
Define captxa_secret SecretGoesHere
<Macro nobots>
# Push captxa secret into environment, for the CGI script.
SetEnv CAPTXA_SECRET ${captxa_secret}
# These agents get sent to /honeypot
RewriteCond %{HTTP_USER_AGENT} .*(ezoom|spider|crawler|scan|yandex|coccoc|github|python|amazonbot[Pp]etalbot|ahrefsbot|semrush|anthropic|Facebot|meta|openai|GPT|OPR/|Edg/).* [NC]
RewriteCond %{REQUEST_URI} !honeypot
RewriteRule ^.*$ /honeypot [R,L]
# Clients which have the cookie with the captxa_secret get a pass
RewriteCond %{HTTP_COOKIE} "^(.*; ?)?captxa=${captxa_secret}(;.*)?$"
RewriteRule ^.*$ - [L]
# Rules for no-cookie clients:
# Root level /cgit site: ok for all
RewriteRule ^/cgit/$ - [L]
# First level of /cgit: the project pages: OK
RewriteRule ^/cgit/([^/]+)/?$ - [L]
# Certain project sub-pages like about, spanshot, README: Pass for hassle-free landing.
RewriteRule ^/cgit/([^/]+)/(about|snapshot|refs|tree/RELNOTES|tree/README|tree/README\.md)/?$ - [L]
# Some distros fetch TXR tarball snapshots from CGIT; make it easy for them.
RewriteRule ^/cgit/txr/snapshot/txr-([0-9]+)\.tar\. - [L]
# Everyone else off to captxa script: which will install the cookie
# for those clients who solve it. $0 is the entire URL;
# it gets passed to the script as a parameter, so the
# script can redirect to that URL upon a successful solve.
RewriteRule ^/cgit/?(.*)$ /captxa/?quiz=$0? [R,L,NE,QSA]
</Macro>
Inside a VirtualHost I just do
RewriteEngine on
Use nobots # include above macro