3 ms·
Botnets would not surprise me. By chance can you post one of the log lines for them? Any unique characteristics at layer 7 or even 3/4? Are they using HTTP/1
by LinuxBender 2y ago
Botnets would not surprise me. By chance can you post one of the log lines for them? Any unique characteristics at layer 7 or even 3/4? Are they using HTTP/1.1 or lower? If so try dropping anything that is not HTTP/2.0 when you are under attack. This can break unmaintained API clients. All major browsers support HTTP/2.0 whereas many old bots do not.
if ($server_protocol != HTTP/2.0) { return 444; }
Another thing to check for is cors and navigate headers which bots often lack (Nginx example):
if ($http_sec_fetch_mode !~ (cors|no-cors|navigate) ) { return 444; }
If you run:
tcpdump -p --dont-verify-checksums -i any -NNnnttvvv -B32768 -c16384 -s0 proto 6 and port 443 and 'tcp[13] == 2'
do you see nominal MSS values such as 1280 to 1460? If not try dropping any SYN packets that are either missing MSS or have a strange value. In Netfilter IPTables in the raw table it looks like:
iptables -t raw -I PREROUTING -d ${My_IP} -i eth0 -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 1280:1460 -j DROP
Test these things in a staging environment of course. These things assume you have a device at the edge that can do something similar or that you are running iptables on the node itself and that gets into topics of state table size, stateless rules, etc... Example from the raw tables
-A PREROUTING -i lo -j NOTRACK
-A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j NOTRACK
-A OUTPUT -o lo -j NOTRACK
-A OUTPUT -o eth0 -p tcp -m tcp --sport 443 -j NOTRACK
Otherwise the attacker can fill your state tables even if you increase the size.
Another common option is to require an account and then regardless of IP address limit how many requests per (second/minute/hour/day) can be made by said account and limit that account to existing in less than 3 CIDR blocks per day or perhaps 3 /16's per day and then make it harder to botters to mass create accounts. All of this should be adjustable on the fly by DEFCON status. If you are not under attack DEFCON 5, ease the restrictions so the site is more usable and friendly. If under attack DEFCON 1 then rules and account restrictions are tightly enforced but still usable by real people using the site the way it was intended.