4 ms·

He's quite tame compared to me I suppose. I block anything that is not HTTP/2.0 which currently knocks out all the bots and all crawlers except Bing. But I ju

He's quite tame compared to me I suppose. I block anything that is not HTTP/2.0 which currently knocks out all the bots and all crawlers except Bing. But I just have hobby sites these days. Nobody would notice or care if my sites went offline. Using NGinx as an example: if ($server_protocol != HTTP/2.0) { return 403 'Nope'; } Another thing I have found useful to drops some bots is to become invisible to them. Many of the poorly written scanning tools do not properly set MSS for reasons I still don't understand. I use this to my advantage. Using IPTables as an example: /sbin/iptables -t raw -I PREROUTING -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 420:16384 -j DROP Any TCP packets setting a very low or high MSS or missing MSS will be silently dropped. I drop about 35K packets per host per day on average. This also drops hping3 floods.

judge2020 4y ago
IMO blocking bots isn't too big of a concern, the problem is when a dedicated attacker realizes you serve valuable data (in your HTML). Next thing you know, they're running puppeteer or a similar remote controlled browser to scrape your site, which is both undesirable in itself and the scraper might overload your site/database by scraping with no internal parallel request limit. If you're not a startup with an unlimited early cloud budget, it can be costly if you want to handle both bot usage (including official API-based or scraping bots) and regular users.
toast0 4y ago
> Many of the poorly written scanning tools do not properly set MSS for reasons I still don't understand. MSS issues attract me like a moth to flame [1], so let me ask some questions. It looks like this is dropping syns with MSS over 16384??? That is indeed a pretty crazy high number. 9000ish seems reasonable for someone on a jumbo network without a mss clamping router, but above that is someone weird for sure. Under 420 seems unlikely too, but technically acceptable, but sure, I'd drop it. In theory, a proper OS will send several SYNs with MSS, then assume your server doesn't support TCP options and send you a SYN with no options. Going to take a while, but if someone legitimately has a mss less than 536, their internet is probably pretty junky anyway, so ok, seems fine. [1] I just built a browser based pmtud test site, http://pmtud.enslaves.us/ http://pmtud.enslaves.us/
- LinuxBender 4y ago
  You are right. I just happen to use a very safe range. If I didn't care about anyone using jumbo frames I could set the range to 1220:1536 and nearly all legit traffic would pass just fine. 1220 (to 13xx) for the people using VPN's and ip6-ip4 gateways. I just try to give really conservative examples so that it is less likely I break someones unusual setup. Anything just over 9k is fine for most jumbo-frame setups. All of this said, I could set the range to 1:65536 and it would still drop most bots as they don't even bother to set MSS at all in their scans. I'm not sure which tool they are using.
  - LinuxBender 4y ago
    As a follow-up to this, I experimented with dropping any packet that does not have the exact MSS of 1460. Nearly all spam bots and crawlers are gone but I would expect that some people using VPN's would also be blocked. Oddly enough most of the bots in Asia are exactly 1398 and most of the bots in Russia are 1424. I find that interesting.

judge2020 4y ago

IMO blocking bots isn't too big of a concern, the problem is when a dedicated attacker realizes you serve valuable data (in your HTML). Next thing you know, they're running puppeteer or a similar remote controlled browser to scrape your site, which is both undesirable in itself and the scraper might overload your site/database by scraping with no internal parallel request limit. If you're not a startup with an unlimited early cloud budget, it can be costly if you want to handle both bot usage (including official API-based or scraping bots) and regular users.

toast0 4y ago

> Many of the poorly written scanning tools do not properly set MSS for reasons I still don't understand. MSS issues attract me like a moth to flame [1], so let me ask some questions. It looks like this is dropping syns with MSS over 16384??? That is indeed a pretty crazy high number. 9000ish seems reasonable for someone on a jumbo network without a mss clamping router, but above that is someone weird for sure. Under 420 seems unlikely too, but technically acceptable, but sure, I'd drop it. In theory, a proper OS will send several SYNs with MSS, then assume your server doesn't support TCP options and send you a SYN with no options. Going to take a while, but if someone legitimately has a mss less than 536, their internet is probably pretty junky anyway, so ok, seems fine. [1] I just built a browser based pmtud test site, http://pmtud.enslaves.us/ http://pmtud.enslaves.us/

LinuxBender 4y ago
You are right. I just happen to use a very safe range. If I didn't care about anyone using jumbo frames I could set the range to 1220:1536 and nearly all legit traffic would pass just fine. 1220 (to 13xx) for the people using VPN's and ip6-ip4 gateways. I just try to give really conservative examples so that it is less likely I break someones unusual setup. Anything just over 9k is fine for most jumbo-frame setups. All of this said, I could set the range to 1:65536 and it would still drop most bots as they don't even bother to set MSS at all in their scans. I'm not sure which tool they are using.
- LinuxBender 4y ago
  As a follow-up to this, I experimented with dropping any packet that does not have the exact MSS of 1460. Nearly all spam bots and crawlers are gone but I would expect that some people using VPN's would also be blocked. Oddly enough most of the bots in Asia are exactly 1398 and most of the bots in Russia are 1424. I find that interesting.