18 ms·

Fail2ban Sucks (2020)

98 points by ementally 3y ago

BSDobelix 3y ago
First blacklistd would be better, because as written the applications knows better what to "ban" than a log-scanner. I often use a combination of: -blacklistd and fail2ban (for applications who don't support blacklistd) -pfbadhost (nearly always) -CrowdSec (just sometimes, but atm pretty happy with the free service) I use those to have clean logs, block spammers at ip level, and "marketing" crawlers...well and other stuff. And no, ssh is never on 22, i like clean logs to better find the real "interesting" stuff.
lamontcg 3y ago
I've used fail2ban and a nonstandard ssh port before, for like 12 years. Never had any of the issues this article screams at you about, didn't take any time to setup. I spent a lot more time locking down the sshd_config itself and maintaining that over time. I do agree that is more important to do. Fuck me, I guess.
egberts1 3y ago
One time I had an inadvert DDoS against my non-recursive authoritative DNS server and no way to stop the flooding of log file. Problem is DNS query can come from anywhere so latching to the top-level domain (TLD) name became a must. Fail2ban, to the rescue. https://egbert.net/blog/articles/troubleshooting-regex-in-fail2ban-during-bind9-ddos.html https://egbert.net/blog/articles/troubleshooting-regex-in-fa...
hwpky 3y ago
This post is accurate, fail2ban does suck. But that's not a reflection on the fail2ban maintainers, the engineering work that's gone into building it, its usability, bugs or indeed anything else connected to its implementation. It sucks simply because it's solving the problem in the wrong way. There is no sensible security onion, or defence in depth model for open ports in front of private systems. If the technical solution for keeping private, or limited access systems actually and effectively private on the public Internet requires open ports, it's the wrong solution. Unless we're running services designed for public and anonymous access, exposing technologies like VPN servers, SSH or any other software for private consumption to the Internet is a mistake. When we do this we're putting out invitations for abuse and pulling the burden of security responsibility onto ourselves. I'd argue however, that unless it's a full time concern for us, we're close to the project or maintainers, and able to contribute security fixes to source code, we're probably ill equipped to directly handle that responsibility and remain dependent on engaging with the surrounding security eco-system to remain secure. I'm not knocking those security eco-systems, I'm just pointing out that there's a time lag inherent on depending on best-efforts from third parties which creates windows of vulnerability, and open ports on infrastructure ALWAYS puts us on the back foot. From getting patches applied in a timely fashion to the time and convenience cost of updating ACLs each time IPs change. Even if we get the maintenance tasks right 100% of the time, we're still open to the risks of something outside of our control happening, like zero day being sprayed across open ports. The U.S. Government correctly issued an executive order mandating Federal Agencies move to adopt Zero Trust principles just six days after the Colonial Pipeline ransomware attack in 2021. Despite the marketing hype which has followed, if we take nothing else away from this motion, it should be that opening ports in our networks to access private systems using the public Internet is over. There are at least 90 projects and businesses today dedicated to building modern private access technologies which allow secure remote connections and access to private networks without opening firewall ports. Many are commercial options serving businesses, but there are also lots of compelling open source offerings for non-commercial use too. There's a directory of vendors and technologies here https://zerotrustnetworkaccess.info/ https://zerotrustnetworkaccess.info/ which attempts to dispense with some the Zero Trust marketing BS and instead focus on technical discourse, architecture and approach which some might find helpful. Disclosure; founder of one of the businesses (enclave.io) with a commercial interest in this space.
deleted 3y ago
[deleted]
aborsy 3y ago
It’s unnecessary, at least for SSH, in the first place.
theandrewbailey 3y ago
> * changing your SSH port to non-root port (>1024) means that if the SSH daemon dies, any userland process can take its' place. this is bad. If you're relying on privileged ports, you've already lost the battle.
- lxgr 3y ago
  This doesn't even have to happen maliciously. Even if it happens by accident, it can result in locking yourself out of a server.
- closewith 3y ago
  Defense in depth.
- o11c 3y ago
  Specifically: * your SSH client will check the host certificate. There's a reason it does that prompt for new hosts! * a userland process can no more MITM that than any random middlebox can.
- pepa65 3y ago
  And you can always use net.ipv4.ip_local_reserved_ports to reserve the port for privileged use. (I have never had to do this in over a decade of using a high port for ssh access.)
plz-remove-card 3y ago
On {Free,Net}BSD[1][2] you get Bl*cklistd which is a much better approach than parsing log files. [1] https://man.freebsd.org/cgi/man.cgi?query=blacklistd&sektion=8&format=html https://man.freebsd.org/cgi/man.cgi?query=blacklistd&sektion... [2] https://man.netbsd.org/blocklistd.8 https://man.netbsd.org/blocklistd.8
- loeg 3y ago
  The primary author died in 2019. Is it still being worked on? I agree it seems like a superior approach.
- sedatk 3y ago
  Isn't that specifically DoS oriented? Doesn't seem to cover fail2ban's use case.
  - BSDobelix 3y ago
    The difference is blacklistd is just a api and a database for telling the firewall to block ip's. The decision to inform blacklistd is on the application side, and not a scanner that checks the log's produced by that application. And yes, DoS is a very good example, if your application writes so much log's (during a attack) and fail2ban has to analyze them during that, then machine could very well DoS itself, the blacklistd approach is much simpler and straightforward BUT (and that's a big one) the applications has to support it.
    - sedatk 3y ago
      Thanks!
  - plz-remove-card 3y ago
    No, once an IP had failed authentication a configured number of times, the IP is blocked (for a configurable amount of time.)
    - sedatk 3y ago
solarpunk 3y ago
hell yeah, brother
ufmace 3y ago
I agree with this viewpoint. IMO, a lot of these types of threads fill up fast with gimmicky things that sound good at first but do nothing to increase security against realistic threats, plus are at least inconvenient to set up and maintain, if not prone to misfiring and blocking you. The real advice for setting up new servers IMO is, SSH pubkey auth is perfectly adequate security. Set it up normally on the default port logging in as root, disable password login, and don't worry about the noisy failed bot logins. This is a lot more convenient for automated setup and maintenance and has no practical loss of security.
- sedatk 3y ago
  If you shouldn't worry about noisy failed logins, what's the purpose of monitoring login logs? Do you mean to stop monitoring them altogether?
  - ufmace 3y ago
    Yes. I don't think there's anything to be gained from monitoring login logs. The only real use is after you recognize some sort of breach to try and help determine exactly what did or didn't happen.
kseifried 3y ago
Fail2ban could be potentially useful in an era of passwords, especially when systems allowed anyone to login remotely, and you only had the one server because servers were very expensive. However, the moment the industry went to SSH Keys en masse, and got away from passwords, fail2ban stop serving any real purpose other than to make people feel like they had done something to improve security. Which it didn’t really, especially if you enforced key usage only for logins. Literally, the only argument you can make is that fail2ban might reduce the number of log entries.
- zimpenfish 3y ago
  > the moment the industry went to SSH Keys en masse [...] fail2ban stop serving any real purpose That assumes people only use it for sshd, no? Which isn't the case [anecdatum] for me, at least - I use it for ssh, http (several servers), imap, smtp, mqtt, etc. Make a request for phpMyAdmin pages on my server? 10 day timeout for you, sir. IMAPS connection with incorrect username? 10 days in the bin. etc. Yeah, it reduces log entries but it also reduces unwanted resource consumption and keeps my servers working instead of chugging under non-useful load.
- seized 3y ago
  Fail2ban can parse other logs and taken other actions. I have it on my NVR VM, reading the logs from the old Unifi NVR software. Three failed attempts and Fail2ban runs a script that does an API call to my OpnSense firewall to ban that IP from every port. Forever. Haproxy also monitors the HTTP response codes and drops the TCP connection, allowing the block to happen once the bad IP tries to reconnect. So it's not just for password based SSH.
kuon 3y ago
While I agree with most of the points, I think the most important one is: your time is precious. Spending hours setting up things is are really high price to pay for something that might never be of use.
- GuB-42 3y ago
  Setting up fail2ban is essentially just "apt install fail2ban", it already comes with sensible defaults. That's on Debian, I think it is the same on most distros. For me, it is less of a hassle than looking at logs flooded with failed connection attempts.
  - justsomehnguy 3y ago
    > it already comes with sensible defaults Just two days ago I found out the sensible defaults doesn't include automagically enabling even the basic jails. The machine run for a year (or more) with enabled fail2ban which... did nothing. Not a Debian machine and it's entirely on me, but fail2ban setup is not "just apt install". You need to be at least somewhat familiar with it's lingo and the minimals on working with it.
  - zamadatix 3y ago
    Genuine question: Why not just tell the daemon to stop logging failed attempts? I personally have a hard time reconciling local username+password authentication and the need to audit failed attempts in the same use case but that may just be because I haven't run across the right use case.
    - justsomehnguy 3y ago
      If someone would fail to properly code something you can find your basic auth is slow to a crawl or even not responding but you would have no idea what is going on. [0] ie Morris worm
    - GuB-42 3y ago
      The simplest reason is that installing fail2ban is just as easy. Also, I sometimes want to know about failed attempts, mostly for legitimate attempts that failed and shouldn't have. As in: "why couldn't bob connect the the server?", and failed attempts would just show that he used the wrong password, or at least, that it is a password related problem.
throwaway892238 3y ago
I've stopped trying to convince people against using things like Fail2ban. As long as it's not a company or government where security is important, let them waste their time.
- masfoobar 3y ago
  I would be interested on hearing your reasons for this.
michaelt 3y ago
> at worst, fail2ban: causes you massive inconvinience or total lockout [...] it'll probably block you out. I agree that fail2ban is useless once you've disabled password login, which you should have done. But is anyone really getting locked out of their own systems by fail2ban unless they're doing something pretty weird? I thought it gave you about 5 password attempts, and then blocked you for 10 minutes - doesn't seem that draconian to me.
- kleiba 3y ago
  In one of my previous jobs, it blocked me out after 3 attempts indefinitely - that is, until I got a hold of the sysadmin and asked him to unblock me. It was a PITA, as you can imagine. And I don't understand why any sysadmin would want to put themselves in a position to get bothered regularly by people who tend to mistype their passwords like me.
- Scaevolus 3y ago
  The default configuration of fail2ban on Debian considers every offered key as a login attempt, so you can be immediately banned for merely connecting with too many keys in your ~/.ssh.
- baz00 3y ago
  Never been locked out when I used to use it. But it got replaced with a default firewall rule to only allow SSH from my address in the end. Didn’t need it at all then!
zamadatix 3y ago
In general I agree with the recommendations here. Though it may be slightly over prescriptive in what security approaches universally make sense (e.g. privileged ports) it carries a very valid conclusion about the applicability of Fail2Ban. The only time I almost used Fail2Ban was with a personal "guaranteed to work from wherever and whatever I'm on" jumphost which ran SSH on 22 and 443. 443 because, like the post says, outbound filter compatibility (Fun note: to listen on multiple ports simultaneously you can literally just include multiple "Port xyz" lines in your sshd_config). Ultimately I decided for this use case it made more sense to just let bad attempts to fail silently by changing the logging config. If I was ever in a situation I felt I needed the failed attempts logged I'm not sure I could simultaneously justify username+password login on a public host.
NateEag 3y ago
The author makes a number of good points. In fail2ban's defense, it can be used for a lot more than just blocking IP addresses based on SSH failures. At one point in the past I adapted an existing fail2ban setup to ban IP addresses that were running malicious vulnerability scans against a large legacy PHP webapp, using some dumb heuristics that worked and were easy to put in place. Would it have stopped sufficiently motivated expert attackers? Of course not. Did it reduce the load on the legacy PHP app and improve the signal to noise ratio in our webserver logs, while reducing the chance a brute-force attacker would find some hole lurking in a long-forgotten corner of the app's unfortunately-large attack surface? Yup. So, I'd argue it can be useful, even if naive SSH blocking may not offer much improvement.
- Retr0id 3y ago
  I think the log SnR point is a great one. Debugging aside, if you're able to filter out the "obviously malicious" traffic, any more sophisticated attacks will be easier to spot (or at the very least, analyse after-the-fact).
rigid 3y ago
fail2ban is fine for effectively slowing brute-force attacks for services that can't by themselves or where significant complexity would be needed. it's easy to setup and can stop multi stage attacks that generate quirky log output. EDIT: fail2ban vulnerabilities? what is this guy talking about?
- marcus0x62 3y ago
  > EDIT: fail2ban vulnerabilities? what is this guy talking about? There have been several: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=fail2ban https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=fail2ban. Mostly DoS. At least one RCE, admittedly with a non-default fail2ban config and a somewhat unlikely attack chain: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-32749 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3274...
  - rigid 3y ago
    > At least one RCE, admittedly with a non-default fail2ban config and a somewhat unlikely attack chain: you had to manipulate answers from a whois server. Sure, it increases attack surface (like any additional piece of code) but argueing to better let an IP hammer your mailserver with infinite stuffed credentials just to avoid possible bugs, is questionable at least. fail2ban's CVE track record is quite good in comparision, if you take that as metric. It might be even more secure than using multiple different rate-limiting implementations that were written in C.
    - marcus0x62 3y ago
      Yes, as I said, the attack chain was unlikely to be used it practice.
WesolyKubeczek 3y ago
I disagree with just one point. Non-standard ports. They work wonders against lots and lots of traffic from Chinese bots which are too stupid to see that password logins are disabled but there's a fucking lot of them, they just won't stop knocking, and it does take cycles to fend them off in any way. You stop listening on port 22, and 99% of that shit traffic just goes away. I know, it's not by the book, and that well ackchyually I should yak yak this and yada yada that. But it's so fucking cheap and simple to do, and it goes brrr, so I just don't care.
- WesolyKubeczek 3y ago
  Case in point: > changing your SSH port to non-root port (>1024) means that if the SSH daemon dies, any userland process can take its' place. this is bad. Man, if you have sshd that likes to die a lot, you've got bigger problems already. If you have userland processes that can take your sshd port over maliciously, you're already fucked and should really reprioritize your worries. > changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working. ~/.ssh/config goes brrrrrr > you will very likely waste hours of your life. Only if I need to dictate the contents of that file to my iPhone, otherwise it's seconds. > you will have to google "rsync ssh non standard port" every time you want to use rsync rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr. Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin? > you will have to remember scp flags. this is also bad. probably worse. scp from OpenSSH honours OpenSSH configs, so it goes brrrr. > you will have to tell the port to anyone else who tries legitimately accessing your system. "Here's a fragment that you copy and paste into your ~/.ssh/config. Remember to do a chmod 0600 on it if the file is not there already." My god, is it me getting old, but when I was young it was customary to know the basics of your tools.
  - lxgr 3y ago
    > userland processes that can take your sshd port over maliciously, It doesn't need to happen maliciously. The result is the same: You might be locked out permanently. > if you have sshd that likes to die a lot, you've got bigger problems already It doesn't have to die, it can also just happen during startup. Let's say you initially configure port 2222, restart sshd, everything works well, so you remove port 22, restart one more time, and stop thinking about it. Then, on your next reboot, it turns out that port 2222 is actually the preferred port of some other automatically starting service that just happened to not be running during your experiment, but which is now racing sshd for port 2222: That's bad. > rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr. And just as many applications including an SSH client won't pick up the config from .ssh/config (for example because they can't read that directory because they're sandboxed). > Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin? SSH is used by all kinds of people, not all of them condescending sysadmins.
out-of-ideas 3y ago
close the port open if you dont want ssh to be used. i think i read once the rsync.net dude uses port-knocking and maybe f2b? what was the other note from rachaelbythebay, mac os updates always revert changes sshd settings or something like that; maybe they stopped? i do hate the idea of ip whitelisting; IP's should be more dynamic and less able to fingerprint an individual - but that is opening another can of wurms. Generally never trust ANY ip; dont use whitelists, and the blacklists should never be "forever". if your app is secured by a keypair like openssh then thats the route to take. honestly though fail2ban is great in that it does silence some noise in the logs - there will still be an unbareable amount - but at least some less; but still regular log maint is required. Also the groups Ive seen never recommend f2b for use to fight against a DoS attack; not even for "cleaning up logs" - i dont see where that point was created other than from this page
- kemotep 3y ago
  I wouldn’t call it a whitelist necessarily but there’s no reason why ssh has to be open to the world if it is possible to use a network overlay or vpn. If ssh is only allowed on the management wireguard network, not much can beat that other than a host compromise which, additional checks to verify that a host can be trusted at that moment could potentially help stop unauthorized access.
- rsync 3y ago
  I do, indeed, highly value port knocking and I use it in my personal infrastructure. It is simple and elegant and we're coming up on ~20 years of deployment in various places with zero hangs or crashes or unexpected interactions. While nobody should rely solely on port knocking, it is a wonderful layer added to a defense-in-depth strategy. Which is to say: lock down everything the way you always do, then put the port behind a knock.
  - out-of-ideas 3y ago
    I actually want to do this (one day i'll get to nerd-out again) but then tie it to a totp setup so the knocking-algorithm changes ever so slightly; could even utilize different ports
unixhero 3y ago
I use fail2ban. It is a mild security control which I know can fail. I install it, use its defaults and get on with my life. At times, for bastion hosts, I use crowdsec instead. I could not capture why the author was so bitter. I guess I would be the same if I was supporting some product, and its users were getting problems because of fail2ban - and I had to support it for them.
dig1 3y ago
I'm not fond of this writing style without concrete substance; it feels immature. Yes, the author may have tried to be funny, but I don't see a compelling reason to stop using fail2ban. Let's be clear here - fail2ban will not save you if you have a poorly configured SSH server or widely open Nginx/Postfix services, but it will offload traffic blocking to iptables (or whatever firewall you are using). This way, the mentioned services will not have to waste resources just to cancel some connections. Will it save it against large DDoS? Surely not, but it will stop mild DDoS without problems, which it is designed for. We can argue that iptables/nftables can do a fail2ban job with rate limiting and other techniques many people have never heard of, but if you are at this level, you already know your craft, and I'm pretty sure you considered, tried or used fail2ban before.
klaushardt 3y ago
* changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working. you will very likely waste hours of your life. you will have to google "rsync ssh non standard port" every time you want to use rsync. you will have to remember scp flags. this is also bad. probably worse. --- I dont need to remember non standard ports if i just use my .ssh/config. Host myhost.tld Hostname 127.0.0.1 Port 333 User MyUser IdentityFile ~/.ssh/myhost.tld
- lxgr 3y ago
  This works fine until you find yourself in a shitty public Wi-Fi that blocks outgoing traffic to "non-essential ports". But then again, SSH is often not considered essential and you need a VPN or other tunnel on TCP/443 anyway...
- oliwarner 3y ago
  Right! It's dead simple. Normally I might put a title like this down to opinion but here, the author suffers not knowing enough about the subject to justify having an opinion, let alone publish one. Fail2ban is a stupidly easy way to block lazy hacking scripts. It's easy to extend to handle simple honeypot services. It's no replacement for real security but it makes focussing easier.
- theatomheart 3y ago
  THIS. I use non standard port for SSH by default and my single SSH config file manages this for me with no extra effort. Seems like a common sense behavior to me.
- thelittleone 3y ago
  Another solution is tailscale. Disable all admin ports (including ssh) on public IP.
- bshipp 3y ago
  I was brand new to managing an Ubuntu Hetzner server and the moment I saw how many port 22 scans the server received i decided to try changing the port number, followed by key-only passwordless logins. My logs immediately shrank in size. I have never once had an issue having moved to non standard ports and, moreso, feel almost naked logging into port 22. I know security through obscurity is not an answer, but judging by the reduction in port scanning i've seen after moving as many standard ports as possible to new addresses above 20000 I have to believe its a reasonable first step. How many script kiddies are scanning all 65500 ports for each IP address?
devaiops9001 3y ago
The approach my team takes is we put Wireguard gateways running on OpenBSD in front of network services. We go a step further and very carefully select what chips are used for the network interfaces and also attach a hardware random number generator to add to entropy. Internal users' machines have these tunnels transparently loaded and running, configured by our internal GitOps-driven config management (SaltStack). Has been pretty comfy so far.
- ahoka 3y ago
  Kinda sounds like pure security theater, TBH. I don’t mean the jump hosts, but all the other stuff.
kazinator 3y ago
FreePBX/Asterisk comes with a default fail2ban setup that will lock out your SIP connections (from phones) while you're just fiddling to get it working, adding to the frustration.
kazinator 3y ago
I use something that I made almost exactly ten years ago: txrban. https://www.kylheku.com/cgit/txrban/ https://www.kylheku.com/cgit/txrban/ It's way simpler. There is a tiny core to it, and a couple of modules specific to Exim, Apache, SSH. These just run as daemons on the machine. I tried using fail2ban at that time, but it wasn't at all what I expected. It was way overcomplicated for what it achieves, and poorly documented. Just getting it running at all was fiddly, never mind the custom behaviors I had in mind. I should review the Apache module; some of it is quite out of date, due to the changing landscape. Also, that webserver it is used with is now behind a reverse proxy, which only receives connections for specific applications.
- pepa65 3y ago
  What is 'txr'? The script requires it to run the modules.
  - kazinator 3y ago
    That's this: https://www.kylheku.com/cgit/txr/ https://www.kylheku.com/cgit/txr/ Home paged here: https://nongnu.org/txr https://nongnu.org/txr
tzs 3y ago
A particularly fun combination is Fail2Ban on a mail server and Apple's Mail desktop email client when you need to change your password. We've got to change passwords every 90 days at work, and probably 90% of the times I try to do this I end up getting banned by Fail2Ban. The problem apparently is that Apple Mail is pretty aggressive at retrying if a login fails, and so if it tries to connect before you have finished changing the password in the client it can try enough times in quick succession to trigger Fail2Ban. There are at least 3 places you can change your password(s) for Apple Mail. • Invoke "Mail/Settings..." in Mail to open the settings dialog. Select the "Accounts" tab. Then for the account you want to change go to the "Server Settings" pane. That has password fields for your incoming mail server and your outgoing mail server. • Invoke "/System Settings...", then go to the "Internet Accounts" settings. (You can also get there from "Mail/Account..." in Mail). Select you mail account, click the "Detail..." button, and you'll get a dialog that has a password field. Note it only has one password field. I'm not sure what you are supposed to do if you have different passwords for you incoming and outgoing mail servers. • Open "Keychain Access", and search for your incoming mail server. Look for an entry whose kind is "application password" named com.apple.account.IMAP.password. (I assume that IMAP would be POP3 if you use a POP3 server). Open it, verify that the Account field is you@that_server. Then you can click "Show password" to reveal the current password. Change that to the new password and save. Do the same for your outgoing mail server, looking for com.apple.account.SMTP.password. My current procedure is to first disable my work mail account in "Mail/Settings...", then change the password on the server, then update the client side passwords via Keychain access, and then via "Mail/Settings...", then enable my work mail account. That seems to usually work, although once it didn't until I also changed it in "Internet Accounts" in the system settings. I also had a time where I just could not get the new password to take until I rebooted. Apple Calendar is also a pain with password changes, but we don't have Fail2Ban on our calendar server (it is not on a public facing network so there would be no point) so it is not as annoying as Mail. For Calendar what I've found to mostly work is: 1. In "Calendar/Settings..." in the settings for my work account in the Accounts tab turn off all delegates. Then disable my work account. The quit Calendar. On MacOS prior to 13 find and kill the CalendarAgent and CalNCService processes. On 13 they went away. 2. Change the password on the server. 3. Change the password in "Keychain Access". I find the entry by searching for name@calendar_server. The right entry is the "application password" named "com.apple.account.CalDAV.password". 4. Then change the password in "Internet Accounts". As with Mail, Calendar provides convenient access to this via "Calendar/Accounts...". 5. Open Calendar, reenable the account, and turn back on delegates. All of the above comes from trial and error and I have no idea how much of it is actually necessary. I haven't had the patience to do the time consuming experimentation it would take to figure out the actual simplest way to change these passwords reliably and without hitting the server with the old password during the changing.
macNchz 3y ago
> what i'm saying is, if you're fucked, you're fucked. fail2ban will not help I see this attitude frequently in online discussions around {security thing}, but it really doesn't match my understanding of reality. It's not about making any one part of the system perfectly secure, but instead about having all of the pieces of it contribute to a whole. Many real-life stories of caught/thwarted attacks include the threat actor successfully breaching/moving through multiple layers of a system but ultimately getting stuck on something relatively simple. Defense in depth is not just a catchphrase. In evaluating whether something like this is genuinely useful, it can make sense to look at how difficult it is to operate, and how much it might reduce the probability of a successful attack. In my experience, fail2ban has been extremely easy to install and operate, and I think it brings at least some meaningful impact (particularly if applied to other things than SSH). At the very least it can cause someone making a focused attack to have work more slowly and take alternative approaches–"noisy" behavior that may itself trigger an alarm and get a human involved.
- ddtaylor 3y ago
  I agree, defense by depth.
- djoletina 3y ago
  I agree completely. Security is not black and white, same goes for agents that are malicious, if you’re protected from script kiddies by changing your ssh port and adding fail2van then it’s a net positive any way you look at it. It’s important to be aware of course that you’ll need to go a step further if you want to protect from more competent attackers, but what percentage of us will be a target of one of those.
- rstuart4133 3y ago
  You're being far too polite. He's just plain wrong on this point: > do not change your standard ports. do not use fail2ban. That used to be my attitude too. But reality intruded. Example: I ran a SIP Server for a business. SIP Servers are very attractive targets because if you get past the security, you can make then call paid numbers and make real money. Knowing that, I used huge security tokens. No one got past them. But - it didn't stop them from trying. 100's of times per second at times. The CPU load created by processing the login attempts was so great the SIP server started dropping calls. As it happens, asking the kernel to drop packets from miscreant IP's takes far, far less CPU. Fail2ban to the rescue. Example: Move on a few years, and now spammers costs have dropped so much it is worth their while to target IMAP, POP and SMTP servers. The story is the same: they never got through, but despite that their attempts increased to the point the server became overloaded. I can't listen on a different port. Fail2ban to the rescue. Example: Move on to the last couple of years. I move my personal stuff to a VPS. To save costs it's very memory constrained. A problem soon arises. After a few weeks of trouble free operation it keeps slowing down, and sometimes becoming unresponsive. Turns out it's running out of memory; the OOM killer is running amok. That's odd, as there should be almost no load on this thing. Well, apart from ssh login attempts it seems. Turns out there is one of them every few seconds. They have no hope of succeeding of course as I only allow pubic key auth, but the load it creates is a problem. Fail2ban to the rescue. I don't see anything wrong with the principle, which is you block these attacks at the cheapest point possible. That happens to be the point of entry, before you spend CPU cycles processing the packet (even in TCP). Fail2ban leverages that by using the kernel's networking stack. It's forced to glean the information on who to ban from log files, which is messy, error prone and fragile. I don't like that either, but I'll take what I can get.
bioneuralnet 3y ago
If you can't prevent some users from having passwords, it's a decent, low-effort way to discourage intrusion attempts. One place it DOESN'T necessarily work great is in k8s. A few years ago I had to migrate an ancient VM running OpenSSH (for SFTP only) to our k8s cluster. 99% of the users had passwords instead of keys. Easy enough to run sshd in a container with a mounted volume, but I wanted something to recognize and block repeated bad login attempts (the old VM had no such protections). sshd doesn't do it natively, fail2ban wasn't a good option since you generally don't want a pod messing with a k8 node's iptables. What I ended up with was a Ruby script with 3 threads: One to monitor the auth log for failed logins, a second to lock accounts with repeated failures during a given window, and a third to unlock them after some period (and in the darkness bind them). Definitely some caveats in there, but it was an improvement over the status quo.
theginger 3y ago
Always use multiple layers of defence. A sturdy 6ft fence and a locked wooden front door can be more secure than a 50m impenetrable stone wall, if it turns out that there is no such thing as an impenetrable stone wall. Brute force protection is completely unnecessary for openssh when you've disabled password authentication because it's impossible[really expensive] to brute force. Until you are impacted by an authentication vulnerability which still requires a valid user, then you are brute forcing a user away from being totally compromised. All of a sudden doing all the "unnecessary" things, like brute force protection, disabling root logins and not using default accounts make the difference between you being at the front or back of the line to get compromised in the race to get patched. If you are one of those who believes a vulnerability like that is extremely unlikely rather than inevitable, then consider this. An ssh key gets leaked, the system it has access to is known / discoverable, the username is not. Brute force protection is now the main difference between being highly at risk and still reasonable safe. It's a contrived scenario, but hardly an outrageous one.
- o11c 3y ago
  A sturdy 6-foot fence keeps your yard looking nice, even if you still have to lock the front door.
shishcat 3y ago
I remember having a friend with an extemely sluggish server. Turns out the reason behind this sluggishness was that he set a permanent fail2ban bantime, which added thousands of records to iptabes; any packet, even not SSH, the server had to check the source IP against thousands of records. After purging the rules and switching the bantime the server was usable again.
- pepa65 3y ago
  This should happen in-kernel and is lightning-fast, so should not be an issue. Maybe a pre-netfilter box..?
  - 72deluxe 3y ago
    Big iptables lists are indeed incredibly slow; use ipset for large lists instead.
j45 3y ago
Fail2ban is better than nothing at all. Fail2ban has never seemed to be the sophisticated or final solution, just a bare minimum. Inconvenience to users is definitely a form of the security vs. Convenience debate.
pierat 3y ago
This article sucks. Fail2Ban can prevent password spraying from working, since an eventual good password will still be blocked if done within the fail2ban block period. And those block periods aren't publicized. And given appropriate configuration, is another layer of security a potential threat has to deal with.
mkasberg 3y ago
On a related note, the number of HTTP 404 errors I see to /wp-login.php on my static website is ridiculous. (But absolutely not security risk.)
- pepa65 3y ago
  That's what you get running a webserver. This is what should happen (but you could choose not to log those...)
  - NateEag 3y ago
    You could also temp-ban those IPs for a few hours and save yourself some CPU and bandwidth. It may not seem like much, but I spent a year or two adminning a >1 million reqs / day cluster, and it does add up. Auto-banning this type of obvious offender (with fail2ban, as it happens) did result in a noticeable improvement for us.
    - pepa65 3y ago
      You could indeed (temp-)ban them based on logs. Of course it is a huge improvement, if you catch offending IPs early, they never hit any app.
LAC-Tech 3y ago
Having full logs sucks more.
patja 3y ago
Every attack of significance I see these days uses rotating IP addresses.
- pepa65 3y ago
  If the reuse of a single IP address is not prevalent, these are hard to act against. Then you need a different kind of mitigation.
  - patja 3y ago
    They usually follow a predictable pattern, like trying to hit every (almost entirely invalid) URL starting /product/1 /product/2 /product/3 and on up through the millions. They also will rotate the User Agent but neglect realistic/authentic values for other request headers.
lostmsu 3y ago
Would you call fail2ban essentially an application firewall?
- pepa65 3y ago
  It adds rules to in-kernel netfilter tables. On Linux at least, there is really no such thing as a firewall app, there are just different ways to add and manage rules. The only 'approved' default app is nft (nftables), but any app can act on the in-kernel tables.
mgkimsal 3y ago
> you will have to remember scp flags. this is also bad. probably worse Worse? How? Hrm... If I don't use commands for a while, yes, I may have to remember them. If that's the tradeoff for one extra layer providing even a modicum of security... why is this a bad thing? We have things like bash scripts... where you can just store the common flags you use for common things. I wonder how often the author port tunnels over ssh? I usually have to go search for the syntax for ... 30 seconds. Then I have it. Or it might be in my history already. "You might have to remember something" seems about the weakest argument against changing ssh ports I've ever read.
usr1106 3y ago
I have always changed my ssh port and I don't see any convincing argument in this rant not continuing to do so. However, scanners have changed drastically during the last 5-10 years. I remember when you saw close to none login attempts when you used a port between 15000 and 20000. Nowdays there are 100s every day. I guess on 22 it's much worse, but I have not used 22 just for the sake of comparing. For fail2ban it's the same: I know it's not a great solution (lightweight code base, better security gains and lower risk to affect legitimate users would be great), but there is no convincing argument and no better alternative in the submission. Good security for the whole installation is not an alternative, it's a precondition for connecting anything to the internet anyway.
pepa65 3y ago
I think ssh on a non-standard port is a no-brainer, have been doing this for years. But at some point I got too many log entries of failed ssh access attempts. So I looked into fail2ban, but found the system too involved, and I don't like python for this kind of thing to start with. Then I started looking for alternatives, because I figured you only need something very simple, that checks certain log files and then instructs the kernel through netfilter to drop traffic from all infracting IPs. I like single-binary applications, especially for things installed outside the normal package manager. Found a skeleton of a golang app that claimed to work (it didn't), but I managed to rework it, and it serves my use cases: https://github.com/pepa65/fail2drop https://github.com/pepa65/fail2drop
ofrzeta 3y ago
> you will have to google "rsync ssh non standard port" every time you want to use rsync. I have not the best memory but after using it several times I managed to remember "rsync -e 'ssh -p 223' ..." and "scp -P 223". It wasn't that hard really.

98 points by ementally 3y ago

First blacklistd would be better, because as written the applications knows better what to "ban" than a log-scanner. I often use a combination of: -blacklistd and fail2ban (for applications who don't support blacklistd) -pfbadhost (nearly always) -CrowdSec (just sometimes, but atm pretty happy with the free service) I use those to have clean logs, block spammers at ip level, and "marketing" crawlers...well and other stuff. And no, ssh is never on 22, i like clean logs to better find the real "interesting" stuff.

lamontcg 3y ago

I've used fail2ban and a nonstandard ssh port before, for like 12 years. Never had any of the issues this article screams at you about, didn't take any time to setup. I spent a lot more time locking down the sshd_config itself and maintaining that over time. I do agree that is more important to do. Fuck me, I guess.

egberts1 3y ago

One time I had an inadvert DDoS against my non-recursive authoritative DNS server and no way to stop the flooding of log file. Problem is DNS query can come from anywhere so latching to the top-level domain (TLD) name became a must. Fail2ban, to the rescue. https://egbert.net/blog/articles/troubleshooting-regex-in-fail2ban-during-bind9-ddos.html https://egbert.net/blog/articles/troubleshooting-regex-in-fa...

hwpky 3y ago

This post is accurate, fail2ban does suck. But that's not a reflection on the fail2ban maintainers, the engineering work that's gone into building it, its usability, bugs or indeed anything else connected to its implementation. It sucks simply because it's solving the problem in the wrong way. There is no sensible security onion, or defence in depth model for open ports in front of private systems. If the technical solution for keeping private, or limited access systems actually and effectively private on the public Internet requires open ports, it's the wrong solution. Unless we're running services designed for public and anonymous access, exposing technologies like VPN servers, SSH or any other software for private consumption to the Internet is a mistake. When we do this we're putting out invitations for abuse and pulling the burden of security responsibility onto ourselves. I'd argue however, that unless it's a full time concern for us, we're close to the project or maintainers, and able to contribute security fixes to source code, we're probably ill equipped to directly handle that responsibility and remain dependent on engaging with the surrounding security eco-system to remain secure. I'm not knocking those security eco-systems, I'm just pointing out that there's a time lag inherent on depending on best-efforts from third parties which creates windows of vulnerability, and open ports on infrastructure ALWAYS puts us on the back foot. From getting patches applied in a timely fashion to the time and convenience cost of updating ACLs each time IPs change. Even if we get the maintenance tasks right 100% of the time, we're still open to the risks of something outside of our control happening, like zero day being sprayed across open ports. The U.S. Government correctly issued an executive order mandating Federal Agencies move to adopt Zero Trust principles just six days after the Colonial Pipeline ransomware attack in 2021. Despite the marketing hype which has followed, if we take nothing else away from this motion, it should be that opening ports in our networks to access private systems using the public Internet is over. There are at least 90 projects and businesses today dedicated to building modern private access technologies which allow secure remote connections and access to private networks without opening firewall ports. Many are commercial options serving businesses, but there are also lots of compelling open source offerings for non-commercial use too. There's a directory of vendors and technologies here https://zerotrustnetworkaccess.info/ https://zerotrustnetworkaccess.info/ which attempts to dispense with some the Zero Trust marketing BS and instead focus on technical discourse, architecture and approach which some might find helpful. Disclosure; founder of one of the businesses (enclave.io) with a commercial interest in this space.

deleted 3y ago

[deleted]

aborsy 3y ago

It’s unnecessary, at least for SSH, in the first place.

theandrewbailey 3y ago

> * changing your SSH port to non-root port (>1024) means that if the SSH daemon dies, any userland process can take its' place. this is bad. If you're relying on privileged ports, you've already lost the battle.

lxgr 3y ago
This doesn't even have to happen maliciously. Even if it happens by accident, it can result in locking yourself out of a server.
closewith 3y ago
Defense in depth.
o11c 3y ago
Specifically: * your SSH client will check the host certificate. There's a reason it does that prompt for new hosts! * a userland process can no more MITM that than any random middlebox can.
pepa65 3y ago
And you can always use net.ipv4.ip_local_reserved_ports to reserve the port for privileged use. (I have never had to do this in over a decade of using a high port for ssh access.)

plz-remove-card 3y ago

On {Free,Net}BSD[1][2] you get Bl*cklistd which is a much better approach than parsing log files. [1] https://man.freebsd.org/cgi/man.cgi?query=blacklistd&sektion=8&format=html https://man.freebsd.org/cgi/man.cgi?query=blacklistd&sektion... [2] https://man.netbsd.org/blocklistd.8 https://man.netbsd.org/blocklistd.8

loeg 3y ago
The primary author died in 2019. Is it still being worked on? I agree it seems like a superior approach.
sedatk 3y ago
Isn't that specifically DoS oriented? Doesn't seem to cover fail2ban's use case.
- BSDobelix 3y ago
  The difference is blacklistd is just a api and a database for telling the firewall to block ip's. The decision to inform blacklistd is on the application side, and not a scanner that checks the log's produced by that application. And yes, DoS is a very good example, if your application writes so much log's (during a attack) and fail2ban has to analyze them during that, then machine could very well DoS itself, the blacklistd approach is much simpler and straightforward BUT (and that's a big one) the applications has to support it.
  - sedatk 3y ago
    Thanks!
- plz-remove-card 3y ago
  No, once an IP had failed authentication a configured number of times, the IP is blocked (for a configurable amount of time.)
  - sedatk 3y ago

solarpunk 3y ago

hell yeah, brother

ufmace 3y ago

I agree with this viewpoint. IMO, a lot of these types of threads fill up fast with gimmicky things that sound good at first but do nothing to increase security against realistic threats, plus are at least inconvenient to set up and maintain, if not prone to misfiring and blocking you. The real advice for setting up new servers IMO is, SSH pubkey auth is perfectly adequate security. Set it up normally on the default port logging in as root, disable password login, and don't worry about the noisy failed bot logins. This is a lot more convenient for automated setup and maintenance and has no practical loss of security.

sedatk 3y ago
If you shouldn't worry about noisy failed logins, what's the purpose of monitoring login logs? Do you mean to stop monitoring them altogether?
- ufmace 3y ago
  Yes. I don't think there's anything to be gained from monitoring login logs. The only real use is after you recognize some sort of breach to try and help determine exactly what did or didn't happen.

kseifried 3y ago

Fail2ban could be potentially useful in an era of passwords, especially when systems allowed anyone to login remotely, and you only had the one server because servers were very expensive. However, the moment the industry went to SSH Keys en masse, and got away from passwords, fail2ban stop serving any real purpose other than to make people feel like they had done something to improve security. Which it didn’t really, especially if you enforced key usage only for logins. Literally, the only argument you can make is that fail2ban might reduce the number of log entries.

zimpenfish 3y ago
> the moment the industry went to SSH Keys en masse [...] fail2ban stop serving any real purpose That assumes people only use it for sshd, no? Which isn't the case [anecdatum] for me, at least - I use it for ssh, http (several servers), imap, smtp, mqtt, etc. Make a request for phpMyAdmin pages on my server? 10 day timeout for you, sir. IMAPS connection with incorrect username? 10 days in the bin. etc. Yeah, it reduces log entries but it also reduces unwanted resource consumption and keeps my servers working instead of chugging under non-useful load.
seized 3y ago
Fail2ban can parse other logs and taken other actions. I have it on my NVR VM, reading the logs from the old Unifi NVR software. Three failed attempts and Fail2ban runs a script that does an API call to my OpnSense firewall to ban that IP from every port. Forever. Haproxy also monitors the HTTP response codes and drops the TCP connection, allowing the block to happen once the bad IP tries to reconnect. So it's not just for password based SSH.

kuon 3y ago

While I agree with most of the points, I think the most important one is: your time is precious. Spending hours setting up things is are really high price to pay for something that might never be of use.

GuB-42 3y ago
Setting up fail2ban is essentially just "apt install fail2ban", it already comes with sensible defaults. That's on Debian, I think it is the same on most distros. For me, it is less of a hassle than looking at logs flooded with failed connection attempts.
- justsomehnguy 3y ago
  > it already comes with sensible defaults Just two days ago I found out the sensible defaults doesn't include automagically enabling even the basic jails. The machine run for a year (or more) with enabled fail2ban which... did nothing. Not a Debian machine and it's entirely on me, but fail2ban setup is not "just apt install". You need to be at least somewhat familiar with it's lingo and the minimals on working with it.
- zamadatix 3y ago
  Genuine question: Why not just tell the daemon to stop logging failed attempts? I personally have a hard time reconciling local username+password authentication and the need to audit failed attempts in the same use case but that may just be because I haven't run across the right use case.
  - justsomehnguy 3y ago
    If someone would fail to properly code something you can find your basic auth is slow to a crawl or even not responding but you would have no idea what is going on. [0] ie Morris worm
  - GuB-42 3y ago
    The simplest reason is that installing fail2ban is just as easy. Also, I sometimes want to know about failed attempts, mostly for legitimate attempts that failed and shouldn't have. As in: "why couldn't bob connect the the server?", and failed attempts would just show that he used the wrong password, or at least, that it is a password related problem.

throwaway892238 3y ago

I've stopped trying to convince people against using things like Fail2ban. As long as it's not a company or government where security is important, let them waste their time.

masfoobar 3y ago
I would be interested on hearing your reasons for this.

michaelt 3y ago

> at worst, fail2ban: causes you massive inconvinience or total lockout [...] it'll probably block you out. I agree that fail2ban is useless once you've disabled password login, which you should have done. But is anyone really getting locked out of their own systems by fail2ban unless they're doing something pretty weird? I thought it gave you about 5 password attempts, and then blocked you for 10 minutes - doesn't seem that draconian to me.

kleiba 3y ago
In one of my previous jobs, it blocked me out after 3 attempts indefinitely - that is, until I got a hold of the sysadmin and asked him to unblock me. It was a PITA, as you can imagine. And I don't understand why any sysadmin would want to put themselves in a position to get bothered regularly by people who tend to mistype their passwords like me.
Scaevolus 3y ago
The default configuration of fail2ban on Debian considers every offered key as a login attempt, so you can be immediately banned for merely connecting with too many keys in your ~/.ssh.
baz00 3y ago
Never been locked out when I used to use it. But it got replaced with a default firewall rule to only allow SSH from my address in the end. Didn’t need it at all then!

zamadatix 3y ago

In general I agree with the recommendations here. Though it may be slightly over prescriptive in what security approaches universally make sense (e.g. privileged ports) it carries a very valid conclusion about the applicability of Fail2Ban. The only time I almost used Fail2Ban was with a personal "guaranteed to work from wherever and whatever I'm on" jumphost which ran SSH on 22 and 443. 443 because, like the post says, outbound filter compatibility (Fun note: to listen on multiple ports simultaneously you can literally just include multiple "Port xyz" lines in your sshd_config). Ultimately I decided for this use case it made more sense to just let bad attempts to fail silently by changing the logging config. If I was ever in a situation I felt I needed the failed attempts logged I'm not sure I could simultaneously justify username+password login on a public host.

NateEag 3y ago

The author makes a number of good points. In fail2ban's defense, it can be used for a lot more than just blocking IP addresses based on SSH failures. At one point in the past I adapted an existing fail2ban setup to ban IP addresses that were running malicious vulnerability scans against a large legacy PHP webapp, using some dumb heuristics that worked and were easy to put in place. Would it have stopped sufficiently motivated expert attackers? Of course not. Did it reduce the load on the legacy PHP app and improve the signal to noise ratio in our webserver logs, while reducing the chance a brute-force attacker would find some hole lurking in a long-forgotten corner of the app's unfortunately-large attack surface? Yup. So, I'd argue it can be useful, even if naive SSH blocking may not offer much improvement.

Retr0id 3y ago
I think the log SnR point is a great one. Debugging aside, if you're able to filter out the "obviously malicious" traffic, any more sophisticated attacks will be easier to spot (or at the very least, analyse after-the-fact).

rigid 3y ago

fail2ban is fine for effectively slowing brute-force attacks for services that can't by themselves or where significant complexity would be needed. it's easy to setup and can stop multi stage attacks that generate quirky log output. EDIT: fail2ban vulnerabilities? what is this guy talking about?

marcus0x62 3y ago
> EDIT: fail2ban vulnerabilities? what is this guy talking about? There have been several: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=fail2ban https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=fail2ban. Mostly DoS. At least one RCE, admittedly with a non-default fail2ban config and a somewhat unlikely attack chain: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-32749 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3274...
- rigid 3y ago
  > At least one RCE, admittedly with a non-default fail2ban config and a somewhat unlikely attack chain: you had to manipulate answers from a whois server. Sure, it increases attack surface (like any additional piece of code) but argueing to better let an IP hammer your mailserver with infinite stuffed credentials just to avoid possible bugs, is questionable at least. fail2ban's CVE track record is quite good in comparision, if you take that as metric. It might be even more secure than using multiple different rate-limiting implementations that were written in C.
  - marcus0x62 3y ago
    Yes, as I said, the attack chain was unlikely to be used it practice.

WesolyKubeczek 3y ago

I disagree with just one point. Non-standard ports. They work wonders against lots and lots of traffic from Chinese bots which are too stupid to see that password logins are disabled but there's a fucking lot of them, they just won't stop knocking, and it does take cycles to fend them off in any way. You stop listening on port 22, and 99% of that shit traffic just goes away. I know, it's not by the book, and that well ackchyually I should yak yak this and yada yada that. But it's so fucking cheap and simple to do, and it goes brrr, so I just don't care.

WesolyKubeczek 3y ago
Case in point: > changing your SSH port to non-root port (>1024) means that if the SSH daemon dies, any userland process can take its' place. this is bad. Man, if you have sshd that likes to die a lot, you've got bigger problems already. If you have userland processes that can take your sshd port over maliciously, you're already fucked and should really reprioritize your worries. > changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working. ~/.ssh/config goes brrrrrr > you will very likely waste hours of your life. Only if I need to dictate the contents of that file to my iPhone, otherwise it's seconds. > you will have to google "rsync ssh non standard port" every time you want to use rsync rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr. Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin? > you will have to remember scp flags. this is also bad. probably worse. scp from OpenSSH honours OpenSSH configs, so it goes brrrr. > you will have to tell the port to anyone else who tries legitimately accessing your system. "Here's a fragment that you copy and paste into your ~/.ssh/config. Remember to do a chmod 0600 on it if the file is not there already." My god, is it me getting old, but when I was young it was customary to know the basics of your tools.
- lxgr 3y ago
  > userland processes that can take your sshd port over maliciously, It doesn't need to happen maliciously. The result is the same: You might be locked out permanently. > if you have sshd that likes to die a lot, you've got bigger problems already It doesn't have to die, it can also just happen during startup. Let's say you initially configure port 2222, restart sshd, everything works well, so you remove port 22, restart one more time, and stop thinking about it. Then, on your next reboot, it turns out that port 2222 is actually the preferred port of some other automatically starting service that just happened to not be running during your experiment, but which is now racing sshd for port 2222: That's bad. > rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr. And just as many applications including an SSH client won't pick up the config from .ssh/config (for example because they can't read that directory because they're sandboxed). > Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin? SSH is used by all kinds of people, not all of them condescending sysadmins.

out-of-ideas 3y ago

close the port open if you dont want ssh to be used. i think i read once the rsync.net dude uses port-knocking and maybe f2b? what was the other note from rachaelbythebay, mac os updates always revert changes sshd settings or something like that; maybe they stopped? i do hate the idea of ip whitelisting; IP's should be more dynamic and less able to fingerprint an individual - but that is opening another can of wurms. Generally never trust ANY ip; dont use whitelists, and the blacklists should never be "forever". if your app is secured by a keypair like openssh then thats the route to take. honestly though fail2ban is great in that it does silence some noise in the logs - there will still be an unbareable amount - but at least some less; but still regular log maint is required. Also the groups Ive seen never recommend f2b for use to fight against a DoS attack; not even for "cleaning up logs" - i dont see where that point was created other than from this page

kemotep 3y ago
I wouldn’t call it a whitelist necessarily but there’s no reason why ssh has to be open to the world if it is possible to use a network overlay or vpn. If ssh is only allowed on the management wireguard network, not much can beat that other than a host compromise which, additional checks to verify that a host can be trusted at that moment could potentially help stop unauthorized access.
rsync 3y ago
I do, indeed, highly value port knocking and I use it in my personal infrastructure. It is simple and elegant and we're coming up on ~20 years of deployment in various places with zero hangs or crashes or unexpected interactions. While nobody should rely solely on port knocking, it is a wonderful layer added to a defense-in-depth strategy. Which is to say: lock down everything the way you always do, then put the port behind a knock.
- out-of-ideas 3y ago
  I actually want to do this (one day i'll get to nerd-out again) but then tie it to a totp setup so the knocking-algorithm changes ever so slightly; could even utilize different ports

unixhero 3y ago

I use fail2ban. It is a mild security control which I know can fail. I install it, use its defaults and get on with my life. At times, for bastion hosts, I use crowdsec instead. I could not capture why the author was so bitter. I guess I would be the same if I was supporting some product, and its users were getting problems because of fail2ban - and I had to support it for them.

dig1 3y ago

I'm not fond of this writing style without concrete substance; it feels immature. Yes, the author may have tried to be funny, but I don't see a compelling reason to stop using fail2ban. Let's be clear here - fail2ban will not save you if you have a poorly configured SSH server or widely open Nginx/Postfix services, but it will offload traffic blocking to iptables (or whatever firewall you are using). This way, the mentioned services will not have to waste resources just to cancel some connections. Will it save it against large DDoS? Surely not, but it will stop mild DDoS without problems, which it is designed for. We can argue that iptables/nftables can do a fail2ban job with rate limiting and other techniques many people have never heard of, but if you are at this level, you already know your craft, and I'm pretty sure you considered, tried or used fail2ban before.

klaushardt 3y ago

* changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working. you will very likely waste hours of your life. you will have to google "rsync ssh non standard port" every time you want to use rsync. you will have to remember scp flags. this is also bad. probably worse. --- I dont need to remember non standard ports if i just use my .ssh/config. Host myhost.tld Hostname 127.0.0.1 Port 333 User MyUser IdentityFile ~/.ssh/myhost.tld

lxgr 3y ago
This works fine until you find yourself in a shitty public Wi-Fi that blocks outgoing traffic to "non-essential ports". But then again, SSH is often not considered essential and you need a VPN or other tunnel on TCP/443 anyway...
oliwarner 3y ago
Right! It's dead simple. Normally I might put a title like this down to opinion but here, the author suffers not knowing enough about the subject to justify having an opinion, let alone publish one. Fail2ban is a stupidly easy way to block lazy hacking scripts. It's easy to extend to handle simple honeypot services. It's no replacement for real security but it makes focussing easier.
theatomheart 3y ago
THIS. I use non standard port for SSH by default and my single SSH config file manages this for me with no extra effort. Seems like a common sense behavior to me.
thelittleone 3y ago
Another solution is tailscale. Disable all admin ports (including ssh) on public IP.
bshipp 3y ago
I was brand new to managing an Ubuntu Hetzner server and the moment I saw how many port 22 scans the server received i decided to try changing the port number, followed by key-only passwordless logins. My logs immediately shrank in size. I have never once had an issue having moved to non standard ports and, moreso, feel almost naked logging into port 22. I know security through obscurity is not an answer, but judging by the reduction in port scanning i've seen after moving as many standard ports as possible to new addresses above 20000 I have to believe its a reasonable first step. How many script kiddies are scanning all 65500 ports for each IP address?

devaiops9001 3y ago

The approach my team takes is we put Wireguard gateways running on OpenBSD in front of network services. We go a step further and very carefully select what chips are used for the network interfaces and also attach a hardware random number generator to add to entropy. Internal users' machines have these tunnels transparently loaded and running, configured by our internal GitOps-driven config management (SaltStack). Has been pretty comfy so far.

ahoka 3y ago
Kinda sounds like pure security theater, TBH. I don’t mean the jump hosts, but all the other stuff.

kazinator 3y ago

FreePBX/Asterisk comes with a default fail2ban setup that will lock out your SIP connections (from phones) while you're just fiddling to get it working, adding to the frustration.

kazinator 3y ago

I use something that I made almost exactly ten years ago: txrban. https://www.kylheku.com/cgit/txrban/ https://www.kylheku.com/cgit/txrban/ It's way simpler. There is a tiny core to it, and a couple of modules specific to Exim, Apache, SSH. These just run as daemons on the machine. I tried using fail2ban at that time, but it wasn't at all what I expected. It was way overcomplicated for what it achieves, and poorly documented. Just getting it running at all was fiddly, never mind the custom behaviors I had in mind. I should review the Apache module; some of it is quite out of date, due to the changing landscape. Also, that webserver it is used with is now behind a reverse proxy, which only receives connections for specific applications.

pepa65 3y ago
What is 'txr'? The script requires it to run the modules.
- kazinator 3y ago
  That's this: https://www.kylheku.com/cgit/txr/ https://www.kylheku.com/cgit/txr/ Home paged here: https://nongnu.org/txr https://nongnu.org/txr

tzs 3y ago

A particularly fun combination is Fail2Ban on a mail server and Apple's Mail desktop email client when you need to change your password. We've got to change passwords every 90 days at work, and probably 90% of the times I try to do this I end up getting banned by Fail2Ban. The problem apparently is that Apple Mail is pretty aggressive at retrying if a login fails, and so if it tries to connect before you have finished changing the password in the client it can try enough times in quick succession to trigger Fail2Ban. There are at least 3 places you can change your password(s) for Apple Mail. • Invoke "Mail/Settings..." in Mail to open the settings dialog. Select the "Accounts" tab. Then for the account you want to change go to the "Server Settings" pane. That has password fields for your incoming mail server and your outgoing mail server. • Invoke "/System Settings...", then go to the "Internet Accounts" settings. (You can also get there from "Mail/Account..." in Mail). Select you mail account, click the "Detail..." button, and you'll get a dialog that has a password field. Note it only has one password field. I'm not sure what you are supposed to do if you have different passwords for you incoming and outgoing mail servers. • Open "Keychain Access", and search for your incoming mail server. Look for an entry whose kind is "application password" named com.apple.account.IMAP.password. (I assume that IMAP would be POP3 if you use a POP3 server). Open it, verify that the Account field is you@that_server. Then you can click "Show password" to reveal the current password. Change that to the new password and save. Do the same for your outgoing mail server, looking for com.apple.account.SMTP.password. My current procedure is to first disable my work mail account in "Mail/Settings...", then change the password on the server, then update the client side passwords via Keychain access, and then via "Mail/Settings...", then enable my work mail account. That seems to usually work, although once it didn't until I also changed it in "Internet Accounts" in the system settings. I also had a time where I just could not get the new password to take until I rebooted. Apple Calendar is also a pain with password changes, but we don't have Fail2Ban on our calendar server (it is not on a public facing network so there would be no point) so it is not as annoying as Mail. For Calendar what I've found to mostly work is: 1. In "Calendar/Settings..." in the settings for my work account in the Accounts tab turn off all delegates. Then disable my work account. The quit Calendar. On MacOS prior to 13 find and kill the CalendarAgent and CalNCService processes. On 13 they went away. 2. Change the password on the server. 3. Change the password in "Keychain Access". I find the entry by searching for name@calendar_server. The right entry is the "application password" named "com.apple.account.CalDAV.password". 4. Then change the password in "Internet Accounts". As with Mail, Calendar provides convenient access to this via "Calendar/Accounts...". 5. Open Calendar, reenable the account, and turn back on delegates. All of the above comes from trial and error and I have no idea how much of it is actually necessary. I haven't had the patience to do the time consuming experimentation it would take to figure out the actual simplest way to change these passwords reliably and without hitting the server with the old password during the changing.

macNchz 3y ago

> what i'm saying is, if you're fucked, you're fucked. fail2ban will not help I see this attitude frequently in online discussions around {security thing}, but it really doesn't match my understanding of reality. It's not about making any one part of the system perfectly secure, but instead about having all of the pieces of it contribute to a whole. Many real-life stories of caught/thwarted attacks include the threat actor successfully breaching/moving through multiple layers of a system but ultimately getting stuck on something relatively simple. Defense in depth is not just a catchphrase. In evaluating whether something like this is genuinely useful, it can make sense to look at how difficult it is to operate, and how much it might reduce the probability of a successful attack. In my experience, fail2ban has been extremely easy to install and operate, and I think it brings at least some meaningful impact (particularly if applied to other things than SSH). At the very least it can cause someone making a focused attack to have work more slowly and take alternative approaches–"noisy" behavior that may itself trigger an alarm and get a human involved.

ddtaylor 3y ago
I agree, defense by depth.
djoletina 3y ago
I agree completely. Security is not black and white, same goes for agents that are malicious, if you’re protected from script kiddies by changing your ssh port and adding fail2van then it’s a net positive any way you look at it. It’s important to be aware of course that you’ll need to go a step further if you want to protect from more competent attackers, but what percentage of us will be a target of one of those.
rstuart4133 3y ago
You're being far too polite. He's just plain wrong on this point: > do not change your standard ports. do not use fail2ban. That used to be my attitude too. But reality intruded. Example: I ran a SIP Server for a business. SIP Servers are very attractive targets because if you get past the security, you can make then call paid numbers and make real money. Knowing that, I used huge security tokens. No one got past them. But - it didn't stop them from trying. 100's of times per second at times. The CPU load created by processing the login attempts was so great the SIP server started dropping calls. As it happens, asking the kernel to drop packets from miscreant IP's takes far, far less CPU. Fail2ban to the rescue. Example: Move on a few years, and now spammers costs have dropped so much it is worth their while to target IMAP, POP and SMTP servers. The story is the same: they never got through, but despite that their attempts increased to the point the server became overloaded. I can't listen on a different port. Fail2ban to the rescue. Example: Move on to the last couple of years. I move my personal stuff to a VPS. To save costs it's very memory constrained. A problem soon arises. After a few weeks of trouble free operation it keeps slowing down, and sometimes becoming unresponsive. Turns out it's running out of memory; the OOM killer is running amok. That's odd, as there should be almost no load on this thing. Well, apart from ssh login attempts it seems. Turns out there is one of them every few seconds. They have no hope of succeeding of course as I only allow pubic key auth, but the load it creates is a problem. Fail2ban to the rescue. I don't see anything wrong with the principle, which is you block these attacks at the cheapest point possible. That happens to be the point of entry, before you spend CPU cycles processing the packet (even in TCP). Fail2ban leverages that by using the kernel's networking stack. It's forced to glean the information on who to ban from log files, which is messy, error prone and fragile. I don't like that either, but I'll take what I can get.

bioneuralnet 3y ago

If you can't prevent some users from having passwords, it's a decent, low-effort way to discourage intrusion attempts. One place it DOESN'T necessarily work great is in k8s. A few years ago I had to migrate an ancient VM running OpenSSH (for SFTP only) to our k8s cluster. 99% of the users had passwords instead of keys. Easy enough to run sshd in a container with a mounted volume, but I wanted something to recognize and block repeated bad login attempts (the old VM had no such protections). sshd doesn't do it natively, fail2ban wasn't a good option since you generally don't want a pod messing with a k8 node's iptables. What I ended up with was a Ruby script with 3 threads: One to monitor the auth log for failed logins, a second to lock accounts with repeated failures during a given window, and a third to unlock them after some period (and in the darkness bind them). Definitely some caveats in there, but it was an improvement over the status quo.

theginger 3y ago

Always use multiple layers of defence. A sturdy 6ft fence and a locked wooden front door can be more secure than a 50m impenetrable stone wall, if it turns out that there is no such thing as an impenetrable stone wall. Brute force protection is completely unnecessary for openssh when you've disabled password authentication because it's impossible[really expensive] to brute force. Until you are impacted by an authentication vulnerability which still requires a valid user, then you are brute forcing a user away from being totally compromised. All of a sudden doing all the "unnecessary" things, like brute force protection, disabling root logins and not using default accounts make the difference between you being at the front or back of the line to get compromised in the race to get patched. If you are one of those who believes a vulnerability like that is extremely unlikely rather than inevitable, then consider this. An ssh key gets leaked, the system it has access to is known / discoverable, the username is not. Brute force protection is now the main difference between being highly at risk and still reasonable safe. It's a contrived scenario, but hardly an outrageous one.

o11c 3y ago
A sturdy 6-foot fence keeps your yard looking nice, even if you still have to lock the front door.

shishcat 3y ago

I remember having a friend with an extemely sluggish server. Turns out the reason behind this sluggishness was that he set a permanent fail2ban bantime, which added thousands of records to iptabes; any packet, even not SSH, the server had to check the source IP against thousands of records. After purging the rules and switching the bantime the server was usable again.

pepa65 3y ago
This should happen in-kernel and is lightning-fast, so should not be an issue. Maybe a pre-netfilter box..?
- 72deluxe 3y ago
  Big iptables lists are indeed incredibly slow; use ipset for large lists instead.

j45 3y ago

Fail2ban is better than nothing at all. Fail2ban has never seemed to be the sophisticated or final solution, just a bare minimum. Inconvenience to users is definitely a form of the security vs. Convenience debate.

pierat 3y ago

This article sucks. Fail2Ban can prevent password spraying from working, since an eventual good password will still be blocked if done within the fail2ban block period. And those block periods aren't publicized. And given appropriate configuration, is another layer of security a potential threat has to deal with.

mkasberg 3y ago

On a related note, the number of HTTP 404 errors I see to /wp-login.php on my static website is ridiculous. (But absolutely not security risk.)

pepa65 3y ago
That's what you get running a webserver. This is what should happen (but you could choose not to log those...)
- NateEag 3y ago
  You could also temp-ban those IPs for a few hours and save yourself some CPU and bandwidth. It may not seem like much, but I spent a year or two adminning a >1 million reqs / day cluster, and it does add up. Auto-banning this type of obvious offender (with fail2ban, as it happens) did result in a noticeable improvement for us.
  - pepa65 3y ago
    You could indeed (temp-)ban them based on logs. Of course it is a huge improvement, if you catch offending IPs early, they never hit any app.

LAC-Tech 3y ago

Having full logs sucks more.

patja 3y ago

Every attack of significance I see these days uses rotating IP addresses.

pepa65 3y ago
If the reuse of a single IP address is not prevalent, these are hard to act against. Then you need a different kind of mitigation.
- patja 3y ago
  They usually follow a predictable pattern, like trying to hit every (almost entirely invalid) URL starting /product/1 /product/2 /product/3 and on up through the millions. They also will rotate the User Agent but neglect realistic/authentic values for other request headers.

lostmsu 3y ago

Would you call fail2ban essentially an application firewall?

pepa65 3y ago
It adds rules to in-kernel netfilter tables. On Linux at least, there is really no such thing as a firewall app, there are just different ways to add and manage rules. The only 'approved' default app is nft (nftables), but any app can act on the in-kernel tables.

mgkimsal 3y ago

> you will have to remember scp flags. this is also bad. probably worse Worse? How? Hrm... If I don't use commands for a while, yes, I may have to remember them. If that's the tradeoff for one extra layer providing even a modicum of security... why is this a bad thing? We have things like bash scripts... where you can just store the common flags you use for common things. I wonder how often the author port tunnels over ssh? I usually have to go search for the syntax for ... 30 seconds. Then I have it. Or it might be in my history already. "You might have to remember something" seems about the weakest argument against changing ssh ports I've ever read.

usr1106 3y ago

I have always changed my ssh port and I don't see any convincing argument in this rant not continuing to do so. However, scanners have changed drastically during the last 5-10 years. I remember when you saw close to none login attempts when you used a port between 15000 and 20000. Nowdays there are 100s every day. I guess on 22 it's much worse, but I have not used 22 just for the sake of comparing. For fail2ban it's the same: I know it's not a great solution (lightweight code base, better security gains and lower risk to affect legitimate users would be great), but there is no convincing argument and no better alternative in the submission. Good security for the whole installation is not an alternative, it's a precondition for connecting anything to the internet anyway.

pepa65 3y ago

I think ssh on a non-standard port is a no-brainer, have been doing this for years. But at some point I got too many log entries of failed ssh access attempts. So I looked into fail2ban, but found the system too involved, and I don't like python for this kind of thing to start with. Then I started looking for alternatives, because I figured you only need something very simple, that checks certain log files and then instructs the kernel through netfilter to drop traffic from all infracting IPs. I like single-binary applications, especially for things installed outside the normal package manager. Found a skeleton of a golang app that claimed to work (it didn't), but I managed to rework it, and it serves my use cases: https://github.com/pepa65/fail2drop https://github.com/pepa65/fail2drop

ofrzeta 3y ago

> you will have to google "rsync ssh non standard port" every time you want to use rsync. I have not the best memory but after using it several times I managed to remember "rsync -e 'ssh -p 223' ..." and "scp -P 223". It wasn't that hard really.