4 ms·

For anybody considering fail2ban here, install keys and disable passwords via SSH (don't remove the password from root as most naïve administrators suggest, sin

For anybody considering fail2ban here, install keys and disable passwords via SSH (don't remove the password from root as most naïve administrators suggest, since most hosting shops will give you tty1 access; just set PermitRootLogin without-password and PasswordAuthentication no. You keep a back way into the system in the unlikely event that you lose all of your keys and retain physical access to the machine). For an additional layer, you can also remove all terminals except tty1 from securetty. Once you're in that fairly secure state, fail2ban is completely pointless and serves only to reduce noise in your log. Which you should stop caring about. I'm serious. fail2ban is a complete waste of resources and will peg a core tailing your SSH log because it's a resource-intensive piece of software that serves no valuable purpose. If you're worried about the resources of writing a log, redirect the log off the box and prevent your local logger from writing to disk -- which you should be doing for security reasons anyway, as discussed in another comment elsewhere in this thread (how do you trust a box's logs once compromised?). While you're at it, don't move SSH to a separate port. There are interactivity QoS defaults in most networks that expect SSH to be on 22, and the clever bots will find your "cleverly hidden" daemon anyway. And wait until I tell you that in the event of your "cleverly hidden" port 2222 SSH daemon crashing, one of your local users can put up a trojan SSH daemon in its place without your permission (which is why SSH is on 22 in the first place). Just deal with the log noise. We all do on edge boxes exposed to the Internet.

tomp 14y ago
Is there a way to rate-limit ssh password-based logins to your system (i.e. 4 per minute or so)? I mean, across all users, all IPs.
- ohboyacomment 14y ago
  Locking yourself out of the machine lies that way. Strongly, strongly advise against. But, yes. There's an iptables module called ipt_recent[1] that will do what you're after. It's been built-in for ages and you don't need to install it. You can't use it by itself, though. You will have to design a full iptables ruleset. [1]: http://snowman.net/projects/ipt_recent/ http://snowman.net/projects/ipt_recent/
- jrabone 14y ago
  Simple 3 attempts per minute rate-limiter: echo " (eth0 - SSH rate check)" iptables -N SSH_CHECK iptables -A SSH_CHECK -m recent --set --name SSH iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 3 --name SSH -j DROP
  - ohboyacomment 14y ago
    Well, that's omitting the rules required to get an SSH packet to that chain. I also hope you've already ACCEPTed ESTABLISHED,RELATED so we're only considering these rules on new connections, otherwise you've entirely locked out SSH after the third packet. Assuming that's true, now guess what happens when you do this during a crisis: for i in backup-file-{0..999}; do scp $i that-machine:/var/db/ done Unless you're muxing in your local SSH config, you will cause the firewall to intervene on you during the absolute worst time, and you will have no idea why. Promise. I really wish we could reverse the trend of fiddling with iptables when it comes to SSH. Everybody does it and nobody thinks about the ramifications.
    - jrabone 14y ago
      OP didn't ask for a complete firewall script, and besides they should be writing their own rather than copy-pasting the whole thing. Of course there's more to it. The global limit on scp is fine; this is an internet-facing server with minimum user data for which I have a cold spare - there's nothing to back up. You're coming over like you're the only person in the world who knows what they're doing; might want to rethink that.
    - tomp 14y ago
      You're right, I didn't think it through... So, is there any way (in your opinion) how one can safely retain password access for root/one user, so it's still possible to control the server, even if you don't have your keys?
    - asb 14y ago
      You are totally right, and it seems non-trivial to work around this. One approach would be to whitelist an IP upon successful logon. Perhaps by modifying /etc/ssh/sshrc to invoke iptables (though this is not executed as root...). An alternative would be to use pam_exec and have that invoke iptables to whitelist the IP (assuming it can find out the IP). But I don't think PAM is even used if SSH private key authentication is used...
asb 14y ago
Or just stick a few iptables rules to slow down brute-force attempts from the same IP http://www.digitalsanctuary.com/tech-blog/debian/using-iptables-to-prevent-ssh-brute-force-attacks.html http://www.digitalsanctuary.com/tech-blog/debian/using-iptab...
- ohboyacomment 14y ago
  And then pound your desk in frustration when Postgres goes down on the box and your clever little firewall locks you out on your third connection attempt. Leaving your entire site down for the 60 minutes your firewall takes to release the lock on your IP address, or leaving you driving to a friend's house, or frantically scrambling for a jump box to use. Every HOWTO I see that people copy and paste considers every SSH connection as ammunition for the ruleset. So if you connect to a machine however many times -- and, scping a couple files opens new connections unless you have a clever muxing config on the client side -- boom, the firewall will intervene based on most of those copy-and-paste blog entries. Smart admin edict: do not mess with my ability to SSH in under any circumstances. I've been in this situation, if you can't tell from the specificity. ("Why is the edge not responding to SSH any more? Oh, crap, we're flagged by the firewall and have to wait ... I forget how long.")
moe 14y ago
The part about not moving SSH is wrong. It's almost always a good idea to move it to a high-port; none of your arguments hold water (QoS is irrelevant to most, bots do not scan !=22, and if an attacker can launch daemons on your server then you've already lost anyway).
- ohboyacomment 14y ago
  Well, since you outright said I was wrong... > It's almost always a good idea to change ssh to a high-port [citation needed] > QoS is irrelevant to most You sure? QoS is even in consumer routers these days. I'm speaking from experience, and have shown a 20% drop in scp performance by moving away from 22 on a consumer Netgear. This point really isn't worth caring about, though, but... > bots do not scan !=22 I run a honeypot on 2222 with a well-known address. It's been scanned by 47 bots (out of 1,449 against 22) in the last 48 hours. They're out there, because it's not exactly a secret that administrators move SSH. APTs will scan every single port looking for OpenSSH, because it announces itself in the opening conversation. Granted, they are a smaller figure, but your assertion does not hold water. You are also building upon obscurity. Ports are just endpoints. If you shuffle five feet to the left, you're still very likely standing in the same room. Your system is secure with SSH on 22. Period. > if an attacker can launch daemons on your server then you've already lost anyway You didn't read what I wrote carefully. I did not say attacker. Ports under 1024 are reserved for root. Unless you are UID 0, you cannot bind to a port below 1024. That's why SSH is, by default, on 22. That is a service that only root should be able to start. If you move it to 2222, say, inside a company of a bunch of employees, one day I might get clever after your sshd crashes or I somehow coerce it to crash (and there are ways). Now there is nothing listening on 2222, but I have a physical console, and I launch my own trojan sshd on 2222 (totally legal, because I can bind to 2222) and capture passwords from everyone that connects. Now I have passwords from all of my fellow employees, and I can start trying other systems. That party is not an attacker. He is an employee that you gave an account. And do you have monitoring checking that the sshd listening on 2222 is running as root? Didn't think so. Ports above 1024 subvert the security model of Unix, and should not be used for a system-critical service. Ever. If you are going to move it even against this advice, do not go above 1023. That being said, I'd like you to provide one good security reason to move SSH to a separate port. To be honest, this whole "move SSH to a high port" is a complete and utter lie started by someone and parroted by every administrator who heard it from another guy, and it is rooted in absolutely nothing. It's not my job, as you've demanded, to justify leaving SSH at 22. It's your job to justify moving it.

tomp 14y ago

Is there a way to rate-limit ssh password-based logins to your system (i.e. 4 per minute or so)? I mean, across all users, all IPs.

ohboyacomment 14y ago
Locking yourself out of the machine lies that way. Strongly, strongly advise against. But, yes. There's an iptables module called ipt_recent[1] that will do what you're after. It's been built-in for ages and you don't need to install it. You can't use it by itself, though. You will have to design a full iptables ruleset. [1]: http://snowman.net/projects/ipt_recent/ http://snowman.net/projects/ipt_recent/
jrabone 14y ago
Simple 3 attempts per minute rate-limiter: echo " (eth0 - SSH rate check)" iptables -N SSH_CHECK iptables -A SSH_CHECK -m recent --set --name SSH iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 3 --name SSH -j DROP
- ohboyacomment 14y ago
  Well, that's omitting the rules required to get an SSH packet to that chain. I also hope you've already ACCEPTed ESTABLISHED,RELATED so we're only considering these rules on new connections, otherwise you've entirely locked out SSH after the third packet. Assuming that's true, now guess what happens when you do this during a crisis: for i in backup-file-{0..999}; do scp $i that-machine:/var/db/ done Unless you're muxing in your local SSH config, you will cause the firewall to intervene on you during the absolute worst time, and you will have no idea why. Promise. I really wish we could reverse the trend of fiddling with iptables when it comes to SSH. Everybody does it and nobody thinks about the ramifications.
  - jrabone 14y ago
    OP didn't ask for a complete firewall script, and besides they should be writing their own rather than copy-pasting the whole thing. Of course there's more to it. The global limit on scp is fine; this is an internet-facing server with minimum user data for which I have a cold spare - there's nothing to back up. You're coming over like you're the only person in the world who knows what they're doing; might want to rethink that.
  - tomp 14y ago
    You're right, I didn't think it through... So, is there any way (in your opinion) how one can safely retain password access for root/one user, so it's still possible to control the server, even if you don't have your keys?
  - asb 14y ago
    You are totally right, and it seems non-trivial to work around this. One approach would be to whitelist an IP upon successful logon. Perhaps by modifying /etc/ssh/sshrc to invoke iptables (though this is not executed as root...). An alternative would be to use pam_exec and have that invoke iptables to whitelist the IP (assuming it can find out the IP). But I don't think PAM is even used if SSH private key authentication is used...

asb 14y ago

Or just stick a few iptables rules to slow down brute-force attempts from the same IP http://www.digitalsanctuary.com/tech-blog/debian/using-iptables-to-prevent-ssh-brute-force-attacks.html http://www.digitalsanctuary.com/tech-blog/debian/using-iptab...

ohboyacomment 14y ago
And then pound your desk in frustration when Postgres goes down on the box and your clever little firewall locks you out on your third connection attempt. Leaving your entire site down for the 60 minutes your firewall takes to release the lock on your IP address, or leaving you driving to a friend's house, or frantically scrambling for a jump box to use. Every HOWTO I see that people copy and paste considers every SSH connection as ammunition for the ruleset. So if you connect to a machine however many times -- and, scping a couple files opens new connections unless you have a clever muxing config on the client side -- boom, the firewall will intervene based on most of those copy-and-paste blog entries. Smart admin edict: do not mess with my ability to SSH in under any circumstances. I've been in this situation, if you can't tell from the specificity. ("Why is the edge not responding to SSH any more? Oh, crap, we're flagged by the firewall and have to wait ... I forget how long.")

moe 14y ago

The part about not moving SSH is wrong. It's almost always a good idea to move it to a high-port; none of your arguments hold water (QoS is irrelevant to most, bots do not scan !=22, and if an attacker can launch daemons on your server then you've already lost anyway).

ohboyacomment 14y ago
Well, since you outright said I was wrong... > It's almost always a good idea to change ssh to a high-port [citation needed] > QoS is irrelevant to most You sure? QoS is even in consumer routers these days. I'm speaking from experience, and have shown a 20% drop in scp performance by moving away from 22 on a consumer Netgear. This point really isn't worth caring about, though, but... > bots do not scan !=22 I run a honeypot on 2222 with a well-known address. It's been scanned by 47 bots (out of 1,449 against 22) in the last 48 hours. They're out there, because it's not exactly a secret that administrators move SSH. APTs will scan every single port looking for OpenSSH, because it announces itself in the opening conversation. Granted, they are a smaller figure, but your assertion does not hold water. You are also building upon obscurity. Ports are just endpoints. If you shuffle five feet to the left, you're still very likely standing in the same room. Your system is secure with SSH on 22. Period. > if an attacker can launch daemons on your server then you've already lost anyway You didn't read what I wrote carefully. I did not say attacker. Ports under 1024 are reserved for root. Unless you are UID 0, you cannot bind to a port below 1024. That's why SSH is, by default, on 22. That is a service that only root should be able to start. If you move it to 2222, say, inside a company of a bunch of employees, one day I might get clever after your sshd crashes or I somehow coerce it to crash (and there are ways). Now there is nothing listening on 2222, but I have a physical console, and I launch my own trojan sshd on 2222 (totally legal, because I can bind to 2222) and capture passwords from everyone that connects. Now I have passwords from all of my fellow employees, and I can start trying other systems. That party is not an attacker. He is an employee that you gave an account. And do you have monitoring checking that the sshd listening on 2222 is running as root? Didn't think so. Ports above 1024 subvert the security model of Unix, and should not be used for a system-critical service. Ever. If you are going to move it even against this advice, do not go above 1023. That being said, I'd like you to provide one good security reason to move SSH to a separate port. To be honest, this whole "move SSH to a high port" is a complete and utter lie started by someone and parroted by every administrator who heard it from another guy, and it is rooted in absolutely nothing. It's not my job, as you've demanded, to justify leaving SSH at 22. It's your job to justify moving it.