6 ms·

Question: I've had an SSH server open to the world for months on a nonstandard port. I've never gotten any attacks logged on fail2ban, to the point where I susp

Question: I've had an SSH server open to the world for months on a nonstandard port. I've never gotten any attacks logged on fail2ban, to the point where I suspect fail2ban is either not logging the attacks or not operating correctly... even though it logs every login I'm aware of just fine, so it really seems like there really haven't been any attacks. I find this so unlikely. Is this normal? Does no one scan high ports ever? Are there any more plausible explanations?

perpetualcrayon 9y ago
That sounds like a much better solution than I've used in the past. By simply restricting total # of connections / IP to one my attacks dropped dramatically, but not gone completely. I think it's just a matter of numbers. The IP search space is already pretty large with IPv4 on just a single port. With IPv6 it'll be more or less impossible for attackers to make any substantial progress by just searching for IP addresses to attack, let alone an IPv6 address on a non-standard port. After reading this I'm likely going to be upgrading my technique. SSH on IPv6 only (eventually shutting down IPv4 altogether, can't wait), listening on a non-standard port, limit 1 connection per IP.
- throwaway613834 9y ago
  Yeah, definitely listen on a nonstandard port. I'm not sure how to limit to 1 connection per IP but it sounds rather limiting, so I wouldn't do it... I often have multiple SSH sessions from my own IP.
  - perpetualcrayon 9y ago
    I just run a 'screen' session to open 10 bash sessions simultaneously when I want to do anything substantial. That has never felt restrictive to me.
    - throwaway613834 9y ago
      Can you see two screens simultaneously with screen though?
      - perpetualcrayon 9y ago
        I'm not sure. The need has never come up for me though.
      - blfr 9y ago
        If you want this, I would suggest tmux. Tmux by default comes with tabs (multiple terminals) and panes (multiple terminals visible at once within a tab) where you can split your screen vertically and horizontally however you like. Usually, you want one full-height terminal and two about half the height of your screen. http://www.kaltpost.de/wp-content/uploads/2013/02/tmux.png http://www.kaltpost.de/wp-content/uploads/2013/02/tmux.png
        throwaway613834 9y ago
        But these are so much more restrictive. Right now I just open two terminal windows -- they can overlap each other, I can make it so one covers every part of the other except a tiny part whose update I want to see, and make changes in these first one. I can kill programs in one without being worried about accidentally killing them in the other. I just don't see the appeal of forcing everything into a single terminal screen honestly. What's the benefit?
        merijnv 9y ago
        If your network connection drops long running jobs keep running, no setup. I do most of my work on our university cluster over SSH. I can close/shutdown my laptop at any time (or have my connection drop) and as soon as I SSH back into the server all my editors, shells, running jobs, etc. are exactly how I left them.
        throwaway613834 9y ago
  - blfr 9y ago
    Give yourself some breathing room. I limit connections to four in a three minute window. iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 4 --rttl --name SSH --mask 255.255.255.255 --rsource -j DROP
Macuyiko 9y ago
No, this is pretty expected. All the script kiddies are scanning port 22 and nothing else. Recently I made the mistake of changing a root password (I know, SSH root acc ss, bad idea) to a dictionary word. In 1 minute the system was penetrated. With a nonstandard port, bots mostly don't even bother to scan you.
- throwaway613834 9y ago
  Interesting, good to know, thanks for confirming. I guess security by obscurity is quite effective!
  - MayeulC 9y ago
    Technically, for an uninformed attacker, this is just one more level of entropy added to your password (multiply the combinations by 65635 - and then more in practice, as you have to account for the fact that this has a low probability of occurring from the viewpoint of the attacker). It just really is a part of your password (albeit one that can be guessed easily by a more dedicated attacker, just like the username).
    - throwaway613834 9y ago
      Right :)
    - marcosdumay 9y ago
      It is not plain added entropy. You can't multiply the complexity (or add the entropy) because there is a signal when the attack makes a partial guess (he gets the openssh prompt). The added entropy is a bit harder to calculate, but unless your password has 16 bits of entropy¹, it will round pretty well to zero. 1 - Why TF would you run a ssh server that accepts passwords anyway? But nearly everybody I have seen that thinks changing the port is a good form of protection also sees no problem on accepting them.
dp30 9y ago
I've been running non-standard ports for a few years and while it certainly cuts down on the non-stop brutes on 22, my servers still get smacked pretty often in 400-500 attempt blocks.
- throwaway613834 9y ago
  Oh, interesting. Are your servers targets of interest somehow? Or are your ports possibly still low numbers?
  - dp30 9y ago
    definitely not interesting, but they're on port 8022 so maybe that's a common one for attack scanners. i mean with masscan its so fast and easy to find ssh ports, i cant imagine any non-standard port is more obscure than the next. they are hosted on a low cost vps provider so maybe their ranges are common low hanging fruit.
    - throwaway613834 9y ago
      I wouldn't have revealed the exact port number here, but thanks :) yeah, I think your explanation sounds plausible.
      - firethief 9y ago
        A port number isn't a real secret, it just screens out the bulk of mass scans. There's no point trying to hide it from targeted attackers.
    - user5994461 9y ago
      The hosting providers have known IP ranges and they are scanned permanently. I wouldn't be surprised if some guys had a rule to try 8022 and the most common non standard ports.
tombrossman 9y ago
Is this server running a recent version of Ubuntu and using systemd? Also, have you verified that fail2ban actually blocks you by intentionally making bad login attempts to trigger a ban? I do this with every new machine and never assume it works until I can trigger it on demand. (test using a VPN or set a very short bantime so you don't lock yourself out) I ask because systemd may not be writing the failed login attempts to the log file fail2ban monitors, which means fail2ban may not be responding to attacks. I don't recall exactly how I resolved this, but I think I had to set "backend = systemd" in my jail.local under the [DEFAULT] section header.
- throwaway613834 9y ago
  Oh wow, thanks a ton for mentioning this. It's Ubuntu indeed but I seem to have sysvinit rather than systemd from what I can tell. I have no idea how to actually test it in a way that ensures I will trigger it though (since I don't really have other setups that I can test the same method on and ensure I'm doing it correctly). What do I have to do to trigger it? Just fail a bunch of SSH logins in a row? (I have ssh, dropbear, pam-generic, and ssh-ddos enabled.)
  - tombrossman 9y ago
    > What do I have to do to trigger it? Just fail a bunch of SSH logins in a row? Yes, that's what I do. If you have SSH login via password disabled (you should) then just temporarily move your private key(s) out of your ~/.ssh directory and try logging in a few times. "ssh user@machine && ssh user@machine && ssh user@machine" will do it if you have a ban set for three failed attempts. If it's working, your target machine should no longer respond at all, and connections will time out. If it's a web server the site should be unreachable with your web browser until after the bantime passes.
    - throwaway613834 9y ago
      Thanks, I'll give it a try :) I've gotten myself locked out before like that, but ironically from a different machine...
andimm 9y ago
Don't know if it is a too obvious answer, but have a look at the output of 'sudo lastb', it will show failed attempts quickly to verify if it is plausible and fail2ban properly setup.
- throwaway613834 9y ago
  I've never heard of the command, so it's not obvious, thanks. However, I'm not sure how to interpret this. All I see is a single line: btmp begins Fri Sep 1 06:25:05 2017 Does this just mean nothing has happened? I'm not sure what's special about that date; the setup has certainly been running for much longer than that...
  - andimm 9y ago
    That's not a bad sign then. Mine shows lots of attempts. lastb shows the last unsuccessful (bad) login attempts. it uses the file /var/log/btmp as input. the time stamp of "btmp begins .." either is the same as the creation date of the btmp file or if the btmp file was rotate of an older version i.e. /var/log/btmp.0

perpetualcrayon 9y ago

That sounds like a much better solution than I've used in the past. By simply restricting total # of connections / IP to one my attacks dropped dramatically, but not gone completely. I think it's just a matter of numbers. The IP search space is already pretty large with IPv4 on just a single port. With IPv6 it'll be more or less impossible for attackers to make any substantial progress by just searching for IP addresses to attack, let alone an IPv6 address on a non-standard port. After reading this I'm likely going to be upgrading my technique. SSH on IPv6 only (eventually shutting down IPv4 altogether, can't wait), listening on a non-standard port, limit 1 connection per IP.

throwaway613834 9y ago
Yeah, definitely listen on a nonstandard port. I'm not sure how to limit to 1 connection per IP but it sounds rather limiting, so I wouldn't do it... I often have multiple SSH sessions from my own IP.
- perpetualcrayon 9y ago
  I just run a 'screen' session to open 10 bash sessions simultaneously when I want to do anything substantial. That has never felt restrictive to me.
  - throwaway613834 9y ago
    Can you see two screens simultaneously with screen though?
    - perpetualcrayon 9y ago
      I'm not sure. The need has never come up for me though.
    - blfr 9y ago
      If you want this, I would suggest tmux. Tmux by default comes with tabs (multiple terminals) and panes (multiple terminals visible at once within a tab) where you can split your screen vertically and horizontally however you like. Usually, you want one full-height terminal and two about half the height of your screen. http://www.kaltpost.de/wp-content/uploads/2013/02/tmux.png http://www.kaltpost.de/wp-content/uploads/2013/02/tmux.png
      - throwaway613834 9y ago
        But these are so much more restrictive. Right now I just open two terminal windows -- they can overlap each other, I can make it so one covers every part of the other except a tiny part whose update I want to see, and make changes in these first one. I can kill programs in one without being worried about accidentally killing them in the other. I just don't see the appeal of forcing everything into a single terminal screen honestly. What's the benefit?
        merijnv 9y ago
        If your network connection drops long running jobs keep running, no setup. I do most of my work on our university cluster over SSH. I can close/shutdown my laptop at any time (or have my connection drop) and as soon as I SSH back into the server all my editors, shells, running jobs, etc. are exactly how I left them.
        throwaway613834 9y ago
- blfr 9y ago
  Give yourself some breathing room. I limit connections to four in a three minute window. iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 4 --rttl --name SSH --mask 255.255.255.255 --rsource -j DROP

Macuyiko 9y ago

No, this is pretty expected. All the script kiddies are scanning port 22 and nothing else. Recently I made the mistake of changing a root password (I know, SSH root acc ss, bad idea) to a dictionary word. In 1 minute the system was penetrated. With a nonstandard port, bots mostly don't even bother to scan you.

throwaway613834 9y ago
Interesting, good to know, thanks for confirming. I guess security by obscurity is quite effective!
- MayeulC 9y ago
  Technically, for an uninformed attacker, this is just one more level of entropy added to your password (multiply the combinations by 65635 - and then more in practice, as you have to account for the fact that this has a low probability of occurring from the viewpoint of the attacker). It just really is a part of your password (albeit one that can be guessed easily by a more dedicated attacker, just like the username).
  - throwaway613834 9y ago
    Right :)
  - marcosdumay 9y ago
    It is not plain added entropy. You can't multiply the complexity (or add the entropy) because there is a signal when the attack makes a partial guess (he gets the openssh prompt). The added entropy is a bit harder to calculate, but unless your password has 16 bits of entropy¹, it will round pretty well to zero. 1 - Why TF would you run a ssh server that accepts passwords anyway? But nearly everybody I have seen that thinks changing the port is a good form of protection also sees no problem on accepting them.

dp30 9y ago

I've been running non-standard ports for a few years and while it certainly cuts down on the non-stop brutes on 22, my servers still get smacked pretty often in 400-500 attempt blocks.

throwaway613834 9y ago
Oh, interesting. Are your servers targets of interest somehow? Or are your ports possibly still low numbers?
- dp30 9y ago
  definitely not interesting, but they're on port 8022 so maybe that's a common one for attack scanners. i mean with masscan its so fast and easy to find ssh ports, i cant imagine any non-standard port is more obscure than the next. they are hosted on a low cost vps provider so maybe their ranges are common low hanging fruit.
  - throwaway613834 9y ago
    I wouldn't have revealed the exact port number here, but thanks :) yeah, I think your explanation sounds plausible.
    - firethief 9y ago
      A port number isn't a real secret, it just screens out the bulk of mass scans. There's no point trying to hide it from targeted attackers.
  - user5994461 9y ago
    The hosting providers have known IP ranges and they are scanned permanently. I wouldn't be surprised if some guys had a rule to try 8022 and the most common non standard ports.

tombrossman 9y ago

Is this server running a recent version of Ubuntu and using systemd? Also, have you verified that fail2ban actually blocks you by intentionally making bad login attempts to trigger a ban? I do this with every new machine and never assume it works until I can trigger it on demand. (test using a VPN or set a very short bantime so you don't lock yourself out) I ask because systemd may not be writing the failed login attempts to the log file fail2ban monitors, which means fail2ban may not be responding to attacks. I don't recall exactly how I resolved this, but I think I had to set "backend = systemd" in my jail.local under the [DEFAULT] section header.

throwaway613834 9y ago
Oh wow, thanks a ton for mentioning this. It's Ubuntu indeed but I seem to have sysvinit rather than systemd from what I can tell. I have no idea how to actually test it in a way that ensures I will trigger it though (since I don't really have other setups that I can test the same method on and ensure I'm doing it correctly). What do I have to do to trigger it? Just fail a bunch of SSH logins in a row? (I have ssh, dropbear, pam-generic, and ssh-ddos enabled.)
- tombrossman 9y ago
  > What do I have to do to trigger it? Just fail a bunch of SSH logins in a row? Yes, that's what I do. If you have SSH login via password disabled (you should) then just temporarily move your private key(s) out of your ~/.ssh directory and try logging in a few times. "ssh user@machine && ssh user@machine && ssh user@machine" will do it if you have a ban set for three failed attempts. If it's working, your target machine should no longer respond at all, and connections will time out. If it's a web server the site should be unreachable with your web browser until after the bantime passes.
  - throwaway613834 9y ago
    Thanks, I'll give it a try :) I've gotten myself locked out before like that, but ironically from a different machine...

andimm 9y ago

Don't know if it is a too obvious answer, but have a look at the output of 'sudo lastb', it will show failed attempts quickly to verify if it is plausible and fail2ban properly setup.

throwaway613834 9y ago
I've never heard of the command, so it's not obvious, thanks. However, I'm not sure how to interpret this. All I see is a single line: btmp begins Fri Sep 1 06:25:05 2017 Does this just mean nothing has happened? I'm not sure what's special about that date; the setup has certainly been running for much longer than that...
- andimm 9y ago
  That's not a bad sign then. Mine shows lots of attempts. lastb shows the last unsuccessful (bad) login attempts. it uses the file /var/log/btmp as input. the time stamp of "btmp begins .." either is the same as the creation date of the btmp file or if the btmp file was rotate of an older version i.e. /var/log/btmp.0