5 ms·

Here's a counter-example (I said else-where in this thread): Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will b

by acoard 6y ago

Here's a counter-example (I said else-where in this thread): Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. I'll also point out that we're generally talking about different threat vectors here, so it's good to lay them out. I don't think obscurity helps against a persistent threat probing your network, it helps against swarms. > a non-privileged user can bind to a port above 10k, but can't bind to 22. sshd restarts for an upgrade, or your iptables rules remapping a high port to 22 get flushed, that non-privileged user that got access via a RCE on your web application can now set up their own fake sshd and listen in to whatever you are sending if it manages to bind to that port first and you ignore the host key mismatch error on the client side. This is getting closer to APT territory, but I'll bite. If someone has RCE on your SSH server it honestly doesn't matter what port you're running on. They already have the server. You're completely right it would work if you have separate linux users for SSH and web server. Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). But let's assume it here. In reality, even if you did have this setup this is a skilled persistent threat we're talking about (not quite an APT, but definitely a PT). They already own your website. Your compromised web/SSH server is being monitored by a skilled hacker, it's inevitable they'll escalate privileges. If they're smart enough to put in fake SSH daemons, they're smart enough to figure something else out. Is your server perfectly patched? Has anyone in your organization re-used passwords on your website and gmail? You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact: * Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!). * Use standard port, but you still have an APT who owns your web server and will find other exploits.

cthalupa 6y ago
>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. Yep! And I should be clear: I am not saying just don't change the SSH port. I'm saying if you care about security, at a minimum disallow public access to SSH and set up a VPN at a minimum. >Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). I'm a bit confused here. In every major distro I've worked on (RHEL/Cent, Ubuntu, Debian, SUSE) the default httpd and nginx packages are all configured to use their own user for the running service. I haven't seen a system where httpd or nginx are running as root in over a decade. I think the bare minimum for anyone that is running a business or keeping customer/end user data should be the following: 1) Only allow public access to the public facing services. All other ports should be firewalled off or not listening at all on the public interface 2) Public facing services should not be running as root (I'm terrified that you've not seen this to be the case in the majority of places!) 3) Access to the secure side should only be available via VPN. 4) SSH is only available via key access and not password. 5) 2FA is required I think the following are also good practices to follow and are not inherently high complexity with the tooling we have available today: 1) SSH access from the VPN is only allowed to jumpboxes 2) These jumpboxes are recycled on a frequent basis from a known good image 3) There is auditing in place for all SSH access to these jumpboxes 4) SSH between production hosts (e.g. webserver 1 to appserver 1 or webserver 2) is disabled and will result in an alarm With the first set, you take care of the overwhelming majority of both swarms and persistent threats. The second set will take care of basically everyone except an APT. The first set you can roll out in an afternoon. With the first set, you take care of the overwhelming majority of situations.
- bleepblorp 6y ago
  Protecting sshd behind a VPN just moves your 0day risk from sshd to the VPN server. Choosing between exposing sshd or a VPN server is just a bet on which of these services is most at risk of a 0day. If you need to defend against 0days then you need to do things like leveraging AppArmor/Selinux, complex port knocking, and/or restricting VPN/SSH access only to whitelisted IP blocks.
  - cthalupa 6y ago
    Except you don't assume that just because someone is on the VPN you're secure. If the VPN server has a 0day, they now have... only as much access as they had before when things were public facing. You still need there to be a simultaneous sshd 0day. I'll take my chances on there being a 0day for wireguard at the same time there's a 0day for sshd. (I do also use selinux and think that you should for reasons far beyond just ssh security)
    - bleepblorp 6y ago
      A remote code execution 0day in your VPN server doesn't give an attacker an unauthorized VPN connection, it gives them remote code execution inside the VPN server process, which gives the attacker whatever access rights the VPN server has on the host. At this point, connecting to sshd is irrelevant. Worse, since Wireguard runs in kernel space, if there's an RCE 0day in Wireguard, an attacker would be able to execute hostile code within the kernel. One remote code exploit in a public-facing service is all it takes for an attacker to get a foothold.
      - cthalupa 6y ago
        I do not run my VPNs on the same systems I am running other services on, so an RCE at most compromises the VPN concentrator and does not inherently give them access to other systems. Access to SSH on production systems is only available through a jumphost which has auditing of all logins sent to another system, and requires 2FA. There are some other services accessible via VPN, but those also require auth and 2FA. If you are running them all on the same system, then yes, that is a risk.
gruez 6y ago
>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. But how much of a threat is this? Who's going to drop a ssh 0day with PoC for script kiddies to use? If it's a bad guy he's going to sell it on the black market for $$$. If it's a bad guy he's going to responsibly disclose. >You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact: >* Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!). But blocking 50% of the hacking attempts don't make you 50% more secure, or even 1% more secure. You're blocking 50% of the bottom of the barrel when it comes to effort, so having a reasonably secure password (ie. not on a wordlist) or using public key authentication would already stop them.
- benttoothpaste 6y ago
  It makes the logs less noisy. And with much less noisy logs it is easier to notice if something undesirable is happening. Also from my experience this 50% is more like 99%.
ayyy 6y ago
> Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). If you made a list of things like this which annoy you, I would enjoy reading it.
ryanlol 6y ago
> Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. And with all those compromised servers they could easily scan for sshd on all ports.

by acoard 6y ago

cthalupa 6y ago

>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. Yep! And I should be clear: I am not saying just don't change the SSH port. I'm saying if you care about security, at a minimum disallow public access to SSH and set up a VPN at a minimum. >Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). I'm a bit confused here. In every major distro I've worked on (RHEL/Cent, Ubuntu, Debian, SUSE) the default httpd and nginx packages are all configured to use their own user for the running service. I haven't seen a system where httpd or nginx are running as root in over a decade. I think the bare minimum for anyone that is running a business or keeping customer/end user data should be the following: 1) Only allow public access to the public facing services. All other ports should be firewalled off or not listening at all on the public interface 2) Public facing services should not be running as root (I'm terrified that you've not seen this to be the case in the majority of places!) 3) Access to the secure side should only be available via VPN. 4) SSH is only available via key access and not password. 5) 2FA is required I think the following are also good practices to follow and are not inherently high complexity with the tooling we have available today: 1) SSH access from the VPN is only allowed to jumpboxes 2) These jumpboxes are recycled on a frequent basis from a known good image 3) There is auditing in place for all SSH access to these jumpboxes 4) SSH between production hosts (e.g. webserver 1 to appserver 1 or webserver 2) is disabled and will result in an alarm With the first set, you take care of the overwhelming majority of both swarms and persistent threats. The second set will take care of basically everyone except an APT. The first set you can roll out in an afternoon. With the first set, you take care of the overwhelming majority of situations.

bleepblorp 6y ago
Protecting sshd behind a VPN just moves your 0day risk from sshd to the VPN server. Choosing between exposing sshd or a VPN server is just a bet on which of these services is most at risk of a 0day. If you need to defend against 0days then you need to do things like leveraging AppArmor/Selinux, complex port knocking, and/or restricting VPN/SSH access only to whitelisted IP blocks.
- cthalupa 6y ago
  Except you don't assume that just because someone is on the VPN you're secure. If the VPN server has a 0day, they now have... only as much access as they had before when things were public facing. You still need there to be a simultaneous sshd 0day. I'll take my chances on there being a 0day for wireguard at the same time there's a 0day for sshd. (I do also use selinux and think that you should for reasons far beyond just ssh security)
  - bleepblorp 6y ago
    A remote code execution 0day in your VPN server doesn't give an attacker an unauthorized VPN connection, it gives them remote code execution inside the VPN server process, which gives the attacker whatever access rights the VPN server has on the host. At this point, connecting to sshd is irrelevant. Worse, since Wireguard runs in kernel space, if there's an RCE 0day in Wireguard, an attacker would be able to execute hostile code within the kernel. One remote code exploit in a public-facing service is all it takes for an attacker to get a foothold.
    - cthalupa 6y ago
      I do not run my VPNs on the same systems I am running other services on, so an RCE at most compromises the VPN concentrator and does not inherently give them access to other systems. Access to SSH on production systems is only available through a jumphost which has auditing of all logins sent to another system, and requires 2FA. There are some other services accessible via VPN, but those also require auth and 2FA. If you are running them all on the same system, then yes, that is a risk.

gruez 6y ago

>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. But how much of a threat is this? Who's going to drop a ssh 0day with PoC for script kiddies to use? If it's a bad guy he's going to sell it on the black market for $$$. If it's a bad guy he's going to responsibly disclose. >You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact: >* Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!). But blocking 50% of the hacking attempts don't make you 50% more secure, or even 1% more secure. You're blocking 50% of the bottom of the barrel when it comes to effort, so having a reasonably secure password (ie. not on a wordlist) or using public key authentication would already stop them.

benttoothpaste 6y ago
It makes the logs less noisy. And with much less noisy logs it is easier to notice if something undesirable is happening. Also from my experience this 50% is more like 99%.

ayyy 6y ago

> Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). If you made a list of things like this which annoy you, I would enjoy reading it.

ryanlol 6y ago

> Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. And with all those compromised servers they could easily scan for sshd on all ports.