4 ms·

I know we've spoken in another thread, but I think it's important for people to understand that this sshd thing is a perfect example of why it isn't this easy:

I know we've spoken in another thread, but I think it's important for people to understand that this sshd thing is a perfect example of why it isn't this easy: You reduce log spam moving to a non-privileged port, but also reduce overall security - a non-privileged user can bind to a port above 10k, but can't bind to 22. sshd restarts for an upgrade, or your iptables rules remapping a high port to 22 get flushed, that non-privileged user that got access via a RCE on your web application can now set up their own fake sshd and listen in to whatever you are sending if it manages to bind to that port first and you ignore the host key mismatch error on the client side. Or you can implement real security, like not allowing SSH access via the public internet at all and not have to make this trade off.

Sebb767 6y ago
Fair enough, I've edited the comment to reflect this :)
acoard 6y ago
Here's a counter-example (I said else-where in this thread): Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. I'll also point out that we're generally talking about different threat vectors here, so it's good to lay them out. I don't think obscurity helps against a persistent threat probing your network, it helps against swarms. > a non-privileged user can bind to a port above 10k, but can't bind to 22. sshd restarts for an upgrade, or your iptables rules remapping a high port to 22 get flushed, that non-privileged user that got access via a RCE on your web application can now set up their own fake sshd and listen in to whatever you are sending if it manages to bind to that port first and you ignore the host key mismatch error on the client side. This is getting closer to APT territory, but I'll bite. If someone has RCE on your SSH server it honestly doesn't matter what port you're running on. They already have the server. You're completely right it would work if you have separate linux users for SSH and web server. Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). But let's assume it here. In reality, even if you did have this setup this is a skilled persistent threat we're talking about (not quite an APT, but definitely a PT). They already own your website. Your compromised web/SSH server is being monitored by a skilled hacker, it's inevitable they'll escalate privileges. If they're smart enough to put in fake SSH daemons, they're smart enough to figure something else out. Is your server perfectly patched? Has anyone in your organization re-used passwords on your website and gmail? You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact: * Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!). * Use standard port, but you still have an APT who owns your web server and will find other exploits.
- cthalupa 6y ago
  >Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. Yep! And I should be clear: I am not saying just don't change the SSH port. I'm saying if you care about security, at a minimum disallow public access to SSH and set up a VPN at a minimum. >Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). I'm a bit confused here. In every major distro I've worked on (RHEL/Cent, Ubuntu, Debian, SUSE) the default httpd and nginx packages are all configured to use their own user for the running service. I haven't seen a system where httpd or nginx are running as root in over a decade. I think the bare minimum for anyone that is running a business or keeping customer/end user data should be the following: 1) Only allow public access to the public facing services. All other ports should be firewalled off or not listening at all on the public interface 2) Public facing services should not be running as root (I'm terrified that you've not seen this to be the case in the majority of places!) 3) Access to the secure side should only be available via VPN. 4) SSH is only available via key access and not password. 5) 2FA is required I think the following are also good practices to follow and are not inherently high complexity with the tooling we have available today: 1) SSH access from the VPN is only allowed to jumpboxes 2) These jumpboxes are recycled on a frequent basis from a known good image 3) There is auditing in place for all SSH access to these jumpboxes 4) SSH between production hosts (e.g. webserver 1 to appserver 1 or webserver 2) is disabled and will result in an alarm With the first set, you take care of the overwhelming majority of both swarms and persistent threats. The second set will take care of basically everyone except an APT. The first set you can roll out in an afternoon. With the first set, you take care of the overwhelming majority of situations.
  - bleepblorp 6y ago
    Protecting sshd behind a VPN just moves your 0day risk from sshd to the VPN server. Choosing between exposing sshd or a VPN server is just a bet on which of these services is most at risk of a 0day. If you need to defend against 0days then you need to do things like leveraging AppArmor/Selinux, complex port knocking, and/or restricting VPN/SSH access only to whitelisted IP blocks.
deleted 6y ago
[deleted]
kaydub 6y ago
Yeah, I'm seeing a lot of nonsense in here. Why is SSH publicly accessible in the first place??? Security through obscurity is just some feel good bullshit.
- Symbiote 6y ago
  Why shouldn't SSH be public? It is useful and simple to secure.
  - cthalupa 6y ago
    Well, there's basically two stances you can reasonably take: 1) SSH is secure enough just by using key based auth to not worry about it. 2) SSH isn't secure enough just by using key based auth so we need to do more stuff. If you believe #1, then you don't need to do anything else. If you believe #2, then you should be doing the things that provide the most effective security. Personally, I believe #1 is probably correct, but when it comes to any system that contains data for users other than myself, or for anything related to a company, I should not make that bet and should instead follow #2 and implement proper security for that eventuality. I'm willing to risk my own shit when it comes to #1, but not other people's.

Sebb767 6y ago

Fair enough, I've edited the comment to reflect this :)

acoard 6y ago

Here's a counter-example (I said else-where in this thread): Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. I'll also point out that we're generally talking about different threat vectors here, so it's good to lay them out. I don't think obscurity helps against a persistent threat probing your network, it helps against swarms. > a non-privileged user can bind to a port above 10k, but can't bind to 22. sshd restarts for an upgrade, or your iptables rules remapping a high port to 22 get flushed, that non-privileged user that got access via a RCE on your web application can now set up their own fake sshd and listen in to whatever you are sending if it manages to bind to that port first and you ignore the host key mismatch error on the client side. This is getting closer to APT territory, but I'll bite. If someone has RCE on your SSH server it honestly doesn't matter what port you're running on. They already have the server. You're completely right it would work if you have separate linux users for SSH and web server. Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). But let's assume it here. In reality, even if you did have this setup this is a skilled persistent threat we're talking about (not quite an APT, but definitely a PT). They already own your website. Your compromised web/SSH server is being monitored by a skilled hacker, it's inevitable they'll escalate privileges. If they're smart enough to put in fake SSH daemons, they're smart enough to figure something else out. Is your server perfectly patched? Has anyone in your organization re-used passwords on your website and gmail? You're right that these events could happen. But you have to ask yourself what's actions of yours will have a bigger impact: * Changing to non-standard SSH port, blocking out ~50% of all automated hacking attempts. Or port-knocking to get >90% (just a guess!). * Use standard port, but you still have an APT who owns your web server and will find other exploits.

cthalupa 6y ago
>Imagine a 0day for SSH drops tomorrow. Almost immediately script kiddies all over the world will be trying to take over everything running on port 22. Yep! And I should be clear: I am not saying just don't change the SSH port. I'm saying if you care about security, at a minimum disallow public access to SSH and set up a VPN at a minimum. >Unfortunately that's all too rare in most web-servers I see (<10%), as most just add SSH and secure it and call it a day (even worse when CI/CD scripts just copy files without chowning them). I'm a bit confused here. In every major distro I've worked on (RHEL/Cent, Ubuntu, Debian, SUSE) the default httpd and nginx packages are all configured to use their own user for the running service. I haven't seen a system where httpd or nginx are running as root in over a decade. I think the bare minimum for anyone that is running a business or keeping customer/end user data should be the following: 1) Only allow public access to the public facing services. All other ports should be firewalled off or not listening at all on the public interface 2) Public facing services should not be running as root (I'm terrified that you've not seen this to be the case in the majority of places!) 3) Access to the secure side should only be available via VPN. 4) SSH is only available via key access and not password. 5) 2FA is required I think the following are also good practices to follow and are not inherently high complexity with the tooling we have available today: 1) SSH access from the VPN is only allowed to jumpboxes 2) These jumpboxes are recycled on a frequent basis from a known good image 3) There is auditing in place for all SSH access to these jumpboxes 4) SSH between production hosts (e.g. webserver 1 to appserver 1 or webserver 2) is disabled and will result in an alarm With the first set, you take care of the overwhelming majority of both swarms and persistent threats. The second set will take care of basically everyone except an APT. The first set you can roll out in an afternoon. With the first set, you take care of the overwhelming majority of situations.
- bleepblorp 6y ago
  Protecting sshd behind a VPN just moves your 0day risk from sshd to the VPN server. Choosing between exposing sshd or a VPN server is just a bet on which of these services is most at risk of a 0day. If you need to defend against 0days then you need to do things like leveraging AppArmor/Selinux, complex port knocking, and/or restricting VPN/SSH access only to whitelisted IP blocks.

deleted 6y ago

[deleted]

kaydub 6y ago

Yeah, I'm seeing a lot of nonsense in here. Why is SSH publicly accessible in the first place??? Security through obscurity is just some feel good bullshit.

Symbiote 6y ago
Why shouldn't SSH be public? It is useful and simple to secure.
- cthalupa 6y ago
  Well, there's basically two stances you can reasonably take: 1) SSH is secure enough just by using key based auth to not worry about it. 2) SSH isn't secure enough just by using key based auth so we need to do more stuff. If you believe #1, then you don't need to do anything else. If you believe #2, then you should be doing the things that provide the most effective security. Personally, I believe #1 is probably correct, but when it comes to any system that contains data for users other than myself, or for anything related to a company, I should not make that bet and should instead follow #2 and implement proper security for that eventuality. I'm willing to risk my own shit when it comes to #1, but not other people's.