6 ms·

This article was very illuminating. And, I wish it was written like this: To really secure your SSH server, do these three things: 1. Setup a trusted autho

by xrd 5y ago

This article was very illuminating. And, I wish it was written like this: To really secure your SSH server, do these three things: 1. Setup a trusted authority 2. Use ssh certificates 3. With ssh certs, you can easily add MFA for logins expire until you reauth. Now that we've established this, here are the gory details. Lorem ipsum, lorem ipsum, lorem ipsum. It's a great narrative, but I wish I knew the big payout in advance.

michael_j_ward 5y ago
Is that what I get out of the box using tailscale for ssh? https://tailscale.com/kb/1009/protect-ssh-servers/ https://tailscale.com/kb/1009/protect-ssh-servers/
- tener 5y ago
  Tailscale (and other similar solutions) works on the network level. This is not a bad idea in itself, but SSH certs operate on the application level. The fact you can ping the server shouldn't mean you are allowed to actually access it.
- tptacek 5y ago
  Two examples of things Tailscale doesn't give you for this usage model that SSH CAs can: * Transcript-level audit trails for what people are actually doing on SSH sessions. * Differential access to different groups of users to the same machines. Tailscale and SSH CAs work together nicely: require membership in the right Tailscale group to talk to SSH at all, thus tying access to SSH to your (e.g.) Google login and MFA requirement, and use something like Teleport for the actual SSH login, to get the audit log, group access, and an additional authentication factor.
AtlasBarfed 5y ago
MFA for ssh means you can't automate cluster-level ops. Unless, well, you automate the MFA, which defeats the MFA. This drives me a bit nuts about security people in the age of cloudscale. They assume you don't mind MFA'ing every hour and are manually doing logins and accesses for everything. Yeah, uh, I need to script orchestration on several hundred machines at once, and orchestrate/access on the scale of hours or even days for some things like "Big Data" backups or restores. If certificates are anything like SSL certs and the horrorshow cli tools / options / management involved in those, no thanks. I'd rather have an automated sshkey switchover, or for stateless just routinely cycle the infrastructure with new keys. It's been a long time tenet of security that you want an open algorithm that gets broadly and publicly challenged so you know its secure. Well, in the age of the state actors, this might not be the whole truth. I think layering some klugy not-invented-here obfuscation atop the more battletested methods is a useful and important deterrent/delay. Sure someone will figure it out, but they have to TRY HARD. A lot of the institutional attacks seem to be based on human attacks on standardized systems, which HAVE to allow human access and vectors. So in AWS land, secrets manager is secure unless you get the permissions. Then you have everything. But if each of those secrets has some whacko obfuscation for the various apps, then that is a big slowdown to the human attack vectors. And the state actors? Well, even they have budgets. They'll probably move on to easier targets. If a state actor is motivated at targeting your company in particular, well, given that they'll have malware in the firmware of your hard drives and motherboards and the like, you're probably helpless. Finally, what really bothers me about most security is that it leaves one of the most important "canaries in the coal mines" aspects of security: honeypots. Sure do the diligence on securing the access, but how about some turnkey approaches for setting up honeypots to detect when people are poking around? Honeypots are perfect for that, because the devs only care about the stuff they are working on. The intruders are doing the scanning.
- fishpen0 5y ago
  Cluster-level ops really shouldn't be done by direct SSH. Tools like ansible, salt, chef, puppet, etc... all can be run in some form of daemon mode with a central management server for a reason. You should be authenticating against the management service and running your configuration or automation scripts from there, not as a massive pool of CSSH or whatever directly from your laptop. Over the last 6 years I've worked for three different companies that all universally disabled ssh and we never had troubles running management scripts or tooling.
  - outworlder 5y ago
    Until the daemon breaks and someone has to get in with SSH anyway? Chef is notorious for doing this. Or at least it was before we got rid of it. Some random script would break and then the run would be incomplete. And we couldn't just fix cookbooks as the run wouldn't complete. Some deployments make the daemon approach (that phones home) difficult. Such as management in a corporate network. It's easy to configure AWS and the like to accept requests from well known corporate gateways. It's not as easy to make them from the outside the corporate network in. And even when that's doable, different cloud providers and regions make it difficult. You end up having a bunch of chef (or similar) servers scattered around.
    - fishpen0 5y ago
      In the rare case this happens we still don't use raw SSH. We rely on something identity-driven like SSM in AWS or IAP in GCP to initiate the tunnel.
      - AtlasBarfed 5y ago
        GCP IAP sounds like Teleport, which we've already run into issues with since the Teleport daemon will die/not accept connections in some situations, while the good ol sshd does. Like: full disks, memory stress, or (I think) the teleport daemon getting killed. SSM sounds like an advanced port knock. Or you could toggle the security group port access, or keep the bastion down and spin it up if you need it.
zepearl 5y ago
I agree, something like that. From what I've experienced it's (for me) a common problem in that realm: a lot of blah-blah that makes my eyes glaze/lose focus before getting to the core/overview. ~15 years ago, when the company I'm working for first implemented PKI, I needed something like 100 hours of "help" from local security engineers and the external software-vendor's programmers to understand how that worked and what the SW was supposed to do. After that, explaining to colleagues at least on a high level how that works became a matter of minutes. To be fair towards myself, even the local scrty eng gurus (relaxed, as they themselves didn't have to "deliver" anything) and the external vendor's programming gurus (hardcore, as nothing would be paid if the SW could not be implmemented) were often absolutely not understanding each other, so I ended up becoming their unofficial mediator/translator: if I felt like an obvious question was not dared to be asked by one of the parties then I sacrificed my self-esteem to dare asking it, if "even I" did manage to understand some concept then both parties were supposed to get it as well and if not then at least I could act as gateway to explain/expand offline :P It was an interesting time - not nice nor very bad (a little bit bad, as we had of course an implementation deadline), but at least I learned a lot, as well in the area of social skills :)

by xrd 5y ago

michael_j_ward 5y ago

Is that what I get out of the box using tailscale for ssh? https://tailscale.com/kb/1009/protect-ssh-servers/ https://tailscale.com/kb/1009/protect-ssh-servers/

tener 5y ago
Tailscale (and other similar solutions) works on the network level. This is not a bad idea in itself, but SSH certs operate on the application level. The fact you can ping the server shouldn't mean you are allowed to actually access it.
tptacek 5y ago
Two examples of things Tailscale doesn't give you for this usage model that SSH CAs can: * Transcript-level audit trails for what people are actually doing on SSH sessions. * Differential access to different groups of users to the same machines. Tailscale and SSH CAs work together nicely: require membership in the right Tailscale group to talk to SSH at all, thus tying access to SSH to your (e.g.) Google login and MFA requirement, and use something like Teleport for the actual SSH login, to get the audit log, group access, and an additional authentication factor.

AtlasBarfed 5y ago

MFA for ssh means you can't automate cluster-level ops. Unless, well, you automate the MFA, which defeats the MFA. This drives me a bit nuts about security people in the age of cloudscale. They assume you don't mind MFA'ing every hour and are manually doing logins and accesses for everything. Yeah, uh, I need to script orchestration on several hundred machines at once, and orchestrate/access on the scale of hours or even days for some things like "Big Data" backups or restores. If certificates are anything like SSL certs and the horrorshow cli tools / options / management involved in those, no thanks. I'd rather have an automated sshkey switchover, or for stateless just routinely cycle the infrastructure with new keys. It's been a long time tenet of security that you want an open algorithm that gets broadly and publicly challenged so you know its secure. Well, in the age of the state actors, this might not be the whole truth. I think layering some klugy not-invented-here obfuscation atop the more battletested methods is a useful and important deterrent/delay. Sure someone will figure it out, but they have to TRY HARD. A lot of the institutional attacks seem to be based on human attacks on standardized systems, which HAVE to allow human access and vectors. So in AWS land, secrets manager is secure unless you get the permissions. Then you have everything. But if each of those secrets has some whacko obfuscation for the various apps, then that is a big slowdown to the human attack vectors. And the state actors? Well, even they have budgets. They'll probably move on to easier targets. If a state actor is motivated at targeting your company in particular, well, given that they'll have malware in the firmware of your hard drives and motherboards and the like, you're probably helpless. Finally, what really bothers me about most security is that it leaves one of the most important "canaries in the coal mines" aspects of security: honeypots. Sure do the diligence on securing the access, but how about some turnkey approaches for setting up honeypots to detect when people are poking around? Honeypots are perfect for that, because the devs only care about the stuff they are working on. The intruders are doing the scanning.

fishpen0 5y ago
Cluster-level ops really shouldn't be done by direct SSH. Tools like ansible, salt, chef, puppet, etc... all can be run in some form of daemon mode with a central management server for a reason. You should be authenticating against the management service and running your configuration or automation scripts from there, not as a massive pool of CSSH or whatever directly from your laptop. Over the last 6 years I've worked for three different companies that all universally disabled ssh and we never had troubles running management scripts or tooling.
- outworlder 5y ago
  Until the daemon breaks and someone has to get in with SSH anyway? Chef is notorious for doing this. Or at least it was before we got rid of it. Some random script would break and then the run would be incomplete. And we couldn't just fix cookbooks as the run wouldn't complete. Some deployments make the daemon approach (that phones home) difficult. Such as management in a corporate network. It's easy to configure AWS and the like to accept requests from well known corporate gateways. It's not as easy to make them from the outside the corporate network in. And even when that's doable, different cloud providers and regions make it difficult. You end up having a bunch of chef (or similar) servers scattered around.
  - fishpen0 5y ago
    In the rare case this happens we still don't use raw SSH. We rely on something identity-driven like SSM in AWS or IAP in GCP to initiate the tunnel.
    - AtlasBarfed 5y ago
      GCP IAP sounds like Teleport, which we've already run into issues with since the Teleport daemon will die/not accept connections in some situations, while the good ol sshd does. Like: full disks, memory stress, or (I think) the teleport daemon getting killed. SSM sounds like an advanced port knock. Or you could toggle the security group port access, or keep the bastion down and spin it up if you need it.

zepearl 5y ago

I agree, something like that. From what I've experienced it's (for me) a common problem in that realm: a lot of blah-blah that makes my eyes glaze/lose focus before getting to the core/overview. ~15 years ago, when the company I'm working for first implemented PKI, I needed something like 100 hours of "help" from local security engineers and the external software-vendor's programmers to understand how that worked and what the SW was supposed to do. After that, explaining to colleagues at least on a high level how that works became a matter of minutes. To be fair towards myself, even the local scrty eng gurus (relaxed, as they themselves didn't have to "deliver" anything) and the external vendor's programming gurus (hardcore, as nothing would be paid if the SW could not be implmemented) were often absolutely not understanding each other, so I ended up becoming their unofficial mediator/translator: if I felt like an obvious question was not dared to be asked by one of the parties then I sacrificed my self-esteem to dare asking it, if "even I" did manage to understand some concept then both parties were supposed to get it as well and if not then at least I could act as gateway to explain/expand offline :P It was an interesting time - not nice nor very bad (a little bit bad, as we had of course an implementation deadline), but at least I learned a lot, as well in the area of social skills :)