6 ms·

I mean, this is definitely cool, but we should also try to stop using ssh so much. There's a long list of reasons why using ssh leads to bad things (but not an

I mean, this is definitely cool, but we should also try to stop using ssh so much. There's a long list of reasons why using ssh leads to bad things (but not an anti-pattern - I wish people would stop using that phrase to mean anything that sometimes leads to bad things) so I just hope this functionality doesn't exacerbate its use.

tomcam 7y ago
Ugh, I'm a new full stack guy. Could you enlighten me about the evils of SSH?
- peterwwillis 7y ago
  SSH is wonderful. The problems are more what it enables, and what it lacks (or isn't designed to do). For example, managing ssh keys for an individual is gloriously simple, but managing them for a large organization is a huge headache. You want to use ssh certificates, but even those are implemented in a weird way, and really you should use an SSO system for auth. (This makes that easier/better, so, yay?) When people start sshing into production servers, they end up making local changes. They focus more on the "pets" aspect of managing systems rather than as "cattle". They have to install a litany of extra software to diagnose and troubleshoot bugs, rather than expose system metrics and tightly control the app environment and its operation. Remote access to production app servers is basically a backdoor waiting to happen, and may violate corporate security policies. When you have local user access to a Linux host, it's almost guaranteed you can privesc to root. Finally, almost everyone I have ever seen will either force-ignore/auto-accept host key changes, or just accept them blindly, because IPs and hostnames may change, and there may be multiple environments you haven't logged in to, etc). This completely defeats the purpose behind mitm protection, which is the main intent of using SSH, though these days its other features may be arguably more of a reason to use it. And for the tech hipsters out there: "it isn't serverless!!"
  - cthalupa 7y ago
    >They have to install a litany of extra software to diagnose and troubleshoot bugs, rather than expose system metrics and tightly control the app environment and its operation. There's a lot of things that require more than easily exported system metrics and logs to troubleshoot. While I've played around with using PCP's perf plugin to try and remotely do things with perf, generate flamegraphs, etc., it doesn't work nearly as well as just SSH'ing into the thing and running perf directly, especially if the perf data file is going to be large. I don't see how you could do serious performance engineering work without SSH access. But, I think I'm nitpicking here, because I generally agree that there should be very little to no reason to login to servers via SSH day to day.
    - peterwwillis 7y ago
      As an alternative, I try to implement solutions that can install and run software out of bounds and still get what you need without a persistent connection and opening up security groups (example: AWS SSM Run Command).
      - cthalupa 7y ago
        I make extensive use of Run Command (And Automations!) but they're not a replacement for every use case, and very much not a replacement for the specific use case you're replying to.
    - jameshart 7y ago
      in the ‘pets vs cattle’ analogy, it totally makes sense that even with cattle, occasionally you bring one in for a checkup by a vet to see if you can detect any problems that might affect the herd. Ssh into a production box to check everything is working as expected and take some readings. Sure. On the other hand, I tend to lean more towards a ‘wild animals’ model, where, sure, you can tranquilize one and bring it in to look over, but once it’s got the smell of humans on it, it’s doomed if you let it back out in the wild again. Once you ssh into a production box, it is forever tainted. Sure, poke around in it, install some perf tools to run some diagnostics, learn what you can about its behavior. But then, rather than putting it back into the wild to serve traffic, out of mercy, you should destroy it and replace it with a clean instance.
  - tomcam 7y ago
    > almost everyone I have ever seen will either force-ignore/auto-accept host key changes, Thanks for the super informative answer. About the quoted portion... yeah! I assume it's my responsibility to do something... like manually check the host IP or something? What is the recommended practice to deal with this situation?
- crehn 7y ago
  Nothing bad with SSH per se, but building your infrastructure in a way that makes ad-hoc remote changes unnecessary is something to strive for. For anything but small deployments, automation, immutability and reproducibility will keep you sane. Less moving parts, things don’t suddenly change, easy to audit, easy to rollback, etc.
adamrt 7y ago
Do you mind sharing the long list?
- zxcmx 7y ago
  Not O.P, but: - Encourages you to go to instances and check stuff rather than improve monitoring / health checks. - Can do quick fixes on a few boxes rather than re-running the deploy. Great! But terrible when the person who knows how to do that is away. - Tailing log files rather than centralised log management for all the things. - Trying things out / quickly checking something in production rather than being rigorous about keeping test / staging in sync with prod. The “problem” is ssh is such a great affordance (until you have tons and tons of instances and you can’t do anything by hand anymore) that it means you don’t need to fix internal processes and tools around deployment, configuration and monitoring. If there’s no workaround you feel the pain and will be forced to set things up right, usually with benefits to security and repeatability. As is often the case, the best thing about ssh (in terms of managing infra instances) is also the worst thing. With that said at very small scales it might be overkill to automate all the things so sure, fill in the gaps with ssh and a wiki page.
  - KaiserPro 7y ago
    > Encourages you to go to instances and check stuff rather than improve monitoring / health checks. I don't think it does, well it doesn't when you have > 40 machines anyway. Plus it doesn't give the ability to compare and contrast simply. (graphs are _awesome_) > Tailing log files rather than centralised log management for all the things. Yes, I tend to agree. But proper centralised logging is either exceptionally hard, or a hefty splunk tax. That also encourages people to derive graphs from logs, which is arse about face. Graphs first, logs when you are desperate. > Trying things out / quickly checking something in production I can see this, but normally one would expect people to not have general access to prod, if they are going to do that...
davexunit 7y ago
I am tired of hearing about why I shouldn't use SSH. SSH is fantastic and I always want it enabled on every single server so when things go wrong I can debug. I've had AWS "solutions architects" puzzled about why I'm still using SSH but they can never justify any other solution. First they tell me that I should just log everything we'd want to look at. I do log everything I think would be useful, but unforeseen things happen and it is really handy to have shell access to the misbehaving server. Then they suggest I use Systems Manager to perform server updates but I have no need for that because I use an immutable deployment model. Managing server access in a multi-account organization is a real issue, though. I currently manage 11 AWS accounts and the best solution I've implemented so far is extending NSS and configuring sshd to query our identity provider (Okta) for user/group information and SSH public keys. Each type of server is configured to permit access to a subset of Okta groups. For example, members of the DevOps group get full privileges, anyone else that has a use-case for using SSH (like in a QA environment) gets some form of limited access. With this in place, I can grant/revoke privileges and manage developer keys all from one central location.
- peterwwillis 7y ago
  I hear you, but every time I ask "what are you doing that you need SSH?", the answer comes down to "well I have these crappy random applications and I don't have enough visibility into the system or the apps." For those cases, I think SSH is a crutch that keeps the system from maturing. Another way to think about it is, if you're an SRE, you want to eliminate toil, and an interactive SSH terminal is toil.
  - davexunit 7y ago
    We'll just have to agree to disagree. I just don't buy it.

tomcam 7y ago

Ugh, I'm a new full stack guy. Could you enlighten me about the evils of SSH?

peterwwillis 7y ago
SSH is wonderful. The problems are more what it enables, and what it lacks (or isn't designed to do). For example, managing ssh keys for an individual is gloriously simple, but managing them for a large organization is a huge headache. You want to use ssh certificates, but even those are implemented in a weird way, and really you should use an SSO system for auth. (This makes that easier/better, so, yay?) When people start sshing into production servers, they end up making local changes. They focus more on the "pets" aspect of managing systems rather than as "cattle". They have to install a litany of extra software to diagnose and troubleshoot bugs, rather than expose system metrics and tightly control the app environment and its operation. Remote access to production app servers is basically a backdoor waiting to happen, and may violate corporate security policies. When you have local user access to a Linux host, it's almost guaranteed you can privesc to root. Finally, almost everyone I have ever seen will either force-ignore/auto-accept host key changes, or just accept them blindly, because IPs and hostnames may change, and there may be multiple environments you haven't logged in to, etc). This completely defeats the purpose behind mitm protection, which is the main intent of using SSH, though these days its other features may be arguably more of a reason to use it. And for the tech hipsters out there: "it isn't serverless!!"
- cthalupa 7y ago
  >They have to install a litany of extra software to diagnose and troubleshoot bugs, rather than expose system metrics and tightly control the app environment and its operation. There's a lot of things that require more than easily exported system metrics and logs to troubleshoot. While I've played around with using PCP's perf plugin to try and remotely do things with perf, generate flamegraphs, etc., it doesn't work nearly as well as just SSH'ing into the thing and running perf directly, especially if the perf data file is going to be large. I don't see how you could do serious performance engineering work without SSH access. But, I think I'm nitpicking here, because I generally agree that there should be very little to no reason to login to servers via SSH day to day.
  - peterwwillis 7y ago
    As an alternative, I try to implement solutions that can install and run software out of bounds and still get what you need without a persistent connection and opening up security groups (example: AWS SSM Run Command).
    - cthalupa 7y ago
      I make extensive use of Run Command (And Automations!) but they're not a replacement for every use case, and very much not a replacement for the specific use case you're replying to.
  - jameshart 7y ago
    in the ‘pets vs cattle’ analogy, it totally makes sense that even with cattle, occasionally you bring one in for a checkup by a vet to see if you can detect any problems that might affect the herd. Ssh into a production box to check everything is working as expected and take some readings. Sure. On the other hand, I tend to lean more towards a ‘wild animals’ model, where, sure, you can tranquilize one and bring it in to look over, but once it’s got the smell of humans on it, it’s doomed if you let it back out in the wild again. Once you ssh into a production box, it is forever tainted. Sure, poke around in it, install some perf tools to run some diagnostics, learn what you can about its behavior. But then, rather than putting it back into the wild to serve traffic, out of mercy, you should destroy it and replace it with a clean instance.
- tomcam 7y ago
  > almost everyone I have ever seen will either force-ignore/auto-accept host key changes, Thanks for the super informative answer. About the quoted portion... yeah! I assume it's my responsibility to do something... like manually check the host IP or something? What is the recommended practice to deal with this situation?
crehn 7y ago
Nothing bad with SSH per se, but building your infrastructure in a way that makes ad-hoc remote changes unnecessary is something to strive for. For anything but small deployments, automation, immutability and reproducibility will keep you sane. Less moving parts, things don’t suddenly change, easy to audit, easy to rollback, etc.

adamrt 7y ago

Do you mind sharing the long list?

zxcmx 7y ago
Not O.P, but: - Encourages you to go to instances and check stuff rather than improve monitoring / health checks. - Can do quick fixes on a few boxes rather than re-running the deploy. Great! But terrible when the person who knows how to do that is away. - Tailing log files rather than centralised log management for all the things. - Trying things out / quickly checking something in production rather than being rigorous about keeping test / staging in sync with prod. The “problem” is ssh is such a great affordance (until you have tons and tons of instances and you can’t do anything by hand anymore) that it means you don’t need to fix internal processes and tools around deployment, configuration and monitoring. If there’s no workaround you feel the pain and will be forced to set things up right, usually with benefits to security and repeatability. As is often the case, the best thing about ssh (in terms of managing infra instances) is also the worst thing. With that said at very small scales it might be overkill to automate all the things so sure, fill in the gaps with ssh and a wiki page.
- KaiserPro 7y ago
  > Encourages you to go to instances and check stuff rather than improve monitoring / health checks. I don't think it does, well it doesn't when you have > 40 machines anyway. Plus it doesn't give the ability to compare and contrast simply. (graphs are _awesome_) > Tailing log files rather than centralised log management for all the things. Yes, I tend to agree. But proper centralised logging is either exceptionally hard, or a hefty splunk tax. That also encourages people to derive graphs from logs, which is arse about face. Graphs first, logs when you are desperate. > Trying things out / quickly checking something in production I can see this, but normally one would expect people to not have general access to prod, if they are going to do that...

davexunit 7y ago

I am tired of hearing about why I shouldn't use SSH. SSH is fantastic and I always want it enabled on every single server so when things go wrong I can debug. I've had AWS "solutions architects" puzzled about why I'm still using SSH but they can never justify any other solution. First they tell me that I should just log everything we'd want to look at. I do log everything I think would be useful, but unforeseen things happen and it is really handy to have shell access to the misbehaving server. Then they suggest I use Systems Manager to perform server updates but I have no need for that because I use an immutable deployment model. Managing server access in a multi-account organization is a real issue, though. I currently manage 11 AWS accounts and the best solution I've implemented so far is extending NSS and configuring sshd to query our identity provider (Okta) for user/group information and SSH public keys. Each type of server is configured to permit access to a subset of Okta groups. For example, members of the DevOps group get full privileges, anyone else that has a use-case for using SSH (like in a QA environment) gets some form of limited access. With this in place, I can grant/revoke privileges and manage developer keys all from one central location.

peterwwillis 7y ago
I hear you, but every time I ask "what are you doing that you need SSH?", the answer comes down to "well I have these crappy random applications and I don't have enough visibility into the system or the apps." For those cases, I think SSH is a crutch that keeps the system from maturing. Another way to think about it is, if you're an SRE, you want to eliminate toil, and an interactive SSH terminal is toil.
- davexunit 7y ago
  We'll just have to agree to disagree. I just don't buy it.