3 ms·

Cluster-level ops really shouldn't be done by direct SSH. Tools like ansible, salt, chef, puppet, etc... all can be run in some form of daemon mode with a cent

Cluster-level ops really shouldn't be done by direct SSH. Tools like ansible, salt, chef, puppet, etc... all can be run in some form of daemon mode with a central management server for a reason. You should be authenticating against the management service and running your configuration or automation scripts from there, not as a massive pool of CSSH or whatever directly from your laptop. Over the last 6 years I've worked for three different companies that all universally disabled ssh and we never had troubles running management scripts or tooling.

outworlder 5y ago
Until the daemon breaks and someone has to get in with SSH anyway? Chef is notorious for doing this. Or at least it was before we got rid of it. Some random script would break and then the run would be incomplete. And we couldn't just fix cookbooks as the run wouldn't complete. Some deployments make the daemon approach (that phones home) difficult. Such as management in a corporate network. It's easy to configure AWS and the like to accept requests from well known corporate gateways. It's not as easy to make them from the outside the corporate network in. And even when that's doable, different cloud providers and regions make it difficult. You end up having a bunch of chef (or similar) servers scattered around.
- fishpen0 5y ago
  In the rare case this happens we still don't use raw SSH. We rely on something identity-driven like SSM in AWS or IAP in GCP to initiate the tunnel.
  - AtlasBarfed 5y ago
    GCP IAP sounds like Teleport, which we've already run into issues with since the Teleport daemon will die/not accept connections in some situations, while the good ol sshd does. Like: full disks, memory stress, or (I think) the teleport daemon getting killed. SSM sounds like an advanced port knock. Or you could toggle the security group port access, or keep the bastion down and spin it up if you need it.
AtlasBarfed 5y ago
Stateful databases or disposable api servers? Disposable API servers you can eliminate ssh access and the like. Containers generally don't run sshd, but they do still often have kubectl/dockerrun if you absolutely need to. SSM sucks on a certain level because the output is capped at ?1MB? I think and you need to poll S3 to use it. Salt daemon polls fine. As you kind of alluded to, well, you can do SSM to "port knock" or simply do a change to a secgrp to flip on the ssh access, and then flip it off afterward. I find Salt/Ansible/k8s too big, you can't "step through" to debug your orchestrations. I would categorize them as "heavyweight". SSH is a good substrate for everything else, including adhoc stuff. Anything that can be run off a laptop can be run off an admin server. Remotely debugged. I love it. With the heavyweight stuff there is too much trial and error: try recipe, it craps, guess what's wrong, it craps, guess again, it craps. The stuff I use actually doesn't require SSH. As long as you can deliver a command and get the output, I can use SSH, SSM (with the crappy limitations but its GREAT for stuff in China), kubectl, salt, dockerrun, teleport (until the token runs out), or combo ssh-to-bastion then do other stuff.

outworlder 5y ago

Until the daemon breaks and someone has to get in with SSH anyway? Chef is notorious for doing this. Or at least it was before we got rid of it. Some random script would break and then the run would be incomplete. And we couldn't just fix cookbooks as the run wouldn't complete. Some deployments make the daemon approach (that phones home) difficult. Such as management in a corporate network. It's easy to configure AWS and the like to accept requests from well known corporate gateways. It's not as easy to make them from the outside the corporate network in. And even when that's doable, different cloud providers and regions make it difficult. You end up having a bunch of chef (or similar) servers scattered around.

fishpen0 5y ago
In the rare case this happens we still don't use raw SSH. We rely on something identity-driven like SSM in AWS or IAP in GCP to initiate the tunnel.
- AtlasBarfed 5y ago
  GCP IAP sounds like Teleport, which we've already run into issues with since the Teleport daemon will die/not accept connections in some situations, while the good ol sshd does. Like: full disks, memory stress, or (I think) the teleport daemon getting killed. SSM sounds like an advanced port knock. Or you could toggle the security group port access, or keep the bastion down and spin it up if you need it.

AtlasBarfed 5y ago

Stateful databases or disposable api servers? Disposable API servers you can eliminate ssh access and the like. Containers generally don't run sshd, but they do still often have kubectl/dockerrun if you absolutely need to. SSM sucks on a certain level because the output is capped at ?1MB? I think and you need to poll S3 to use it. Salt daemon polls fine. As you kind of alluded to, well, you can do SSM to "port knock" or simply do a change to a secgrp to flip on the ssh access, and then flip it off afterward. I find Salt/Ansible/k8s too big, you can't "step through" to debug your orchestrations. I would categorize them as "heavyweight". SSH is a good substrate for everything else, including adhoc stuff. Anything that can be run off a laptop can be run off an admin server. Remotely debugged. I love it. With the heavyweight stuff there is too much trial and error: try recipe, it craps, guess what's wrong, it craps, guess again, it craps. The stuff I use actually doesn't require SSH. As long as you can deliver a command and get the output, I can use SSH, SSM (with the crappy limitations but its GREAT for stuff in China), kubectl, salt, dockerrun, teleport (until the token runs out), or combo ssh-to-bastion then do other stuff.