3 ms·

The main problem with SSH keys is that they are good at small scales, but at larger scales the abstractions start to be really leaky. Distributing SSH keys acro

by xena 4y ago

The main problem with SSH keys is that they are good at small scales, but at larger scales the abstractions start to be really leaky. Distributing SSH keys across hundreds of machines is a nontrivial task. Not to mention the lack of audit logging in opensshd and the lack of session recording (these requirements suck but acronym compliance means you need them anyways). It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts.

toast0 4y ago
> It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts. On the other hand, if you knew what I'd need to do with the servers, I probably wouldn't need to do it. It's really hard to investigate problems without tools. Nobody is going to make a graph for how much cpu poorly configured docker/datadog are wasting rereading the same 1GB of json logs once or twice a minute, because nobody thought it would be meaningful, but watching top in one window, and running lsof at the right time to see what's being read, and whatever performance tools are hanging out to confirm.
- AtlasBarfed 4y ago
  Ah the dream of easy to administer, perfectly adapted fine grained access permissions that can predict the future of exactly what an authorized user can run and needs to do. It never works because of the lack of precog, ticket walls, escalation over policy requiring reviews, and other bureaucracy. What they end up replacing ssh with (teleport or aws ssm are the two I've been forced to use) are slow, spotty, crash frequently, and who knows what vulnerabilities they have. If all these companies trying to get rid of sshd want to slap their own daemon on to replace it, why not keep sshd and simply have a deamon that manages the keys and not throw out the entire tool chain? The most fun thing with teleport and aws-ssm: to use it you had to bring up a web page to authenticate. There goes any hope of automation at scale. Because in a world where you're supposed to treat servers like cattle, let's impose (and I mean impose) a regime that forces you to access them one at a time manually, especially in critical situations.
imiric 4y ago
> Distributing SSH keys across hundreds of machines is a nontrivial task. SSH certificates are a thing.
- xena 4y ago
  Now you have two problems
  - arp242 4y ago
    You can probably solve those with some regexps.
  - medellin 4y ago
    Please explain. I have deployed them successfully at both large and small companies without any significant issues
- Spivak 4y ago
  Also user directories are thing and you can instruct OpenSSH to use any command you like to fetch a user’s authorized_keys.
- wkat4242 4y ago
  Yes but the OP was referring to SSH keys being eschewed by the enterprise. I assume they didn't consider certificates part of this category. For enterprise the certificates are a far better option. Or s full blown IDP of course.
dreamcompiler 4y ago
> Distributing SSH keys across hundreds of machines is a nontrivial task. In what sense? If you're distributing public keys to hundreds of servers, go nuts. Put them in git repos and clone them. If you're distributing private keys to hundreds of clients, why do you ever need to do this? Assuming you do (and I'm just missing something), then I see your point but if you ensure they're encrypted with the user's pass phrase and you have a procedure for ensuring their permissions are set to 0x600, what's the problem?
- toast0 4y ago
  Yeah, the trickiest part is getting new users to check in their keys to git if git is authenticated with ssh keys. And if you need sudo to install keys, you've gotta use autopw[1] or something similar, and at least autopw is single threaded, so if you've got more like thousands of servers, it's a bit messy. Smarter people than I setup full automation for this kind of stuff, but then you have to trust the automation. [1] https://github.com/jschauma/sshscan/blob/master/src/autopw https://github.com/jschauma/sshscan/blob/master/src/autopw
CameronNemo 4y ago
There is 0 reason to distribute SSH keys across machines. You can tell SSH to use LDAP to check a user's keys, trivially.

by xena 4y ago

toast0 4y ago

> It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts. On the other hand, if you knew what I'd need to do with the servers, I probably wouldn't need to do it. It's really hard to investigate problems without tools. Nobody is going to make a graph for how much cpu poorly configured docker/datadog are wasting rereading the same 1GB of json logs once or twice a minute, because nobody thought it would be meaningful, but watching top in one window, and running lsof at the right time to see what's being read, and whatever performance tools are hanging out to confirm.

AtlasBarfed 4y ago
Ah the dream of easy to administer, perfectly adapted fine grained access permissions that can predict the future of exactly what an authorized user can run and needs to do. It never works because of the lack of precog, ticket walls, escalation over policy requiring reviews, and other bureaucracy. What they end up replacing ssh with (teleport or aws ssm are the two I've been forced to use) are slow, spotty, crash frequently, and who knows what vulnerabilities they have. If all these companies trying to get rid of sshd want to slap their own daemon on to replace it, why not keep sshd and simply have a deamon that manages the keys and not throw out the entire tool chain? The most fun thing with teleport and aws-ssm: to use it you had to bring up a web page to authenticate. There goes any hope of automation at scale. Because in a world where you're supposed to treat servers like cattle, let's impose (and I mean impose) a regime that forces you to access them one at a time manually, especially in critical situations.

imiric 4y ago

> Distributing SSH keys across hundreds of machines is a nontrivial task. SSH certificates are a thing.

xena 4y ago
Now you have two problems
- arp242 4y ago
  You can probably solve those with some regexps.
- medellin 4y ago
  Please explain. I have deployed them successfully at both large and small companies without any significant issues
Spivak 4y ago
Also user directories are thing and you can instruct OpenSSH to use any command you like to fetch a user’s authorized_keys.
wkat4242 4y ago
Yes but the OP was referring to SSH keys being eschewed by the enterprise. I assume they didn't consider certificates part of this category. For enterprise the certificates are a far better option. Or s full blown IDP of course.

dreamcompiler 4y ago

> Distributing SSH keys across hundreds of machines is a nontrivial task. In what sense? If you're distributing public keys to hundreds of servers, go nuts. Put them in git repos and clone them. If you're distributing private keys to hundreds of clients, why do you ever need to do this? Assuming you do (and I'm just missing something), then I see your point but if you ensure they're encrypted with the user's pass phrase and you have a procedure for ensuring their permissions are set to 0x600, what's the problem?

toast0 4y ago
Yeah, the trickiest part is getting new users to check in their keys to git if git is authenticated with ssh keys. And if you need sudo to install keys, you've gotta use autopw[1] or something similar, and at least autopw is single threaded, so if you've got more like thousands of servers, it's a bit messy. Smarter people than I setup full automation for this kind of stuff, but then you have to trust the automation. [1] https://github.com/jschauma/sshscan/blob/master/src/autopw https://github.com/jschauma/sshscan/blob/master/src/autopw

CameronNemo 4y ago

There is 0 reason to distribute SSH keys across machines. You can tell SSH to use LDAP to check a user's keys, trivially.