5 ms·

Except the enterprise security industry is waging war on ssh keys and ssh/sshd.

by AtlasBarfed 4y ago

themikesanto 4y ago
Interesting. Could you expand on that, or point me towards some related reading? Thanks!
arp242 4y ago
What's wrong with ssh keys? Or ssh/sshd for that matter?
- xena 4y ago
  The main problem with SSH keys is that they are good at small scales, but at larger scales the abstractions start to be really leaky. Distributing SSH keys across hundreds of machines is a nontrivial task. Not to mention the lack of audit logging in opensshd and the lack of session recording (these requirements suck but acronym compliance means you need them anyways). It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts.
  - toast0 4y ago
    > It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts. On the other hand, if you knew what I'd need to do with the servers, I probably wouldn't need to do it. It's really hard to investigate problems without tools. Nobody is going to make a graph for how much cpu poorly configured docker/datadog are wasting rereading the same 1GB of json logs once or twice a minute, because nobody thought it would be meaningful, but watching top in one window, and running lsof at the right time to see what's being read, and whatever performance tools are hanging out to confirm.
    - AtlasBarfed 4y ago
      Ah the dream of easy to administer, perfectly adapted fine grained access permissions that can predict the future of exactly what an authorized user can run and needs to do. It never works because of the lack of precog, ticket walls, escalation over policy requiring reviews, and other bureaucracy. What they end up replacing ssh with (teleport or aws ssm are the two I've been forced to use) are slow, spotty, crash frequently, and who knows what vulnerabilities they have. If all these companies trying to get rid of sshd want to slap their own daemon on to replace it, why not keep sshd and simply have a deamon that manages the keys and not throw out the entire tool chain? The most fun thing with teleport and aws-ssm: to use it you had to bring up a web page to authenticate. There goes any hope of automation at scale. Because in a world where you're supposed to treat servers like cattle, let's impose (and I mean impose) a regime that forces you to access them one at a time manually, especially in critical situations.
  - imiric 4y ago
    > Distributing SSH keys across hundreds of machines is a nontrivial task. SSH certificates are a thing.
    - xena 4y ago
      Now you have two problems
      - arp242 4y ago
        You can probably solve those with some regexps.
      - medellin 4y ago
        Please explain. I have deployed them successfully at both large and small companies without any significant issues
    - Spivak 4y ago
      Also user directories are thing and you can instruct OpenSSH to use any command you like to fetch a user’s authorized_keys.
    - wkat4242 4y ago
      Yes but the OP was referring to SSH keys being eschewed by the enterprise. I assume they didn't consider certificates part of this category. For enterprise the certificates are a far better option. Or s full blown IDP of course.
  - dreamcompiler 4y ago
    > Distributing SSH keys across hundreds of machines is a nontrivial task. In what sense? If you're distributing public keys to hundreds of servers, go nuts. Put them in git repos and clone them. If you're distributing private keys to hundreds of clients, why do you ever need to do this? Assuming you do (and I'm just missing something), then I see your point but if you ensure they're encrypted with the user's pass phrase and you have a procedure for ensuring their permissions are set to 0x600, what's the problem?
    - toast0 4y ago
      Yeah, the trickiest part is getting new users to check in their keys to git if git is authenticated with ssh keys. And if you need sudo to install keys, you've gotta use autopw[1] or something similar, and at least autopw is single threaded, so if you've got more like thousands of servers, it's a bit messy. Smarter people than I setup full automation for this kind of stuff, but then you have to trust the automation. [1] https://github.com/jschauma/sshscan/blob/master/src/autopw https://github.com/jschauma/sshscan/blob/master/src/autopw
  - CameronNemo 4y ago
    There is 0 reason to distribute SSH keys across machines. You can tell SSH to use LDAP to check a user's keys, trivially.
wkat4242 4y ago
In the enterprise key management can become a bitch really quickly. Good luck updating 10.000 servers every time an admin leaves. For them ssh certificates are much better because you can simply revoke them, and new certs don't have to be added to each server. At home I still use keys though, but as an enterprise admin I see the issues. Even at home it's kinda annoying having to add a new yubikey to 20 servers.
- hobobaggins 4y ago
  Unless you use something like Userify. It's like my secret tool that no one knows about..
- janc_ 4y ago
  You could use AuthorizedKeysCommand or some other method to manage your keys in one central place.
  - wkat4242 4y ago
    You can, but you still need to set up a secure infrastructure to manage all the requests going back and forth. And most enterprises already have a completely functional and secured PKI, usually with the whole HSM deal and everything. It's plug and play and works with your existing infra.
- still_grokking 4y ago
  > In the enterprise key management can become a bitch really quickly. Good luck updating 10.000 servers every time an admin leaves. Can't "jump host" solve this? (Not an admin, so please excuse if the question is stupid).
  - xena 4y ago
    No
- account-5 4y ago
  You have 20 servers at home? What's you electrical bill like?
StreamBright 4y ago
It is not as much the enterprise security industry as it is better security for all. Problem with the current implementation with ssh keys is that there are non-trivial problems with it. For example how do you revoke a key? It is not possible in a central AAA service to revoke access for a user like it is possible for other kind of auth systems. Ssh and ssh keys were designed in an era where security was not really the primary concern. I think Facebook DevOps team had some detailed article how they manage ssh access to fb production with keys.
- CameronNemo 4y ago
  There is so many things wrong with your comment it is not even worth explaining.

Except the enterprise security industry is waging war on ssh keys and ssh/sshd.

themikesanto 4y ago

Interesting. Could you expand on that, or point me towards some related reading? Thanks!

arp242 4y ago

What's wrong with ssh keys? Or ssh/sshd for that matter?

xena 4y ago
The main problem with SSH keys is that they are good at small scales, but at larger scales the abstractions start to be really leaky. Distributing SSH keys across hundreds of machines is a nontrivial task. Not to mention the lack of audit logging in opensshd and the lack of session recording (these requirements suck but acronym compliance means you need them anyways). It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts.
- toast0 4y ago
  > It's also kind of the wrong tool for the job anyways. I'd love to see tools that are more built to purpose for what you are _actually doing with the servers_ instead of giving people a REPL and telling them to go nuts. On the other hand, if you knew what I'd need to do with the servers, I probably wouldn't need to do it. It's really hard to investigate problems without tools. Nobody is going to make a graph for how much cpu poorly configured docker/datadog are wasting rereading the same 1GB of json logs once or twice a minute, because nobody thought it would be meaningful, but watching top in one window, and running lsof at the right time to see what's being read, and whatever performance tools are hanging out to confirm.
  - AtlasBarfed 4y ago
    Ah the dream of easy to administer, perfectly adapted fine grained access permissions that can predict the future of exactly what an authorized user can run and needs to do. It never works because of the lack of precog, ticket walls, escalation over policy requiring reviews, and other bureaucracy. What they end up replacing ssh with (teleport or aws ssm are the two I've been forced to use) are slow, spotty, crash frequently, and who knows what vulnerabilities they have. If all these companies trying to get rid of sshd want to slap their own daemon on to replace it, why not keep sshd and simply have a deamon that manages the keys and not throw out the entire tool chain? The most fun thing with teleport and aws-ssm: to use it you had to bring up a web page to authenticate. There goes any hope of automation at scale. Because in a world where you're supposed to treat servers like cattle, let's impose (and I mean impose) a regime that forces you to access them one at a time manually, especially in critical situations.
- imiric 4y ago
  > Distributing SSH keys across hundreds of machines is a nontrivial task. SSH certificates are a thing.
  - xena 4y ago
    Now you have two problems
    - arp242 4y ago
      You can probably solve those with some regexps.
    - medellin 4y ago
      Please explain. I have deployed them successfully at both large and small companies without any significant issues
  - Spivak 4y ago
    Also user directories are thing and you can instruct OpenSSH to use any command you like to fetch a user’s authorized_keys.
  - wkat4242 4y ago
    Yes but the OP was referring to SSH keys being eschewed by the enterprise. I assume they didn't consider certificates part of this category. For enterprise the certificates are a far better option. Or s full blown IDP of course.
- dreamcompiler 4y ago
  > Distributing SSH keys across hundreds of machines is a nontrivial task. In what sense? If you're distributing public keys to hundreds of servers, go nuts. Put them in git repos and clone them. If you're distributing private keys to hundreds of clients, why do you ever need to do this? Assuming you do (and I'm just missing something), then I see your point but if you ensure they're encrypted with the user's pass phrase and you have a procedure for ensuring their permissions are set to 0x600, what's the problem?
  - toast0 4y ago
    Yeah, the trickiest part is getting new users to check in their keys to git if git is authenticated with ssh keys. And if you need sudo to install keys, you've gotta use autopw[1] or something similar, and at least autopw is single threaded, so if you've got more like thousands of servers, it's a bit messy. Smarter people than I setup full automation for this kind of stuff, but then you have to trust the automation. [1] https://github.com/jschauma/sshscan/blob/master/src/autopw https://github.com/jschauma/sshscan/blob/master/src/autopw
- CameronNemo 4y ago
  There is 0 reason to distribute SSH keys across machines. You can tell SSH to use LDAP to check a user's keys, trivially.

wkat4242 4y ago

In the enterprise key management can become a bitch really quickly. Good luck updating 10.000 servers every time an admin leaves. For them ssh certificates are much better because you can simply revoke them, and new certs don't have to be added to each server. At home I still use keys though, but as an enterprise admin I see the issues. Even at home it's kinda annoying having to add a new yubikey to 20 servers.

hobobaggins 4y ago
Unless you use something like Userify. It's like my secret tool that no one knows about..
janc_ 4y ago
You could use AuthorizedKeysCommand or some other method to manage your keys in one central place.
- wkat4242 4y ago
  You can, but you still need to set up a secure infrastructure to manage all the requests going back and forth. And most enterprises already have a completely functional and secured PKI, usually with the whole HSM deal and everything. It's plug and play and works with your existing infra.
still_grokking 4y ago
> In the enterprise key management can become a bitch really quickly. Good luck updating 10.000 servers every time an admin leaves. Can't "jump host" solve this? (Not an admin, so please excuse if the question is stupid).
- xena 4y ago
  No
account-5 4y ago
You have 20 servers at home? What's you electrical bill like?

StreamBright 4y ago

It is not as much the enterprise security industry as it is better security for all. Problem with the current implementation with ssh keys is that there are non-trivial problems with it. For example how do you revoke a key? It is not possible in a central AAA service to revoke access for a user like it is possible for other kind of auth systems. Ssh and ssh keys were designed in an era where security was not really the primary concern. I think Facebook DevOps team had some detailed article how they manage ssh access to fb production with keys.

CameronNemo 4y ago
There is so many things wrong with your comment it is not even worth explaining.