7 ms·

Stealing unencrypted SSH-agent keys from memory

81 points by lelf 12y ago

yock 12y ago
I've upvoted this because I want to see what really smart people have to say about it. That said, I've always been of mind that if someone gains unauthorized root access to my box, then nothing on it can ever be trusted again. This includes things in memory. In other words, this feels obvious to me, once pointed out. Is there something novel about this that I'm missing?
- hemancuso 12y ago
  No. If you have the ability to run code as root it's game over. Even if the agent re-encrypted the key in memory with its own temporary key it's just a layer of obfuscation that can easily be overcome.
- tlrobinson 12y ago
  I agree with you, but in case you missed the author's rationale: "However, this causes the attacker to have to wait for the target to type in their passphrase. This might be hours, days, or weeks, depending on how often the target logs out. This is why obtaining the SSH key from memory is vital to pivoting to other machines in a speedy fashion." I'm not sure how you would solve this problem while providing the functionality ssh-agent provides, aside from perhaps a HSM or something.
  - agwa 12y ago
    Anyone keeping SSH keys in an ssh-agent process for weeks on end is doing it wrong. You can use the -t option to ssh-agent to specify a maximum lifetime for identities added to the agent. I would set it to less than a day.
    - zobzu 12y ago
      if you unlock it as soon as it expires im not sure what that gives you. all the attacker needs anyway in that scheme is 24h, its not a big difference.
  - MichaelGG 12y ago
    Make sure nothing can run as root? Perhaps by using a VM system. You could run ssh-agent on another machine and make key requests over an internal network. Also you can get a cheap "HSM" by using a smartcard.
  - peterwwillis 12y ago
    The problem of a root user inspecting the memory of arbitrary other users is solved with MAC like SELinux. https://security.stackexchange.com/questions/7801/keeping-secrets-from-root-on-linux https://security.stackexchange.com/questions/7801/keeping-se... But if you can exploit a specific user's program, you can (usually) inspect the memory of other programs managed by that user. You can actually create security policies so secure that a user who exploits the sshd program can't read from the same user's ssh-agent process memory, but that's not practical for most people.
- agwa 12y ago
  It's not that novel. If an attacker gets root it's game over. That said, the article is a good reminder of the risk of keeping decrypted keys in a long-running ssh-agent process.
- deleted 12y ago
  [deleted]
- zobzu 12y ago
  nothing's novel - but consider this for example: - if its a server and you copy your private key onto it its likely to be compromised one day and the key reused even thus it was encrypted.. since its decrypted in memory when used. plus keystrokes could be captured. if you use this key elsewhere, all other servers are also compromised now.. - you can keep the key in a hardware token, then even if a laptop is remotely compromised your key - and thus servers you're not currently accessing - are safe. its also a little harder to hijack a connection or silently ask the key (generally its not silent with a token)
e40 12y ago
So, as someone needing to run unattended scripts across a wide variety of hosts, what's the best practice for managing SSH keys? I've been using ssh-agent. Is there a better way?
- agwa 12y ago
  Make sure that the SSH keys are tightly restricted in the authorized_keys file so they can only run the commands that need to be run unattended. If possible, also restrict them by source IP address. Here's an example authorized_keys file that I have in production for doing unattended backups: command="/usr/local/backups/backup_server /etc/snapshot_backup_list",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,from="10.70.0.0/16" ssh-rsa AAAAB... I wouldn't worry unduly about protecting the keys themselves - since they need to be accessible for unattended operation, there's not much you can do to prevent them from being accessible to an intruder.
  - e40 12y ago
    This is hard, since there is such a large number of them. It would be very painful to maintain this list. For a while I had the keys expire in ssh-agent, but that didn't work because you can't extend the expiration before you run a set of unattended commands. That is, if it's 11.5 hours into a 12 hour expiration, and I need to run commands that will span the expiration, then I'm screwed, since 1) I can't query when the keys will expire, and 2) I can't extend it another 12 hours (from now, not from the expiration). I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).
    - agwa 12y ago
      That sounds like a bit of a quandary. One thing I'm not clear on - if these run unattended, how do you enter the passphrase? ssh-agent isn't insecure, it's just not magic. If an attacker gets root access to a box, they can examine the system's memory, and the memory of ssh-agent necessarily contains the unencrypted private keys. Edit: if you're really concerned about private keys being exfiltrated, you can always use a smartcard (the OpenPGP card[1] in a Gemalto USB Shell Token[2] works well with SSH). But if the smartcard is online all the time, then an intruder can always simply use the smartcard to SSH wherever they want, even if they can't actually get the private key itself. [1] http://shop.kernelconcepts.de/product_info.php?products_id=42&osCsid=bdf5854d9214419637adcf0c5215e6fe http://shop.kernelconcepts.de/product_info.php?products_id=4... [2] http://shop.kernelconcepts.de/product_info.php?products_id=119&osCsid=bdf5854d9214419637adcf0c5215e6fe http://shop.kernelconcepts.de/product_info.php?products_id=1...
ambrice 12y ago
So, is there a big advantage to running ssh-agent vs unencrypted 0400 keys in your ~/.ssh directory?
- agwa 12y ago
  Yes. ssh-agent prevents non-root users, including the user running ssh-agent[1] from being able to extract the private key. That's much more secure than storing unencrypted keys in ~/.ssh. [1] ssh-agent uses some tricks, like making itself setgid, and/or using prctl(PR_SET_DUMPABLE, 0) on Linux to make its memory undumpable.
  - 0x0 12y ago
    And on OSX, it integrates with the keychain as well!
    - void-star 12y ago
      You're better off not storing your passphrase in the keychain and just typing it in that one first time, though. But it's still marginally better than an unencrypted private key for automation if you need a 100% unattended setup.
      - lloeki 12y ago
        "In the keychain" doesn't necessarily mean in the login keychain. You can create as many user keychains as you want, with different passwords. Also, add the keychain app to your menubar (via preferences) to get a quick way to manually lock one or all keychains (and alternatively, the whole computer).
  - Canada 12y ago
    That won't prevent your key from being stolen if you leave it on a VM controlled by someone else. Your key belongs on your personal device, and agent forwarding enables you to behave as if your key is everywhere you need it to be.
ipsin 12y ago
The technique of using gdb to attach to ssh is pretty old. Here's an example from 2005, which was presented (iirc) at Defcon as well: http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-boileau.pdf http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-boi... In that case, SSH-Jack would just piggyback on existing (user-level) ssh connections, which is also pretty serious, though that's not as exciting as stealing keys.
- zobzu 12y ago
  personally i find it more exciting than steal the key because harder :) then again this as been done since the beginning of times sadly, or excitingly, i dont know :) any tool that promotes awareness of this inherent issue is good anyway IMO
- void-star 12y ago
  Yep. The general technique of scraping memory for private keys and certificates a pretty well-known and established. Particularly in forensics and malware circles. I don't point it out to engage in the "who was first" thing. But to point out that this is very much an applied attack in the real world. Real attackers (includes "forensics analysts", incase you don't consider them "attackers" too) have been using this technique in malware as well as countermeasures/investigations for quite a while, now. See: http://www.trapkit.de/research/sslkeyfinder/ http://www.trapkit.de/research/sslkeyfinder/ https://github.com/emonti/yara-ruby/blob/master/samples/sslkeyfinder https://github.com/emonti/yara-ruby/blob/master/samples/sslk... http://volatility-labs.blogspot.com/2013/05/movp-ii-21-rsa-private-keys-and.html http://volatility-labs.blogspot.com/2013/05/movp-ii-21-rsa-p... EDIT: actually a much earlier discussion is from '98 by none other than Shamir https://www.cs.jhu.edu/~astubble/600.412/s-c-papers/keys2.pdf https://www.cs.jhu.edu/~astubble/600.412/s-c-papers/keys2.pd... [PDF]
Canada 12y ago
This is exactly why SSH has agent forwarding. Do not copy your private key to other systems. ssh -A is your friend.
- m-app 12y ago
  Well, it prevents your private key from being compromised but an attacker with root on the intermediate box can still authenticate to other servers using your SSH agent socket. If you really want to do secure SSH hopping, use the ProxyCommand directive with `netcat` in your SSH config.[1] [1]: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#ProxyCommand_with_Netcat http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Ju...
  - Canada 12y ago
    Nice. Thanks for sharing
nullc 12y ago
I wish SSH had a narrow kind of delegation capability... E.g. if instead of ssh-agent having access to the key, it used the key to sign a tuple like {time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and used the ephemeral key to actually authenticate to hosts— but they'd only accept it from the listed host ip set and during the specified time limit. Obviously if the attacker has access long enough they'll capture the passphrase and private key eventually, but it would be nice to narrow the window. Right now the best you can probably do is use a gpg smartcard with the ssh-agent/gpg-agent shim stuff, (random tutorial here: https://github.com/herlo/ssh-gpg-smartcard-config/blob/master/YubiKey_NEO.rst https://github.com/herlo/ssh-gpg-smartcard-config/blob/maste...)
- cnvogel 12y ago
  Your scheme could probably be implemented using certificate based SSH: Your users log onto a CA which gives out user-certificates with a specified lifetime, they copy that to their desktop-machine(s), or maybe add it to a ssh-agent like intermediate key-storage and can do their days work. Tomorrow, they'll have to get new certs (automatically, after authenticating to the CA using appropriate means). Here's a tutorial showing cert-based ssh auth: http://neocri.me/documentation/using-ssh-certificate-authentication/ http://neocri.me/documentation/using-ssh-certificate-authent... Obviously with secure key storage (smart-card) prohibiting accidental loss of the private key, and a working cert-revocation infrastructure it's not necessary in the first place.
  - icebraining 12y ago
    That's essentially Mozilla Persona, but for SSH instead of the web. The people at Mozilla are in fact already discussing such possibilities: https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-non-web/ https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-no...
- FedRegister 12y ago
  >I wish SSH had a narrow kind of delegation capability... E.g. if instead of ssh-agent having access to the key, it used the key to sign a tuple like {time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and used the ephemeral key to actually authenticate to hosts— but they'd only accept it from the listed host ip set and during the specified time limit. So Kerberos? You can do that with Kerberos and the GSS-API in OpenSSH already.

81 points by lelf 12y ago

yock 12y ago

I've upvoted this because I want to see what really smart people have to say about it. That said, I've always been of mind that if someone gains unauthorized root access to my box, then nothing on it can ever be trusted again. This includes things in memory. In other words, this feels obvious to me, once pointed out. Is there something novel about this that I'm missing?

hemancuso 12y ago
No. If you have the ability to run code as root it's game over. Even if the agent re-encrypted the key in memory with its own temporary key it's just a layer of obfuscation that can easily be overcome.
tlrobinson 12y ago
I agree with you, but in case you missed the author's rationale: "However, this causes the attacker to have to wait for the target to type in their passphrase. This might be hours, days, or weeks, depending on how often the target logs out. This is why obtaining the SSH key from memory is vital to pivoting to other machines in a speedy fashion." I'm not sure how you would solve this problem while providing the functionality ssh-agent provides, aside from perhaps a HSM or something.
- agwa 12y ago
  Anyone keeping SSH keys in an ssh-agent process for weeks on end is doing it wrong. You can use the -t option to ssh-agent to specify a maximum lifetime for identities added to the agent. I would set it to less than a day.
  - zobzu 12y ago
    if you unlock it as soon as it expires im not sure what that gives you. all the attacker needs anyway in that scheme is 24h, its not a big difference.
- MichaelGG 12y ago
  Make sure nothing can run as root? Perhaps by using a VM system. You could run ssh-agent on another machine and make key requests over an internal network. Also you can get a cheap "HSM" by using a smartcard.
- peterwwillis 12y ago
  The problem of a root user inspecting the memory of arbitrary other users is solved with MAC like SELinux. https://security.stackexchange.com/questions/7801/keeping-secrets-from-root-on-linux https://security.stackexchange.com/questions/7801/keeping-se... But if you can exploit a specific user's program, you can (usually) inspect the memory of other programs managed by that user. You can actually create security policies so secure that a user who exploits the sshd program can't read from the same user's ssh-agent process memory, but that's not practical for most people.
agwa 12y ago
It's not that novel. If an attacker gets root it's game over. That said, the article is a good reminder of the risk of keeping decrypted keys in a long-running ssh-agent process.
deleted 12y ago
[deleted]
zobzu 12y ago
nothing's novel - but consider this for example: - if its a server and you copy your private key onto it its likely to be compromised one day and the key reused even thus it was encrypted.. since its decrypted in memory when used. plus keystrokes could be captured. if you use this key elsewhere, all other servers are also compromised now.. - you can keep the key in a hardware token, then even if a laptop is remotely compromised your key - and thus servers you're not currently accessing - are safe. its also a little harder to hijack a connection or silently ask the key (generally its not silent with a token)

e40 12y ago

So, as someone needing to run unattended scripts across a wide variety of hosts, what's the best practice for managing SSH keys? I've been using ssh-agent. Is there a better way?

agwa 12y ago
Make sure that the SSH keys are tightly restricted in the authorized_keys file so they can only run the commands that need to be run unattended. If possible, also restrict them by source IP address. Here's an example authorized_keys file that I have in production for doing unattended backups: command="/usr/local/backups/backup_server /etc/snapshot_backup_list",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,from="10.70.0.0/16" ssh-rsa AAAAB... I wouldn't worry unduly about protecting the keys themselves - since they need to be accessible for unattended operation, there's not much you can do to prevent them from being accessible to an intruder.
- e40 12y ago
  This is hard, since there is such a large number of them. It would be very painful to maintain this list. For a while I had the keys expire in ssh-agent, but that didn't work because you can't extend the expiration before you run a set of unattended commands. That is, if it's 11.5 hours into a 12 hour expiration, and I need to run commands that will span the expiration, then I'm screwed, since 1) I can't query when the keys will expire, and 2) I can't extend it another 12 hours (from now, not from the expiration). I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).
  - agwa 12y ago
    That sounds like a bit of a quandary. One thing I'm not clear on - if these run unattended, how do you enter the passphrase? ssh-agent isn't insecure, it's just not magic. If an attacker gets root access to a box, they can examine the system's memory, and the memory of ssh-agent necessarily contains the unencrypted private keys. Edit: if you're really concerned about private keys being exfiltrated, you can always use a smartcard (the OpenPGP card[1] in a Gemalto USB Shell Token[2] works well with SSH). But if the smartcard is online all the time, then an intruder can always simply use the smartcard to SSH wherever they want, even if they can't actually get the private key itself. [1] http://shop.kernelconcepts.de/product_info.php?products_id=42&osCsid=bdf5854d9214419637adcf0c5215e6fe http://shop.kernelconcepts.de/product_info.php?products_id=4... [2] http://shop.kernelconcepts.de/product_info.php?products_id=119&osCsid=bdf5854d9214419637adcf0c5215e6fe http://shop.kernelconcepts.de/product_info.php?products_id=1...

ambrice 12y ago

So, is there a big advantage to running ssh-agent vs unencrypted 0400 keys in your ~/.ssh directory?

agwa 12y ago
Yes. ssh-agent prevents non-root users, including the user running ssh-agent[1] from being able to extract the private key. That's much more secure than storing unencrypted keys in ~/.ssh. [1] ssh-agent uses some tricks, like making itself setgid, and/or using prctl(PR_SET_DUMPABLE, 0) on Linux to make its memory undumpable.
- 0x0 12y ago
  And on OSX, it integrates with the keychain as well!
  - void-star 12y ago
    You're better off not storing your passphrase in the keychain and just typing it in that one first time, though. But it's still marginally better than an unencrypted private key for automation if you need a 100% unattended setup.
    - lloeki 12y ago
      "In the keychain" doesn't necessarily mean in the login keychain. You can create as many user keychains as you want, with different passwords. Also, add the keychain app to your menubar (via preferences) to get a quick way to manually lock one or all keychains (and alternatively, the whole computer).
- Canada 12y ago
  That won't prevent your key from being stolen if you leave it on a VM controlled by someone else. Your key belongs on your personal device, and agent forwarding enables you to behave as if your key is everywhere you need it to be.

ipsin 12y ago

The technique of using gdb to attach to ssh is pretty old. Here's an example from 2005, which was presented (iirc) at Defcon as well: http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-boileau.pdf http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-boi... In that case, SSH-Jack would just piggyback on existing (user-level) ssh connections, which is also pretty serious, though that's not as exciting as stealing keys.

zobzu 12y ago
personally i find it more exciting than steal the key because harder :) then again this as been done since the beginning of times sadly, or excitingly, i dont know :) any tool that promotes awareness of this inherent issue is good anyway IMO
void-star 12y ago
Yep. The general technique of scraping memory for private keys and certificates a pretty well-known and established. Particularly in forensics and malware circles. I don't point it out to engage in the "who was first" thing. But to point out that this is very much an applied attack in the real world. Real attackers (includes "forensics analysts", incase you don't consider them "attackers" too) have been using this technique in malware as well as countermeasures/investigations for quite a while, now. See: http://www.trapkit.de/research/sslkeyfinder/ http://www.trapkit.de/research/sslkeyfinder/ https://github.com/emonti/yara-ruby/blob/master/samples/sslkeyfinder https://github.com/emonti/yara-ruby/blob/master/samples/sslk... http://volatility-labs.blogspot.com/2013/05/movp-ii-21-rsa-private-keys-and.html http://volatility-labs.blogspot.com/2013/05/movp-ii-21-rsa-p... EDIT: actually a much earlier discussion is from '98 by none other than Shamir https://www.cs.jhu.edu/~astubble/600.412/s-c-papers/keys2.pdf https://www.cs.jhu.edu/~astubble/600.412/s-c-papers/keys2.pd... [PDF]

Canada 12y ago

This is exactly why SSH has agent forwarding. Do not copy your private key to other systems. ssh -A is your friend.

m-app 12y ago
Well, it prevents your private key from being compromised but an attacker with root on the intermediate box can still authenticate to other servers using your SSH agent socket. If you really want to do secure SSH hopping, use the ProxyCommand directive with `netcat` in your SSH config.[1] [1]: http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#ProxyCommand_with_Netcat http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Ju...
- Canada 12y ago
  Nice. Thanks for sharing

nullc 12y ago

I wish SSH had a narrow kind of delegation capability... E.g. if instead of ssh-agent having access to the key, it used the key to sign a tuple like {time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and used the ephemeral key to actually authenticate to hosts— but they'd only accept it from the listed host ip set and during the specified time limit. Obviously if the attacker has access long enough they'll capture the passphrase and private key eventually, but it would be nice to narrow the window. Right now the best you can probably do is use a gpg smartcard with the ssh-agent/gpg-agent shim stuff, (random tutorial here: https://github.com/herlo/ssh-gpg-smartcard-config/blob/master/YubiKey_NEO.rst https://github.com/herlo/ssh-gpg-smartcard-config/blob/maste...)

cnvogel 12y ago
Your scheme could probably be implemented using certificate based SSH: Your users log onto a CA which gives out user-certificates with a specified lifetime, they copy that to their desktop-machine(s), or maybe add it to a ssh-agent like intermediate key-storage and can do their days work. Tomorrow, they'll have to get new certs (automatically, after authenticating to the CA using appropriate means). Here's a tutorial showing cert-based ssh auth: http://neocri.me/documentation/using-ssh-certificate-authentication/ http://neocri.me/documentation/using-ssh-certificate-authent... Obviously with secure key storage (smart-card) prohibiting accidental loss of the private key, and a working cert-revocation infrastructure it's not necessary in the first place.
- icebraining 12y ago
  That's essentially Mozilla Persona, but for SSH instead of the web. The people at Mozilla are in fact already discussing such possibilities: https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-non-web/ https://hacks.mozilla.org/2013/04/mozilla-persona-for-the-no...
FedRegister 12y ago
>I wish SSH had a narrow kind of delegation capability... E.g. if instead of ssh-agent having access to the key, it used the key to sign a tuple like {time-limit, [host-ip], ephemeral_pubkey}, then forgot the private key, and used the ephemeral key to actually authenticate to hosts— but they'd only accept it from the listed host ip set and during the specified time limit. So Kerberos? You can do that with Kerberos and the GSS-API in OpenSSH already.