13 ms·

SSH Agent Explained

393 points by sourishkrout 6y ago

gbin 6y ago
Nit: The TL;DR is what is called an introduction.
- zymhan 6y ago
  Yeah that is an odd choice of words.
- AnonC 6y ago
  I don’t think it’s a nit. It’s polite and proper feedback on the misuse of a term. The TL;DR here didn’t really convey anything useful from the rest of the post. So it ought to be rightly called “Introduction”, as you said.
- skookum-skuad 6y ago
  Yep. TL;DR = extreme executive summary Intro = introduction Same number of letters, but different meanings It's not a big deal, but the armchair lawyer in me cringes and throws an OCD tantrum when spelling, grammar, semantics, and linguistic conventions are abused. "Weight at the busstop" or "u know what i mean thatswut im saying haha he even said it tiwce" :sigh: :meta-sigh:
nana- 6y ago
Thank you for the well done intro and the security risk implied with agent forwarding. I'd be curious how to actually sign with the ssh-agent as you mentioned it. I'd also like to know if when OS at login automatically decrypts the default private key, for later use e.g. pushing to github, what would be the risk? Would there be another preferred behaviour?
keithwinstein 6y ago
Thanks for this. Wanted to put in a pitch for Dima Kogan's more-secure way of doing ssh-agent forwarding: https://github.com/StanfordSNR/guardian-agent https://github.com/StanfordSNR/guardian-agent It works with SSH and Mosh. The basic idea is that before agreeing to a request, the principal or their agent should know (a) what machine is asking, (b) what remote machine they want to connect to, and (c) what command line they want to run on the principal's behalf. And the principal's authorization should then be limited to that context. The ssh-agent protocol doesn't give the agent any of that information; it's really just intended for local SSH-like processes to ask "please sign this blob so I can connect somewhere" without them having to see the plaintext private key. Forwarding that to untrusted remote machines isn't ideal. It turns out an agent can get access to this information and limit the scope of authorization in a mostly[1] backwards-compatible way, which is how guardian-agent works, but imo it would be preferable if a future version of the SSH protocol were designed more expressly for secure agent forwarding. [1] For (c), the remote server has to be OpenSSH because guardian-agent relies on a nonstandard extension.
- mmalone 6y ago
  YES. It would be so easy for OpenSSH to fix agent forwarding. Just need to limit authority and/or ask for consent for a particular action.
  - undecidabot 6y ago
    You can configure ssh-agent to ask for confirmation if you set the `-c` flag in ssh-add or by setting `AddKeysToAgent` to `confirm` in your ssh config [1]. Once set, authentication will require confirmation via a GUI dialog provided by the ssh-askpass command. However, it does not mention the command or process requesting for authentication. It works great on Linux, but I couldn't get it to work on macOS with the system keychain. [1] https://man.openbsd.org/ssh_config.5#AddKeysToAgent https://man.openbsd.org/ssh_config.5#AddKeysToAgent
    - mmalone 6y ago
      Recent macOS versions don't have `ssh-askpass`, and it's weirdly hard to add one. Since agent confirmation depends on askpass, I don't think there's an easy way to get this work on macOS. Aside from the missing context you mentioned, the other bigger problem with this approach is that agent confirmation is all-or-nothing: it turns on confirmation for local SSH connections in addition to forwarded connections. If you're using SSH a lot, having to confirm every connection is very annoying.
- inetknght 6y ago
  > The basic idea is that before agreeing to a request, the principal or their agent should know (a) what machine is asking, (b) what remote machine they want to connect to, and (c) what command line they want to run on the principal's behalf. And the principal's authorization should then be limited to that context. Holy snap this is exactly the reason that I've gone to great lengths to disable SSH Agents and askpass. Asking for a password without any context whatsoever of exactly which process is wanting it is a nightmare
- tialaramex 6y ago
  At first glance guardian-agent seems to be focused on a happy path where the remote systems are honest. Clearly agent forwarding attacks would mostly involve dishonest (or at least corrupt) remote systems and if guardian-agent specifically defeats that rather than just punting I don't see how in this summary.
oxplot 6y ago
Something that's skimmed over in the article but not addressed is: if the key pair isn't used for encryption, then how are session keys protected? The answer is: using the server's public key which is transmitted to client when establishing the connection. But then it's trivial to perform a person-in-the-middle attack and both observe and manipulate the plain text data by sending the client the attacker's public key. That's why it's crucial to retrieve host keys via secure channels and explicitly whitelist them on clients.
- tialaramex 6y ago
  "Protected" is an odd word to choose here because session keys are agreed long before we know who we're talking to. The approach of SSH (like modern TLS) is to create a secure channel between two participants and only then authenticate one or both participants by binding credentials to this secure channel. The article gets that upside down, which is understandable because most people seem to imagine that it'd be essential to figure out who you're talking to first and only then encrypt things, but actually the opposite is better. If you do Trust On First Use as many SSH users do, then you're correct that bad guys can interpose on that first connection if they happen to get lucky - but that's because they can authenticate as the "correct" server by presenting their own public key since you have no idea what the correct one looks like.
- bogomipz 6y ago
  >"The answer is: using the server's public key which is transmitted to client when establishing the connection." This was true in SSH v1 which is ancient but in modern times v2 uses DH and the the server's pub key is only used to sign the DH parameters.
- emj 6y ago
  You can use: "monster in the middle", mitm, if you want to keep it gender neutral.
volodymyrs 6y ago
What I am missing is why ever use ssh agent, including any re-enforced variations, if ProxyJump has none of the ssh agent issues and has little to no other disadvantages?
- ptman 6y ago
  ssh-agent is very useful, it's agent forwarding that is better handled with proxyjump. And my guess would be that forwarding came first and proxyjump only later.
- ncmncm 6y ago
  ProxyJump doesn't help with the use cases cited in the tl;dr^Wintroduction.
- pritambaral 6y ago
  1. Agent was designed to solve a larger problem; ProxyJump to solve a specialised subset of that problem. 2. The issues and disadvantages claimed with Agent Forwarding don't actually exist anymore: https://news.ycombinator.com/item?id=22753590 https://news.ycombinator.com/item?id=22753590
ncmncm 6y ago
Cool, didn't know about -J! I have been using dodgy port forwarding all this time. Now I just need to set up my Somu, and I will be living in 2020.
skookum-skuad 6y ago
I skip ssh-agent and use gpg-agent instead, which does both ssh and gpg agenting. Combined with monkeysphere, I store ssh private keys in gpg rather than in the encrypted ssh format.
- darkarmani 6y ago
  I haven't heard about monkeysphere in years. Is it still being maintained?
mkj 6y ago
SSH agent has other interesting possibilities too - it's an opaque channel from far server to your local machine, there must be various cool hacks you could do. Years ago I wrote something to copy a file from a remote machine to my desktop over ssh-agent, worked well back then. https://matt.ucc.asn.au/ssh-xfer/ https://matt.ucc.asn.au/ssh-xfer/
ganstyles 6y ago
I restart my computer pretty infrequently and everytime I do, I can't connect to various things. Inevitably I realize I didn't run ssh-add and after debugging, things work again once I have run this command. But I never really looked into the basics. Thanks to this article I realize why I need to do that every time. :)
bogomipz 6y ago
The author states: >"Later in the handshake process, a set of new, ephemeral and symmetric keys are generated and used to encrypt the SSH session traffic." This isn't correct. After the TCP handshake, both systems agree on a session key by using the Diffie-Hellman key exchange. It's only after establishing a session key that user authentication takes place. There's a re-keying event but that's not part of the initial handshake and only happens later when some threshold(time or blocks) has been reached.
rkeene2 6y ago
One thing I like to do with the SSH Agent is also forward my X.509 certificates by adding new opcodes to the SSH Agent protocol, then you can do stuff like PKCS#11 on the remote side with your local smartcard. [0] This gives you, among other things, passwordless but authenticated "sudo" capability (which is actually required by the DOD, though nobody does it). [0] http://cackey.rkeene.org/fossil/artifact/0d0e90bbfdee672c?ln=296-312%20481-482%20561-563%20573-576 http://cackey.rkeene.org/fossil/artifact/0d0e90bbfdee672c?ln...
- rkeene2 6y ago
  Forgot to include a link the PKCS#11 module which talks to the SSH Agent: [0] [0] https://chiselapp.com/user/rkeene/repository/ssh-agent-pkcs11/dir?ci=trunk https://chiselapp.com/user/rkeene/repository/ssh-agent-pkcs1...
- closeparen 6y ago
  Are you aware of any hardware tokens that support PKCS#11 and can be implemented affordably at small scale? The only ones I've found are Yubikey, but Yubikey's other protocols (OTP, U2F) are a lot simpler to implement, so I've never tried. At one point I bought a few literal smart cards and a USB reader, but not being an enterprise customer of any of the vendors, couldn't find the necessary drivers / passwords to get them working with PKCS#11.
  - count 6y ago
    It's not 'simple', but it's definitely doable with Yubikey: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PK... I've used it on macOS to use a PIV cert on a yubikey to ssh to remote linux servers. I have NOT used it to ssh-agent chain through, jump-box style though.
    - closeparen 6y ago
      If you're already bound to Yubikey, though, any advantage to PKCS#11 over OTP? Theoretically it's an open standard, but there seems to be only one real vendor.
vagab0nd 6y ago
I regularly ssh into boxes with varying IPs. Something like "ssh -i <key> <user>@<ip>". Every time I want to scp a file, I quit the ssh session, press UP, modify the ssh command into an scp one, execute then restore the ssh session. It's slow and annoying. Does anyone know of a way to use scp without hassle, once ssh session is established?
- beagle3 6y ago
  Use session sharing (ControlMaster setting). Then, don't quit your session - in another terminal on the same computer, run your scp command, and it will use the existing session (and its authentication), without need for -i. Alternatively, ssh-add is your friend. (But for security, make sure you do not forward ssh-agent unless you understand the risks)
- hylaride 6y ago
  You can setup an `~/.ssh/config` file to make it easier: Host * ControlMaster auto ControlPath ~/.ssh/ssh_mux_%h_%p_%r Host host1 HostName 1.2.3.4 LocalForward 45432 db.internal.net:5432 User user1 Host host2 HostName host2.whatever.com LocalForward 46432 db2.internal.net:5432 User user2 This way you can now scp/ssh to user1@1.2.3.4 with just this: scp file host1: ssh host1 It also setups up port forwarding. In that case I can connect a postgresql client to localhost 45432 and it'll forward (assuming the ssh host has network access to it) the tcp connection to the host db.internal.net . I do this to use GUI SQL clients to our back-end DB. The Control options setup ssh multiplexing. If you ssh into a host, subsequent connections piggy-back off of the initial connection. This is useful if you have 2fa and don't want to do 2 factor for every connection. There's much more you can do: https://linux.die.net/man/5/ssh_config https://linux.die.net/man/5/ssh_config
dooglius 6y ago
The problem is that this breaks with use of tmux or similar tools since the scope of the SSH session is often smaller than the scope of my doing something on a machine. I've moved to just having a limited-use key with access to dev machines and git on the hard drive of each machine I use, which covers the cases I care about.
lisper 6y ago
A pithy way of explaining all this: ssh-agent is exactly like a U2F token but implemented in software and using a slightly different protocol. But both do the same thing and serve the same purpose.
fgnewsom 6y ago
IF YOU SUPPORT THE LOCKDOWN FUCK YOU!

SSH Agent Explained

393 points by sourishkrout 6y ago

gbin 6y ago

Nit: The TL;DR is what is called an introduction.

zymhan 6y ago
Yeah that is an odd choice of words.
AnonC 6y ago
I don’t think it’s a nit. It’s polite and proper feedback on the misuse of a term. The TL;DR here didn’t really convey anything useful from the rest of the post. So it ought to be rightly called “Introduction”, as you said.
skookum-skuad 6y ago
Yep. TL;DR = extreme executive summary Intro = introduction Same number of letters, but different meanings It's not a big deal, but the armchair lawyer in me cringes and throws an OCD tantrum when spelling, grammar, semantics, and linguistic conventions are abused. "Weight at the busstop" or "u know what i mean thatswut im saying haha he even said it tiwce" :sigh: :meta-sigh:

nana- 6y ago

Thank you for the well done intro and the security risk implied with agent forwarding. I'd be curious how to actually sign with the ssh-agent as you mentioned it. I'd also like to know if when OS at login automatically decrypts the default private key, for later use e.g. pushing to github, what would be the risk? Would there be another preferred behaviour?

keithwinstein 6y ago

Thanks for this. Wanted to put in a pitch for Dima Kogan's more-secure way of doing ssh-agent forwarding: https://github.com/StanfordSNR/guardian-agent https://github.com/StanfordSNR/guardian-agent It works with SSH and Mosh. The basic idea is that before agreeing to a request, the principal or their agent should know (a) what machine is asking, (b) what remote machine they want to connect to, and (c) what command line they want to run on the principal's behalf. And the principal's authorization should then be limited to that context. The ssh-agent protocol doesn't give the agent any of that information; it's really just intended for local SSH-like processes to ask "please sign this blob so I can connect somewhere" without them having to see the plaintext private key. Forwarding that to untrusted remote machines isn't ideal. It turns out an agent can get access to this information and limit the scope of authorization in a mostly[1] backwards-compatible way, which is how guardian-agent works, but imo it would be preferable if a future version of the SSH protocol were designed more expressly for secure agent forwarding. [1] For (c), the remote server has to be OpenSSH because guardian-agent relies on a nonstandard extension.

mmalone 6y ago
YES. It would be so easy for OpenSSH to fix agent forwarding. Just need to limit authority and/or ask for consent for a particular action.
- undecidabot 6y ago
  You can configure ssh-agent to ask for confirmation if you set the `-c` flag in ssh-add or by setting `AddKeysToAgent` to `confirm` in your ssh config [1]. Once set, authentication will require confirmation via a GUI dialog provided by the ssh-askpass command. However, it does not mention the command or process requesting for authentication. It works great on Linux, but I couldn't get it to work on macOS with the system keychain. [1] https://man.openbsd.org/ssh_config.5#AddKeysToAgent https://man.openbsd.org/ssh_config.5#AddKeysToAgent
  - mmalone 6y ago
    Recent macOS versions don't have `ssh-askpass`, and it's weirdly hard to add one. Since agent confirmation depends on askpass, I don't think there's an easy way to get this work on macOS. Aside from the missing context you mentioned, the other bigger problem with this approach is that agent confirmation is all-or-nothing: it turns on confirmation for local SSH connections in addition to forwarded connections. If you're using SSH a lot, having to confirm every connection is very annoying.
inetknght 6y ago
> The basic idea is that before agreeing to a request, the principal or their agent should know (a) what machine is asking, (b) what remote machine they want to connect to, and (c) what command line they want to run on the principal's behalf. And the principal's authorization should then be limited to that context. Holy snap this is exactly the reason that I've gone to great lengths to disable SSH Agents and askpass. Asking for a password without any context whatsoever of exactly which process is wanting it is a nightmare
tialaramex 6y ago
At first glance guardian-agent seems to be focused on a happy path where the remote systems are honest. Clearly agent forwarding attacks would mostly involve dishonest (or at least corrupt) remote systems and if guardian-agent specifically defeats that rather than just punting I don't see how in this summary.

oxplot 6y ago

Something that's skimmed over in the article but not addressed is: if the key pair isn't used for encryption, then how are session keys protected? The answer is: using the server's public key which is transmitted to client when establishing the connection. But then it's trivial to perform a person-in-the-middle attack and both observe and manipulate the plain text data by sending the client the attacker's public key. That's why it's crucial to retrieve host keys via secure channels and explicitly whitelist them on clients.

tialaramex 6y ago
"Protected" is an odd word to choose here because session keys are agreed long before we know who we're talking to. The approach of SSH (like modern TLS) is to create a secure channel between two participants and only then authenticate one or both participants by binding credentials to this secure channel. The article gets that upside down, which is understandable because most people seem to imagine that it'd be essential to figure out who you're talking to first and only then encrypt things, but actually the opposite is better. If you do Trust On First Use as many SSH users do, then you're correct that bad guys can interpose on that first connection if they happen to get lucky - but that's because they can authenticate as the "correct" server by presenting their own public key since you have no idea what the correct one looks like.
bogomipz 6y ago
>"The answer is: using the server's public key which is transmitted to client when establishing the connection." This was true in SSH v1 which is ancient but in modern times v2 uses DH and the the server's pub key is only used to sign the DH parameters.
emj 6y ago
You can use: "monster in the middle", mitm, if you want to keep it gender neutral.

volodymyrs 6y ago

What I am missing is why ever use ssh agent, including any re-enforced variations, if ProxyJump has none of the ssh agent issues and has little to no other disadvantages?

ptman 6y ago
ssh-agent is very useful, it's agent forwarding that is better handled with proxyjump. And my guess would be that forwarding came first and proxyjump only later.
ncmncm 6y ago
ProxyJump doesn't help with the use cases cited in the tl;dr^Wintroduction.
pritambaral 6y ago
1. Agent was designed to solve a larger problem; ProxyJump to solve a specialised subset of that problem. 2. The issues and disadvantages claimed with Agent Forwarding don't actually exist anymore: https://news.ycombinator.com/item?id=22753590 https://news.ycombinator.com/item?id=22753590

ncmncm 6y ago

Cool, didn't know about -J! I have been using dodgy port forwarding all this time. Now I just need to set up my Somu, and I will be living in 2020.

skookum-skuad 6y ago

I skip ssh-agent and use gpg-agent instead, which does both ssh and gpg agenting. Combined with monkeysphere, I store ssh private keys in gpg rather than in the encrypted ssh format.

darkarmani 6y ago
I haven't heard about monkeysphere in years. Is it still being maintained?

mkj 6y ago

SSH agent has other interesting possibilities too - it's an opaque channel from far server to your local machine, there must be various cool hacks you could do. Years ago I wrote something to copy a file from a remote machine to my desktop over ssh-agent, worked well back then. https://matt.ucc.asn.au/ssh-xfer/ https://matt.ucc.asn.au/ssh-xfer/

ganstyles 6y ago

I restart my computer pretty infrequently and everytime I do, I can't connect to various things. Inevitably I realize I didn't run ssh-add and after debugging, things work again once I have run this command. But I never really looked into the basics. Thanks to this article I realize why I need to do that every time. :)

bogomipz 6y ago

The author states: >"Later in the handshake process, a set of new, ephemeral and symmetric keys are generated and used to encrypt the SSH session traffic." This isn't correct. After the TCP handshake, both systems agree on a session key by using the Diffie-Hellman key exchange. It's only after establishing a session key that user authentication takes place. There's a re-keying event but that's not part of the initial handshake and only happens later when some threshold(time or blocks) has been reached.

rkeene2 6y ago

One thing I like to do with the SSH Agent is also forward my X.509 certificates by adding new opcodes to the SSH Agent protocol, then you can do stuff like PKCS#11 on the remote side with your local smartcard. [0] This gives you, among other things, passwordless but authenticated "sudo" capability (which is actually required by the DOD, though nobody does it). [0] http://cackey.rkeene.org/fossil/artifact/0d0e90bbfdee672c?ln=296-312%20481-482%20561-563%20573-576 http://cackey.rkeene.org/fossil/artifact/0d0e90bbfdee672c?ln...

rkeene2 6y ago
Forgot to include a link the PKCS#11 module which talks to the SSH Agent: [0] [0] https://chiselapp.com/user/rkeene/repository/ssh-agent-pkcs11/dir?ci=trunk https://chiselapp.com/user/rkeene/repository/ssh-agent-pkcs1...
closeparen 6y ago
Are you aware of any hardware tokens that support PKCS#11 and can be implemented affordably at small scale? The only ones I've found are Yubikey, but Yubikey's other protocols (OTP, U2F) are a lot simpler to implement, so I've never tried. At one point I bought a few literal smart cards and a USB reader, but not being an enterprise customer of any of the vendors, couldn't find the necessary drivers / passwords to get them working with PKCS#11.
- count 6y ago
  It's not 'simple', but it's definitely doable with Yubikey: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PK... I've used it on macOS to use a PIV cert on a yubikey to ssh to remote linux servers. I have NOT used it to ssh-agent chain through, jump-box style though.
  - closeparen 6y ago
    If you're already bound to Yubikey, though, any advantage to PKCS#11 over OTP? Theoretically it's an open standard, but there seems to be only one real vendor.

vagab0nd 6y ago

I regularly ssh into boxes with varying IPs. Something like "ssh -i <key> <user>@<ip>". Every time I want to scp a file, I quit the ssh session, press UP, modify the ssh command into an scp one, execute then restore the ssh session. It's slow and annoying. Does anyone know of a way to use scp without hassle, once ssh session is established?

beagle3 6y ago
Use session sharing (ControlMaster setting). Then, don't quit your session - in another terminal on the same computer, run your scp command, and it will use the existing session (and its authentication), without need for -i. Alternatively, ssh-add is your friend. (But for security, make sure you do not forward ssh-agent unless you understand the risks)
hylaride 6y ago
You can setup an `~/.ssh/config` file to make it easier: Host * ControlMaster auto ControlPath ~/.ssh/ssh_mux_%h_%p_%r Host host1 HostName 1.2.3.4 LocalForward 45432 db.internal.net:5432 User user1 Host host2 HostName host2.whatever.com LocalForward 46432 db2.internal.net:5432 User user2 This way you can now scp/ssh to user1@1.2.3.4 with just this: scp file host1: ssh host1 It also setups up port forwarding. In that case I can connect a postgresql client to localhost 45432 and it'll forward (assuming the ssh host has network access to it) the tcp connection to the host db.internal.net . I do this to use GUI SQL clients to our back-end DB. The Control options setup ssh multiplexing. If you ssh into a host, subsequent connections piggy-back off of the initial connection. This is useful if you have 2fa and don't want to do 2 factor for every connection. There's much more you can do: https://linux.die.net/man/5/ssh_config https://linux.die.net/man/5/ssh_config

dooglius 6y ago

The problem is that this breaks with use of tmux or similar tools since the scope of the SSH session is often smaller than the scope of my doing something on a machine. I've moved to just having a limited-use key with access to dev machines and git on the hard drive of each machine I use, which covers the cases I care about.

lisper 6y ago

A pithy way of explaining all this: ssh-agent is exactly like a U2F token but implemented in software and using a slightly different protocol. But both do the same thing and serve the same purpose.

fgnewsom 6y ago

IF YOU SUPPORT THE LOCKDOWN FUCK YOU!