5 ms·

Even without this announcement, friends don't let friends forward their ssh agent. It essentially grants that machine access to your private keys. A RCE vulne

Even without this announcement, friends don't let friends forward their ssh agent. It essentially grants that machine access to your private keys. A RCE vulnerability is strictly worse than key exposure, but you probably shouldn't have been using it anyways.

starfallg 3y ago
Yup, and this is explained in detail in the ssh(1) man page. No-one should be using -A. Using a jump host via -J is the way to go.
- Arnavion 3y ago
  One use of -A, namely using the ssh server as a jumphost, is covered by -J. The general use of -A, namely doing operations on the ssh server that require keys from the ssh client, is not. If I'm on machine foo and I want to connect to bar.example.org and clone a git repo there from baz.example.org, and baz.example.org requires an identity key that is in foo's ssh agent, then -A is the only option.
  - GauntletWizard 3y ago
    ssh-askpass is a great tool in these situations. It intercedes and allows you to manually control all key usage attempts.
    - devnullbrain 3y ago
      Ah, a skillful execution of Cunningham's Law
    - Arnavion 3y ago
      I'm not sure what relevance ssh-askpass has to what I wrote.
      - GauntletWizard 3y ago
        Right, this recipe isn't well known. `ssh-add -c` will cause your ssh agent to pop up with ssh-askpass every time it does an authentication. This is fiddly; You need to have configured your ssh-agent right, and there's a slightly different version to use keychains on macos that's equivalent.
        Arnavion 3y ago
        Once again, I'm not sure what relevance this has to the discussion.
        pritambaral 3y ago
        Some people denounce all use of agent forwarding because, by default, ssh-agent doesn't confirm with the user before signing a request with the users private key. This means, if you ssh into a compromised/malicious host with your agent forwarded, malicious code on that machine can just silently ssh into other servers as you. This trick has been used in the past by blackhats to escalate from a compromised CI environment to full production takeover. GauntletWizard probably meant to respond to your parent in this thread.
  - gunapologist99 3y ago
    No, you might as well copy the keys over there for all the security you're getting (actually, that's safer given this RCE which compromises both sides, even given that there's no real way to shred the leaked key, but of course I'm not actually suggesting this! they're both really bad.) Your best option is too easy: just generate a new keypair on foo. Then you can populate baz.example.org with that public key instead of your own.
    - Arnavion 3y ago
      >No, you might as well copy the keys over there Only if the keys are insecure enough to be copied, ie not backed by an HSM, or backed by an HSM but marked exportable. >for all the security you're getting (actually, that's safer given this RCE which compromises both sides, even given that there's no real way to shred the leaked key, but of course I'm not actually suggesting this! they're both really bad.) Connecting to a malicious SSH server is already dangeorous with or without this vulnerability, and with or without agent forwarding. Eg a malicious server can modify the shell to emit VT codes to mess up your terminal. >Your best option is too easy: just generate a new keypair on foo. Sure, but this creates additional identities that baz.example.org must be taught to trust, which is not always desirable or even an option.
      - gunapologist99 3y ago
        > Only if the keys are insecure enough to be copied, ie not backed by an HSM, or backed by an HSM but marked exportable. Agent forwarding forwards the keys insecurely. If you can launch an agent-forwarded connection, then the HSM is already irrelevant. > "Connecting to a malicious SSH server" is already dangeorous with or without this vulnerability, and with or without agent forwarding. Eg a malicious server can modify the shell to emit VT codes to mess up your terminal. yes, but messing up your terminal is not an RCE (and you should be able to fix it with stty sane). It's just an annoyance and just tipped you off that obviously there's something wrong there. :) Obviously you should be checking your host keys etc and ideally you woud never accidentally connect to a malicious server. However, in the unlucky but perhaps inevitable event that you do connect to a malicious server, you shouldn't risk exposing your entire client workstation! (through this or another RCE) or your SSH private keys! (through normal agent forwarding operation) with agent forwarding. > Sure, but this creates additional identities that baz.example.org must be taught to trust, which is not always desirable or even an option. A key is not necessarily an identity. Most platforms permit more than one public key to be associated with a user, including github, gitlab, userify, etc. However, ideally you are correct -- this would result in a new user account that has limited access to only the things it really needs (principle of least privilege).
  - drdaeman 3y ago
    Another relatively popular use case is pam_ssh_agent_auth for passwordless-but-authenticated sudo. This, by definition, requires agent forwarding. (I don't use ssh-agent, I use gpg-agent with an SSH socket - I suppose I'm fine?)
- pritambaral 3y ago
  There are uses for `-A` that aren't just for jumping into hosts. I work on code in ephemeral containers and need to use git against remote git servers. If I had to use a key local to the container, I'd have to constantly add new keys to my git servers. Of course, I have secured by agent from abuse. I use gpg-agent as my ssh agent and make it visually confirm every use of my private key. I even set up those containers to use forwarded GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL} envs. So other people can work on the same container as me and git commits we make are still attributed correctly.
willbicks 3y ago
I prefer using hardware tokens (in most cases a PKCS#11 smart card) because it means that even with a forwarded SSH agent, every request to use my private key requires a PIN on my client which is verified by the isolated cryptographic processor. It's impossible for my private key to leave that card and get cached anywhere else. While I haven't enabled it on my Yubikey I understand they can do similar. The downside is that compatibility in edge cases, while much better than I'd expect, is still not perfect. In particular Windows support outside of Putty gets challenging.
- gorkish 3y ago
  The RCE is related to ssh-agent's support for PKCS#11, so, yeah you are right this is a valid method to prevent key access or theft via the agent (I also have to approve every use of my PK), but in this case it's not protecting against the RCE, and the workaround in the meantime is to disable PKCS#11 `ssh-agent -P ''`
- throwawaaarrgh 3y ago
  or you could just use the -c option to ssh-add and be prompted every time the key is handed over
- toast0 3y ago
  The other downside is it's much harder to do bulk operations against a fleet. It's not reasonable to enter a PIN for each access when you need to push something to 1000 nodes. 100 nodes is probably ok, but not great.
- gunapologist99 3y ago
  That won't save you for this RCE!
  - themoonisachees 3y ago
    It will save his keys though. It's a start.
    - 3y ago
salawat 3y ago
Real talk though. How do you manage ssh chaining then? Unique set of keys per user per machine? Short of scp'ing your private key over from your starting box, I can't really think of another way. Then again, if it's not your box, (you don't own the hardware have sole monopoly of root), you probably shouldn't be ssh'ing from it anyway. I've always held there are no true secrets on a computer... Until multi-billion dollar companies decide to collude anyway. Anywho, honest question. I'm a big fan of ssh roaming, never realized that ssh-agent was a thing, but after reading this, if I were to use it, I'd most certainly be doing it witout pkcs11 built in. Shotgunning shared libs for side-effects is absolutely mad. Need to slot that into my source code reading list.
- jcotton42 3y ago
  Depending on your usecase, ProxyJump may suffice https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump
- aidenn0 3y ago
  > How do you manage ssh chaining then ProxyJump in the .ssh/config file. If your SSH is too old to support ProxyJump, you should probably upgrade, but this works as well: ProxyCommand ssh -q -W %h:%p jumpost
  - salawat 3y ago
    Noted... Guess I know what I'm doing today! Edit: Nevermind, tried it, not sufficient for my use case. I tend to do a lot of mesh-y bouncing around between servers, and -J seems to be more intended for a star/hub&spoke topology. Common ssh priv-key to all machines, or alternatively, a unique set of priv_keys per user per dest machine is about the way to go. You still have privilege escalations to worry about, but thems the breaks. It's a neat trick, I'll give it that.
    - gunapologist99 3y ago
      Just generate a new keypair there on your bounce box then. Don't do -A because this RCE means that not only do you lose your keys, but you lose your laptop, too!
TechBro8615 3y ago
I think SSH agent forwarding is a fairly common setup with VSCode Remote, where the developer wants to forward their keys to the remote dev server so that they can perform Git operations using their credentials.
- gunapologist99 3y ago
  Easy fix: generate private keys on the remote dev server and then use those keys instead of your own with git.
  - lxgr 3y ago
    This exposes your private keys to a potentially less-trusted dev server. It also just doesn’t work with hardware tokens for the same reason.
    - TechBro8615 3y ago
      Is it possible to configure SSH agent forwarding to only forward some specific private keys? Although I guess that wouldn't protect you from the root problem of a vulnerable OpenSSH client.
      - lxgr 3y ago
        > Is it possible to configure SSH agent forwarding to only forward some specific private keys? Not sure about that, but you can configure it so that it asks you for confirmation before every private key usage, so I suspect you could script a solution around that confirmation mechanism? > Although I guess that wouldn't protect you from the root problem of a vulnerable OpenSSH client. Yes – in the end, your SSH client, the terminal emulator it's running in etc. are ultimately software too that could be remotely exploited. I think in this particular case, there was a certain mismatch of threat expectations between the attack surfaces of ssh (the client, exposed to lots of potentially malicious input) and ssh-agent (mostly accepting input from semi-trusted processes on the same host – except for agent forwarding, of course).
      - 3y ago

starfallg 3y ago

Yup, and this is explained in detail in the ssh(1) man page. No-one should be using -A. Using a jump host via -J is the way to go.

Arnavion 3y ago
One use of -A, namely using the ssh server as a jumphost, is covered by -J. The general use of -A, namely doing operations on the ssh server that require keys from the ssh client, is not. If I'm on machine foo and I want to connect to bar.example.org and clone a git repo there from baz.example.org, and baz.example.org requires an identity key that is in foo's ssh agent, then -A is the only option.
- GauntletWizard 3y ago
  ssh-askpass is a great tool in these situations. It intercedes and allows you to manually control all key usage attempts.
  - devnullbrain 3y ago
    Ah, a skillful execution of Cunningham's Law
  - Arnavion 3y ago
    I'm not sure what relevance ssh-askpass has to what I wrote.
    - GauntletWizard 3y ago
      Right, this recipe isn't well known. `ssh-add -c` will cause your ssh agent to pop up with ssh-askpass every time it does an authentication. This is fiddly; You need to have configured your ssh-agent right, and there's a slightly different version to use keychains on macos that's equivalent.
      - Arnavion 3y ago
        Once again, I'm not sure what relevance this has to the discussion.
        pritambaral 3y ago
        Some people denounce all use of agent forwarding because, by default, ssh-agent doesn't confirm with the user before signing a request with the users private key. This means, if you ssh into a compromised/malicious host with your agent forwarded, malicious code on that machine can just silently ssh into other servers as you. This trick has been used in the past by blackhats to escalate from a compromised CI environment to full production takeover. GauntletWizard probably meant to respond to your parent in this thread.
- gunapologist99 3y ago
  No, you might as well copy the keys over there for all the security you're getting (actually, that's safer given this RCE which compromises both sides, even given that there's no real way to shred the leaked key, but of course I'm not actually suggesting this! they're both really bad.) Your best option is too easy: just generate a new keypair on foo. Then you can populate baz.example.org with that public key instead of your own.
  - Arnavion 3y ago
    >No, you might as well copy the keys over there Only if the keys are insecure enough to be copied, ie not backed by an HSM, or backed by an HSM but marked exportable. >for all the security you're getting (actually, that's safer given this RCE which compromises both sides, even given that there's no real way to shred the leaked key, but of course I'm not actually suggesting this! they're both really bad.) Connecting to a malicious SSH server is already dangeorous with or without this vulnerability, and with or without agent forwarding. Eg a malicious server can modify the shell to emit VT codes to mess up your terminal. >Your best option is too easy: just generate a new keypair on foo. Sure, but this creates additional identities that baz.example.org must be taught to trust, which is not always desirable or even an option.
    - gunapologist99 3y ago
      > Only if the keys are insecure enough to be copied, ie not backed by an HSM, or backed by an HSM but marked exportable. Agent forwarding forwards the keys insecurely. If you can launch an agent-forwarded connection, then the HSM is already irrelevant. > "Connecting to a malicious SSH server" is already dangeorous with or without this vulnerability, and with or without agent forwarding. Eg a malicious server can modify the shell to emit VT codes to mess up your terminal. yes, but messing up your terminal is not an RCE (and you should be able to fix it with stty sane). It's just an annoyance and just tipped you off that obviously there's something wrong there. :) Obviously you should be checking your host keys etc and ideally you woud never accidentally connect to a malicious server. However, in the unlucky but perhaps inevitable event that you do connect to a malicious server, you shouldn't risk exposing your entire client workstation! (through this or another RCE) or your SSH private keys! (through normal agent forwarding operation) with agent forwarding. > Sure, but this creates additional identities that baz.example.org must be taught to trust, which is not always desirable or even an option. A key is not necessarily an identity. Most platforms permit more than one public key to be associated with a user, including github, gitlab, userify, etc. However, ideally you are correct -- this would result in a new user account that has limited access to only the things it really needs (principle of least privilege).
- drdaeman 3y ago
  Another relatively popular use case is pam_ssh_agent_auth for passwordless-but-authenticated sudo. This, by definition, requires agent forwarding. (I don't use ssh-agent, I use gpg-agent with an SSH socket - I suppose I'm fine?)
pritambaral 3y ago
There are uses for `-A` that aren't just for jumping into hosts. I work on code in ephemeral containers and need to use git against remote git servers. If I had to use a key local to the container, I'd have to constantly add new keys to my git servers. Of course, I have secured by agent from abuse. I use gpg-agent as my ssh agent and make it visually confirm every use of my private key. I even set up those containers to use forwarded GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL} envs. So other people can work on the same container as me and git commits we make are still attributed correctly.

willbicks 3y ago

I prefer using hardware tokens (in most cases a PKCS#11 smart card) because it means that even with a forwarded SSH agent, every request to use my private key requires a PIN on my client which is verified by the isolated cryptographic processor. It's impossible for my private key to leave that card and get cached anywhere else. While I haven't enabled it on my Yubikey I understand they can do similar. The downside is that compatibility in edge cases, while much better than I'd expect, is still not perfect. In particular Windows support outside of Putty gets challenging.

gorkish 3y ago
The RCE is related to ssh-agent's support for PKCS#11, so, yeah you are right this is a valid method to prevent key access or theft via the agent (I also have to approve every use of my PK), but in this case it's not protecting against the RCE, and the workaround in the meantime is to disable PKCS#11 `ssh-agent -P ''`
throwawaaarrgh 3y ago
or you could just use the -c option to ssh-add and be prompted every time the key is handed over
toast0 3y ago
The other downside is it's much harder to do bulk operations against a fleet. It's not reasonable to enter a PIN for each access when you need to push something to 1000 nodes. 100 nodes is probably ok, but not great.
gunapologist99 3y ago
That won't save you for this RCE!
- themoonisachees 3y ago
  It will save his keys though. It's a start.
  - 3y ago

salawat 3y ago

Real talk though. How do you manage ssh chaining then? Unique set of keys per user per machine? Short of scp'ing your private key over from your starting box, I can't really think of another way. Then again, if it's not your box, (you don't own the hardware have sole monopoly of root), you probably shouldn't be ssh'ing from it anyway. I've always held there are no true secrets on a computer... Until multi-billion dollar companies decide to collude anyway. Anywho, honest question. I'm a big fan of ssh roaming, never realized that ssh-agent was a thing, but after reading this, if I were to use it, I'd most certainly be doing it witout pkcs11 built in. Shotgunning shared libs for side-effects is absolutely mad. Need to slot that into my source code reading list.

jcotton42 3y ago
Depending on your usecase, ProxyJump may suffice https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump
aidenn0 3y ago
> How do you manage ssh chaining then ProxyJump in the .ssh/config file. If your SSH is too old to support ProxyJump, you should probably upgrade, but this works as well: ProxyCommand ssh -q -W %h:%p jumpost
- salawat 3y ago
  Noted... Guess I know what I'm doing today! Edit: Nevermind, tried it, not sufficient for my use case. I tend to do a lot of mesh-y bouncing around between servers, and -J seems to be more intended for a star/hub&spoke topology. Common ssh priv-key to all machines, or alternatively, a unique set of priv_keys per user per dest machine is about the way to go. You still have privilege escalations to worry about, but thems the breaks. It's a neat trick, I'll give it that.
  - gunapologist99 3y ago
    Just generate a new keypair there on your bounce box then. Don't do -A because this RCE means that not only do you lose your keys, but you lose your laptop, too!

TechBro8615 3y ago

I think SSH agent forwarding is a fairly common setup with VSCode Remote, where the developer wants to forward their keys to the remote dev server so that they can perform Git operations using their credentials.

gunapologist99 3y ago
Easy fix: generate private keys on the remote dev server and then use those keys instead of your own with git.
- lxgr 3y ago
  This exposes your private keys to a potentially less-trusted dev server. It also just doesn’t work with hardware tokens for the same reason.
  - TechBro8615 3y ago
    Is it possible to configure SSH agent forwarding to only forward some specific private keys? Although I guess that wouldn't protect you from the root problem of a vulnerable OpenSSH client.
    - lxgr 3y ago
      > Is it possible to configure SSH agent forwarding to only forward some specific private keys? Not sure about that, but you can configure it so that it asks you for confirmation before every private key usage, so I suspect you could script a solution around that confirmation mechanism? > Although I guess that wouldn't protect you from the root problem of a vulnerable OpenSSH client. Yes – in the end, your SSH client, the terminal emulator it's running in etc. are ultimately software too that could be remotely exploited. I think in this particular case, there was a certain mismatch of threat expectations between the attack surfaces of ssh (the client, exposed to lots of potentially malicious input) and ssh-agent (mostly accepting input from semi-trusted processes on the same host – except for agent forwarding, of course).
    - 3y ago