5 ms·

> [The attacker] placed a trap which waited for any user to SSH into the jenkins user, which would then hijack any available forwarded SSH keys to try to add th

> [The attacker] placed a trap which waited for any user to SSH into the jenkins user, which would then hijack any available forwarded SSH keys to try to add the attacker’s SSH key to root@ on as many other hosts as possible. Can the system you log into via ssh just dump your forwarded PRIVATE key? That easily? Or this was about ssh client being patched on jenkins box to add malicious keys wherever the devops ssh'd from jenkins box?

bpye 7y ago
You cannot dump it but you can sign data with it.
Arathorn 7y ago
sorry, I think I could have been clearer here. When you log into a host with SSH agent forwarding turned on, the private key data itself isn't available to the host you're logging into. However, when you try to SSH onwards from that host, agent forwarding means that the authentication handshake is forwarded through to the agent running on your client, which of course has access to your private keys. So, even though the private key data itself isn't directly available to the host, any code running which can inspect the SSH_AUTH_SOCK environment variable of the session that just logged in can use that var to silently authenticate with other remote systems on your behalf. If you've already found a list of candidate hosts (e.g. by inspecting ~/.ssh/known_hosts) then your malware can simply loop over the list, trying to log in as root@ (or user@) and compromising them however you like. Which is what happened here, by copying a malicious authorized_keys2 file with a malicious key onto the target hosts. You don't need to patch the ssh client; it's just ssh agent forwarding doing its thing. :|
- mikekchar 7y ago
  There's another scary implication. If the user has set up SSH on their machines and added their own key as a valid key for logging in (for instance if you have 2 machines and you use the same SSH key for logging into everything), the attacker can log into the users own machine and compromise it. This makes remediation even more difficult because even after you have secured all the servers, the attacker can get back in through an affected user's machine.
- voltagex_ 7y ago
  A simple yet clever attack. I wonder how you'd protect against it without banning SSH forwarding, which has almost certainly saved me from (some) RSI.
  - Arathorn 7y ago
    since banning ssh agent forwarding I haven’t missed it at all - ssh -J has been an almost perfect replacement for my use cases. What sort of thing are you using ssh -A for which couldn’t be replaced by ssh -J?
    - mschuster91 7y ago
      > What sort of thing are you using ssh -A for which couldn’t be replaced by ssh -J? git checkouts from private repositories, for example. HTTPS requires username/password which may or may not be checked/monitored.
      - Arathorn 7y ago
        Right. I covered this specifically in the writeup, because it's a use case that we have too. Our proposal is: > If you need to regularly copy stuff from server to another (or use SSH to GitHub to check out something from a private repo), it might be better to have a specific SSH ‘deploy key’ created for this, stored server-side and only able to perform limited actions. And this is the approach we're taking going forwards. If the problem is that you only ever want to read from git when an admin is logged into the machine, i guess the safest bet would be to use a temporary deploy key (or temporarily copy the deploy key onto the machine until you've finished admining). Forwarding all the keys from your agent is a recipe to end up pwned like we did, however.
  - im3w1l 7y ago
    > RSI Manually typing passwords in on an attacker-controlled machine doesn't sound very safe either.

bpye 7y ago

You cannot dump it but you can sign data with it.

Arathorn 7y ago

sorry, I think I could have been clearer here. When you log into a host with SSH agent forwarding turned on, the private key data itself isn't available to the host you're logging into. However, when you try to SSH onwards from that host, agent forwarding means that the authentication handshake is forwarded through to the agent running on your client, which of course has access to your private keys. So, even though the private key data itself isn't directly available to the host, any code running which can inspect the SSH_AUTH_SOCK environment variable of the session that just logged in can use that var to silently authenticate with other remote systems on your behalf. If you've already found a list of candidate hosts (e.g. by inspecting ~/.ssh/known_hosts) then your malware can simply loop over the list, trying to log in as root@ (or user@) and compromising them however you like. Which is what happened here, by copying a malicious authorized_keys2 file with a malicious key onto the target hosts. You don't need to patch the ssh client; it's just ssh agent forwarding doing its thing. :|

mikekchar 7y ago
There's another scary implication. If the user has set up SSH on their machines and added their own key as a valid key for logging in (for instance if you have 2 machines and you use the same SSH key for logging into everything), the attacker can log into the users own machine and compromise it. This makes remediation even more difficult because even after you have secured all the servers, the attacker can get back in through an affected user's machine.
voltagex_ 7y ago
A simple yet clever attack. I wonder how you'd protect against it without banning SSH forwarding, which has almost certainly saved me from (some) RSI.
- Arathorn 7y ago
  since banning ssh agent forwarding I haven’t missed it at all - ssh -J has been an almost perfect replacement for my use cases. What sort of thing are you using ssh -A for which couldn’t be replaced by ssh -J?
  - mschuster91 7y ago
    > What sort of thing are you using ssh -A for which couldn’t be replaced by ssh -J? git checkouts from private repositories, for example. HTTPS requires username/password which may or may not be checked/monitored.
    - Arathorn 7y ago
      Right. I covered this specifically in the writeup, because it's a use case that we have too. Our proposal is: > If you need to regularly copy stuff from server to another (or use SSH to GitHub to check out something from a private repo), it might be better to have a specific SSH ‘deploy key’ created for this, stored server-side and only able to perform limited actions. And this is the approach we're taking going forwards. If the problem is that you only ever want to read from git when an admin is logged into the machine, i guess the safest bet would be to use a temporary deploy key (or temporarily copy the deploy key onto the machine until you've finished admining). Forwarding all the keys from your agent is a recipe to end up pwned like we did, however.
- im3w1l 7y ago
  > RSI Manually typing passwords in on an attacker-controlled machine doesn't sound very safe either.