29 ms·

Author here - hopefully the level of detail here will let others learn from our mistakes (and hopefully benefit from how we've chosen to fix them going forwards

Author here - hopefully the level of detail here will let others learn from our mistakes (and hopefully benefit from how we've chosen to fix them going forwards). Happy to answer any/all questions or comments. TL;DR: keep your services patched; lock down SSH; partition your network; and there's almost never a good reason to use SSH agent forwarding.

Arathorn 8y ago
I should probably also link to the HN thread from when this happened, which has a lot of interesting discussion about the security issues in question: https://news.ycombinator.com/item?id=19642554 https://news.ycombinator.com/item?id=19642554 Also, https://news.ycombinator.com/item?id=19643227 https://news.ycombinator.com/item?id=19643227 is an excellent tl;dr.
vector_spaces 8y ago
Thanks so much for this write-up, really appreciate the candidness and detail. I've learned a lot from it and will be adding some of these practices to the infrastructure that I manage.
voltagex_ 8y ago
>The attacker put an SSH key on the box, which was unfortunately exposed to the internet via a high-numbered SSH port for ease of admin by remote users, and placed a trap which waited for any user to SSH into the jenkins user, which would then hijack any available forwarded SSH keys to try to add the attacker’s SSH key You could also fund/donate to/advocate for a better SSH agent. I use both Pageant and ssh-agent in my home network for ease of ssh'ing into boxes, especially Unifi gear and some dev VMs. I don't think I will stop using agents, but I probably wouldn't use them at work. Why couldn't there be an agent that required you to touch a Yubikey before it'd allow keys to be forwarded? Why couldn't you add prompting and timeouts to an agent?
- JoachimSchipper 8y ago
  ssh-agent has prompting and you can set up a Yubikey with ssh. The problem here was agent forwarding, which you should almost always replace with opening a new connection via ssh -J (or equivalent.)
  - voltagex_ 8y ago
    But can I prompt every time the agent is used?
    - Arathorn 8y ago
      How would you know whether the agent is being used by a legitimate app or a malicious app racing with a legitimate app to steal access? At least you only would leak a single access, and you would have a higher chance of noticing, but I can also see that if the hijack was done intermittently you might write it off as a glitch...
    - jkire 8y ago
      Yup, if you're using ssh-agent (as opposed to something like gnome-keyring) setting `AddKeysToAgent confirm` to your ssh config should cause a pop up to happen every time anything requests a key from the agent.
- 32032141 8y ago
  Just use ProxyJump. You basically should never be using agent forwarding.

Arathorn 8y ago

I should probably also link to the HN thread from when this happened, which has a lot of interesting discussion about the security issues in question: https://news.ycombinator.com/item?id=19642554 https://news.ycombinator.com/item?id=19642554 Also, https://news.ycombinator.com/item?id=19643227 https://news.ycombinator.com/item?id=19643227 is an excellent tl;dr.

vector_spaces 8y ago

Thanks so much for this write-up, really appreciate the candidness and detail. I've learned a lot from it and will be adding some of these practices to the infrastructure that I manage.

voltagex_ 8y ago

>The attacker put an SSH key on the box, which was unfortunately exposed to the internet via a high-numbered SSH port for ease of admin by remote users, and placed a trap which waited for any user to SSH into the jenkins user, which would then hijack any available forwarded SSH keys to try to add the attacker’s SSH key You could also fund/donate to/advocate for a better SSH agent. I use both Pageant and ssh-agent in my home network for ease of ssh'ing into boxes, especially Unifi gear and some dev VMs. I don't think I will stop using agents, but I probably wouldn't use them at work. Why couldn't there be an agent that required you to touch a Yubikey before it'd allow keys to be forwarded? Why couldn't you add prompting and timeouts to an agent?

JoachimSchipper 8y ago
ssh-agent has prompting and you can set up a Yubikey with ssh. The problem here was agent forwarding, which you should almost always replace with opening a new connection via ssh -J (or equivalent.)
- voltagex_ 8y ago
  But can I prompt every time the agent is used?
  - Arathorn 8y ago
    How would you know whether the agent is being used by a legitimate app or a malicious app racing with a legitimate app to steal access? At least you only would leak a single access, and you would have a higher chance of noticing, but I can also see that if the hijack was done intermittently you might write it off as a glitch...
  - jkire 8y ago
    Yup, if you're using ssh-agent (as opposed to something like gnome-keyring) setting `AddKeysToAgent confirm` to your ssh config should cause a pop up to happen every time anything requests a key from the agent.
32032141 8y ago
Just use ProxyJump. You basically should never be using agent forwarding.