6 ms·

Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places. SS

by patio11 11y ago

Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places. SSH is designed so that, even if you connect to an evil host, the host only learns your public key, not your private key. This leaks your private key to an evil host. You might think "I only SSH into boxes that I own, so I'm good, right?", but if you in the future lose a box to the enemy, they can then use that box to grab your private key, and then use that key to get into every other box you can access with that keypair. This turns your sysadmin's photo blog hosted on Digital Ocean into a potential vector into your entire infrastructure, since there is a high likelihood they use the same private key for both. If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. You should expect adversaries to add an exploit for this to their rootkits. That would be bad enough since attackers targeting your infrastructure may exist, but this is also plausibly exploitable in a spray-and-pray fashion. Root any server on the Internet, using e.g. a WordPress vulnerability. Install key-stealing malicious SSH server. Forward all keys to your C&C infrastructure. Your infra does a reverse lookup from public key to Github account (trivial -- try "ssh whoami.filippo.io" if you don't believe me) and immediately optimistically tries to use the stolen key to log into the web tier of any site listed in their email/bio/Twitter profile/etc, pinging the attacker when it succeeds.

rchiniquy 11y ago
If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate. Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.
moviuro 11y ago
This may be a start: Host * IdentityFile ~/.ssh/%h Comment: One key per host, named after the host you connect to
- deleted 11y ago
  [deleted]
- mioelnir 11y ago
  You also need to set IdentitiesOnly yes if I remember the config setting correct. Note however that this only limits the offered keys during the authentication phase. If you use AgentForwarding, this still has the entire keyring available afterwards.
  - aexaey 11y ago
    Don't use SSH agent forwarding. https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-considered-harmful/ https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-con...
    - Xylakant 11y ago
      which is not very helpful if you for example want to run git via ssh or similar commands on the target server. There's places where there are two alternatives: agent forwarding or copy the key. Using ssh agent forwarding allows me to use a smart card for the key. It still has it's problems but it's much better than having the key on the remote machine - for example the use of a smartcard mitigates this vulnerability since the key never enters the process memory.
      - aexaey 11y ago
        Not sure which exact scenario do you have in mind with ssh+git. ProxyCommand method works just fine for me with my private jumphost in the middle and github on the far end. Note that for ProxyCommand to work, you don't need a full shell on a jumphost, just "AllowTcpForwarding yes" is enough. On the other hand, with AgentForwarding method you do need a full shell on a jumphost.
robertcope 11y ago
I really hope that sysadmins/people are not using the same private key for work and pleasure. And for that matter, the same key at work for DEV/TEST/PROD. But you're probably right that a lot of them are.
- arca_vorago 11y ago
  As a sysadmin I am thankful I realized this was a mistake a long time ago, but you are right that I see it happen way too often for comfort.
- izolate 11y ago
  Depends how you look at it. I use my key pair as my digital identification, whether at work or at home. I only have the one set. That logic held up until now I suppose, with the leaking of private keys.
  - mioelnir 11y ago
    For the at-work keys, I differentiate between work/workstation and work/laptop. Full disk encryption and the bcrypt based ssh private key format hopefully buy me enough time to revoke the laptop keys and have a KRL update pushed to all servers if it gets stolen.
  - __jal 11y ago
    It does depend. This may be leftover behavior from my contract days, but I've always had at least one key per entity with which I interact[1]. The reasoning is that it is far easier for me (or the entity, but that's a bit different) to delete one keyfile to sever access than it is for me to rekey everything else. I don't want access to things I'm not actively engaged with - compromises happen, even to engineers' laptops, and that conversation with former employers is too much like calling up your exes to tell them about a VD test result for my comfort. As far as it being digital identification, lots of companies have IDs separate from your DL/passport. This is usually because the company ID provides access to something your other ID doesn't. Same principle. [1] There are lots reasons to have lots of different keys, and only having one per entity is pretty rare for me.
agwa 11y ago
> If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device. For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.
- hamburglar 11y ago
  > But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. This is what I would assume, but has it been confirmed that this vulnerability doesn't affect identities provided by an SSH agent? If any of this "roaming" support was provided by the agent, it could be the case that the leakage can be triggered via SSH agent protocol requests. I doubt this is the case but I don't see anything confirming or denying it. It would be nice to hear some official word on whether agent-based authentication is vulnerable, because if it isn't, the seriousness to me is greatly reduced, as I never load identities in the client directly.
  - pquerna 11y ago
    The main issue identified by Qualys is a more generic memory disclosure bug: https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-0778/openssh-cve-2016-0777-cve-2016-0778.txt https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077... This means they are basically able to dump the memory of just the running `ssh` process -- eerily similar to Heartbleed. This means private keys stored by the `ssh-agent` process, outside of the `ssh` process connecting to an Evil Server(TM) are not affected. This is because the protocol used between SSH Agent and SSH client does not transfer the entire private key, rather the SSH client asks the Agent to do a signing operation on it's behalf.
bisby 11y ago
Ironically, ssh'ing into a random server is the exact thing you shouldn't do if you aren't patched. Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to" This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.

by patio11 11y ago

rchiniquy 11y ago

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate. Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

moviuro 11y ago

This may be a start: Host * IdentityFile ~/.ssh/%h Comment: One key per host, named after the host you connect to

deleted 11y ago
[deleted]
mioelnir 11y ago
You also need to set IdentitiesOnly yes if I remember the config setting correct. Note however that this only limits the offered keys during the authentication phase. If you use AgentForwarding, this still has the entire keyring available afterwards.
- aexaey 11y ago
  Don't use SSH agent forwarding. https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-considered-harmful/ https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-con...
  - Xylakant 11y ago
    which is not very helpful if you for example want to run git via ssh or similar commands on the target server. There's places where there are two alternatives: agent forwarding or copy the key. Using ssh agent forwarding allows me to use a smart card for the key. It still has it's problems but it's much better than having the key on the remote machine - for example the use of a smartcard mitigates this vulnerability since the key never enters the process memory.
    - aexaey 11y ago
      Not sure which exact scenario do you have in mind with ssh+git. ProxyCommand method works just fine for me with my private jumphost in the middle and github on the far end. Note that for ProxyCommand to work, you don't need a full shell on a jumphost, just "AllowTcpForwarding yes" is enough. On the other hand, with AgentForwarding method you do need a full shell on a jumphost.

robertcope 11y ago

I really hope that sysadmins/people are not using the same private key for work and pleasure. And for that matter, the same key at work for DEV/TEST/PROD. But you're probably right that a lot of them are.

arca_vorago 11y ago
As a sysadmin I am thankful I realized this was a mistake a long time ago, but you are right that I see it happen way too often for comfort.
izolate 11y ago
Depends how you look at it. I use my key pair as my digital identification, whether at work or at home. I only have the one set. That logic held up until now I suppose, with the leaking of private keys.
- mioelnir 11y ago
  For the at-work keys, I differentiate between work/workstation and work/laptop. Full disk encryption and the bcrypt based ssh private key format hopefully buy me enough time to revoke the laptop keys and have a KRL update pushed to all servers if it gets stolen.
- __jal 11y ago
  It does depend. This may be leftover behavior from my contract days, but I've always had at least one key per entity with which I interact[1]. The reasoning is that it is far easier for me (or the entity, but that's a bit different) to delete one keyfile to sever access than it is for me to rekey everything else. I don't want access to things I'm not actively engaged with - compromises happen, even to engineers' laptops, and that conversation with former employers is too much like calling up your exes to tell them about a VD test result for my comfort. As far as it being digital identification, lots of companies have IDs separate from your DL/passport. This is usually because the company ID provides access to something your other ID doesn't. Same principle. [1] There are lots reasons to have lots of different keys, and only having one per entity is pretty rare for me.

agwa 11y ago

> If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device. For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.

hamburglar 11y ago
> But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. This is what I would assume, but has it been confirmed that this vulnerability doesn't affect identities provided by an SSH agent? If any of this "roaming" support was provided by the agent, it could be the case that the leakage can be triggered via SSH agent protocol requests. I doubt this is the case but I don't see anything confirming or denying it. It would be nice to hear some official word on whether agent-based authentication is vulnerable, because if it isn't, the seriousness to me is greatly reduced, as I never load identities in the client directly.
- pquerna 11y ago
  The main issue identified by Qualys is a more generic memory disclosure bug: https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-0778/openssh-cve-2016-0777-cve-2016-0778.txt https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077... This means they are basically able to dump the memory of just the running `ssh` process -- eerily similar to Heartbleed. This means private keys stored by the `ssh-agent` process, outside of the `ssh` process connecting to an Evil Server(TM) are not affected. This is because the protocol used between SSH Agent and SSH client does not transfer the entire private key, rather the SSH client asks the Agent to do a signing operation on it's behalf.

bisby 11y ago

Ironically, ssh'ing into a random server is the exact thing you shouldn't do if you aren't patched. Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to" This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.