13 ms·

OpenSSH: client bug CVE-2016-0777

617 points by moviuro 11y ago

jlgaddis 11y ago
Workaround (yes, it's client-side): # echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config Disclaimer: won't work on all operating systems, shells, etc. YMMV. Consult a doctor before following any advice you get from the Internet. Void where prohibited. Restrictions may apply. Edited per comments below
- pmoriarty 11y ago
  Would this work too? $ echo "UseRoaming no" >> ~/.ssh/config
  - Zr40 11y ago
    Yes, but it would not apply to other users.
  - iamtew 11y ago
    Yes, but it'll only affect that user.
    - ralphm 11y ago
      And only the last `Host` entry in that config, if present.
- doogle88 11y ago
  Presumably you would have to connect to a malicious host to be effected? Or perhaps a MITM on your connection to a legit host can exploit you somehow.
  - masklinn 11y ago
    > Presumably you would have to connect to a malicious host to be effected? Malicious or compromised. > roaming code in the ssh client could be tricked by a hostile sshd server, potentially leaking key material.
  - amatix 11y ago
    From the updated OP: > The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers.
    - jon-wood 11y ago
      Malicious, compromised, or new servers. Because really, how many people check the host key for a newly spun up EC2 instance?
- ralphm 11y ago
  Don't ever fix your `ssh_config` by appending stuff to the end of the file. The configuration syntax allows for block constructs without an explicit end marker (like `Match` and `Host`). Appending will cause all kinds of sadness.
  - gnur 11y ago
    It's from the linked page.
    - ralphm 11y ago
      Indeed. The page gives bad advise. If your config has `Host` blocks, like often in people's personal configs, or if you have a `Match` block, the new directive only applies to the last of those blocks in the config file.
      - snowwrestler 11y ago
        Sufficient to place it at the top of ~/.ssh/config? Or does it also need a block header of its own, like: Host * UseRoaming no
        skuhn 11y ago
        I confirmed that it does not need to be placed into a Host block, as long as it is above all of your blocks. For instance: UseRoaming no Host * Blah yes Test with: ssh -v remote.ssh.host.com uptime 2>&1 | grep -i roaming If it returns nothing, the config fix is active. If it isn't active, you'll see 'debug1: Roaming not allowed by server'
      - jldugger 11y ago
        Which is why they're appending a newline and Host *?
        ralphm 11y ago
        Note that the example was changed as a result of my initial comment.
- narsil 11y ago
  I'm emailing the one-liner to folks and found it helpful to just include the root subshell: sudo bash -c 'echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config'
moviuro 11y ago
The most terrible thing here is that OpenBSD usually does some full disclosure. This is sure going to be a nasty bug.
- praseodym 11y ago
  Why is this terrible? In any case, the software is open source so any security fix commit can also be considered as full disclosure.
  - kardos 11y ago
    But they didn't disclose the bug yet. The implication is that this bug is /so bad/ that it justified breaking the normal "just disclose it" approach.
  - deleted 11y ago
    [deleted]
Aissen 11y ago
Roaming feature has been in OpennSSH code for a while, but is undocumented: http://superuser.com/a/826734/47771 http://superuser.com/a/826734/47771
- DrRobinson 11y ago
  "Affects all OpenSSH 5.4 - 7.1". 5.4 was released March 2010 it seems
- ryanlol 11y ago
  AFAIK the roaming feature has been disabled on the server side for a while, but mistakenly left enabled on the client.
  - DrRobinson 11y ago
    Seems correct: https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2 https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2
merqurio 11y ago
As far as I know, this will affect any OSX, am I right ?
- moviuro 11y ago
  s/X//
- deleted 11y ago
  [deleted]
  - ryanlol 11y ago
    This does not affect the sshd.
- ryanlol 11y ago
  According to Theo, yes.
- DrRobinson 11y ago
  It affects all operating systems: http://marc.info/?l=openbsd-tech&m=145278077820529&w=2 http://marc.info/?l=openbsd-tech&m=145278077820529&w=2
  - merqurio 11y ago
    Is there any way to update openssh on my side, without waiting to apple ? Is this enough ?https://mochtu.de/2015/01/07/updating-openssh-on-mac-os-x-10-10-yosemite/ https://mochtu.de/2015/01/07/updating-openssh-on-mac-os-x-10...
    - DrRobinson 11y ago
      You can patch it with: # echo 'UseRoaming no' >> /etc/ssh/ssh_config or $ echo "UseRoaming no" >> ~/.ssh/config
      - mapgrep 11y ago
ryanlol 11y ago
Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264 https://twitter.com/marver/status/687644904575627264 Edit 1: Here's some relevant commits too https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2 https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2 Edit 2: This mailing list post seems to discuss the vulnerable feature http://www.gossamer-threads.com/lists/openssh/dev/49018?do=post_view_threaded#49018 http://www.gossamer-threads.com/lists/openssh/dev/49018?do=p... Edit 3: Got a better description of actual impact of the bugs: >Experimental roaming code in the ssh client could be tricked by a hostile sshd >server, potentially leaking key material. CVE-2016-077 and CVE-0216-078. >Prevent this problem immediately by adding the line "UseRoaming no" to >/etc/ssh/ssh_config.
- masklinn 11y ago
  > Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264 https://twitter.com/marver/status/687644904575627264 The subsystem was apparently experimentally (hence the undocumented option) introduced 6 years ago and essentially never used, the server support was never implemented: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/roaming_serv.c http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/roa...
- sarciszewski 11y ago
  > >server, potentially leaking key material. CVE-2016-077 and CVE-0216-078. Presumably you meant CVE-2016-0777 and CVE-2016-0778
  - ryanlol 11y ago
    I'm just quoting OFFICIAL SOURCES here.
- kardos 11y ago
  So the real question is, can a MITM intercept connections to boxen you frequent to exploit this? Or is it limited to connecting to hostile honeypots?
noir_lord 11y ago
http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/010_ssh.patch.sig http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/010_ss...
- fulafel 11y ago
  Leaking private keys seems to be all the rage these days.
rnhmjoj 11y ago
Does this affect mosh?
- fulafel 11y ago
  No, the roaming in Mosh is unrelated to OpenSSH's roaming feature.
  - sneak 11y ago
    Irrelevant; it still affects mosh as mosh does an ssh connection first to do auth and key exchange. That ssh session is potentially vulnerable when connecting to a malicious or compromised host.
  - TazeTSchnitzel 11y ago
    Doesn't mosh initially establish a connection via SSH, though?
  - teamhappy 11y ago
    But Mosh uses OpenSSH for authentication. And if roaming is enabled by default on the client this could still be an issue, no? I'd say it's still a good idea to add "UseRoaming no" to ssh_config.
  - rakoo 11y ago
    But authentication in Mosh still relies on OpenSSH, so auth could be intercepted by an attacker after which Mosh is completely open to them: https://mosh.mit.edu/ https://mosh.mit.edu/ > However, in typical usage, Mosh relies on SSH to exchange keys at the beginning of a session, so Mosh will inherit the weaknesses of SSH—at least insofar as they affect the brief SSH session that is used to set up a long-running Mosh session.
  - XMPPwocky 11y ago
    Right, but mosh uses ssh for initial key setup. I suspect it is possible to exploit that. (As an occasional Windows user who's using the Mosh Chrome extension, I have no idea how much risk I'm at; can't edit ssh_config, it's not even exposed)
StefanKarpinski 11y ago
Does it strike anyone else as bizarre / poor form for an experimental feature to be enabled by default in OpenSSH, which is normally very conservative with option defaults?
- agwa 11y ago
  What's more, it was not documented in the ssh_config(5) man page. If it had been, I probably would have disabled it long ago when hardening my SSH config. Time to head to the source to look for other undocumented options... Update: my findings are here (scroll to bottom for the upshot): https://gist.github.com/AGWA/e92d4f5343be1f7a941d https://gist.github.com/AGWA/e92d4f5343be1f7a941d UseRoaming is the only one to be concerned about. There are many other undocumented options, but they're all aliases for a documented option or are deprecated/unsupported.
  - noja 11y ago
    Please report back what you find!
  - ryanlol 11y ago
    https://github.com/openssh/openssh-portable/blob/e6c85f8889c5c9eb04796fdb76d2807636b9eef5/readconf.c https://github.com/openssh/openssh-portable/blob/e6c85f8889c... Here you go
    - ryanlol 11y ago
      Actually, I quickly generated a list of undocumented config options: http://sprunge.us/QDSE http://sprunge.us/QDSE Edit: Fixed version http://sprunge.us/LVYB http://sprunge.us/LVYB
      - vesinisa 11y ago
        most of those seem to be documented: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/ssh_config.5?query=ssh_config&sec=5 http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/... I did a quick comparison between the OpCodes enum and the latest man page, and `UseRoaming` is the only real undocumented option.
ctstover 11y ago
What was this experimental / undocumented roaming feature even suppose to do?
- noja 11y ago
  Complete guess: handle changes of ip address on the client side.
  - jlgaddis 11y ago
    Allow a mobile user to reconnect to a disconnected session without (user) re-authentication.
jlgaddis 11y ago
ETA: I was way off... you can ignore this... :-) cf. section 3.3.5 [0], which describes "Roaming (Suspend/Resume)". This is documentation for an application by a company called AppGate (later acquired by Cryptzone) that wrote {some|most} of the code in OpenSSH's "roaming_client.c". This gives a hint of what the ramifications may be: basically, a MITM, who observed the initial session negotiation, can disconnect the client and hijack an active session. > Roaming is a feature which allows clients to suspend the connection to the AppGate server and later to resume it again. The user does not need to re-authenticate when reconnecting. Indeed, the entire process can be completely automated and nearly invisible to the user. All established connections will remain alive while roaming. This feature is intended for mobile users who move around between networks. > Technically, roaming is accomplished by closing the TCP tunnel when the connection is suspended. When resuming, a new TCP connection is made to the server and the SSH data stream is continued through this new connection. The user does not need to authenticate again, instead the client authenticates to the server, without user interaction, with the help of a random password which was made up when the user authenticated at the start of the session. In addition to knowing this password, the client must also know the encryption keys and encryption state to be able to reconnect. It is therefore impossible for a third party to break in and take over a suspended session. I particularly like that last sentence. [0]: http://download.cryptzone.com/files/download/AppGate-10.2.3/doc/manual_chunked_html/ch03s03.html http://download.cryptzone.com/files/download/AppGate-10.2.3/...
- ryanlol 11y ago
  That's not correct though, MITM cannot exploit this. A malicious SSH server can exploit this during a connection though, but only after host key verification.
- hultner 11y ago
  Oh Cryptzone, funny they offered me a job a few years back. Interesting that this was left in OpenSSH like this, even more as a default.
Aissen 11y ago
Release is here, with many more details, and other security issues fixed: https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-January/034680.html https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...
SFjulie1 11y ago
theo must be mad. Shaming people for leaving useless non essential feature in their code that results in security breach. And now the jewel of his crown has been compromised. The funniest part is now that his jewel has been tarnished, maybe people will understand what he was saying. And maybe too, people that believed privacy can be achieved on the internet will finally look at the problem of believing the 2 general paradox can be solved without at least 2 different constant link on different plans. And the problem is belief is a poor substitute for thinking - critical thinking. And maybe people will discover the sad truth of the internet. Security requires a perfect world, where human beings neither makes mistakes nor are corruptible. Errare humanum est, perseverare diabolicum Oh! Some says that is what 2 factor authentication is. I will answer, my intuition is telling me that 2 factor is good for a fixed amount of time/information and that using it correctly would annoy people to the utmost points. Then people would say well let's accept that fraud exists. Business first. (costs/benefits) Then I say giving 3% of all e-commerce to the bad guys is like admitting organized crime have a strong budget for even more crime ... and that we are fucked. Unless you don't understand that ISIS is basically a startup. A startup that overthrow a state to make even more money and industrialize crime.
- marios 11y ago
  What Theo (and other OpenBSD developers) have been saying all these years is that it's impossible to /not/ make mistakes, which is why sane design and exploit mitigations are important. Mad ? Maybe, I wouldn't know. On the other hand, I bet he's really glad all that effort to have ASLR by default was made, because it makes it more difficult for an attacker to exploit vulnerabilites such as this one.
  - SFjulie1 11y ago
    ASLR makes exploits more difficult as long as you have true randomness. It is just a mitigation ... for another problem. In this case the elephant in the room is stack injection and dynamic libraries. If processes where confined to a well known address space that was self contained ASLR would be useless.. But dependency management would be hellish. Back to the case. OpenSSH has been openly criticize by 9plan teams for exactly the same reason openSSL has been criticized by openSSH team : too much complexity (not to say plan9 came to anything usable and were right (2 wrongs do not make a right)). http://harmful.cat-v.org/software/ssh http://harmful.cat-v.org/software/ssh I do have a feeling as a physicist that the S in CS stands for Shortness of thinking. What is obscure is not profound. And, at the opposite of engineers I don't believe that security is achievable at all on computers. It is like believing there exists a way to avoid triangulation with one strong radio source. Computers leak to much information (especially in the physical world), and C is a map that is taken too much for the territory. Every bugs are exploiting a wrong mapping of concepts to implementation. Modern security is like string theory, admired by everyone because it requires great technical knowledge to master, understood by none because it is way to complex for our "human" brains. We live in an era of belief in solutions. A bit of critical thinking and of distrusts of experts and stuff that you cannot understand without devoting your life to a subject is at my opinion a must. I distrust mathematicians for their capacity of dealing with the real world and its uncertainties, and cryptography (as much as functional programming, big data, algorithmic, IA, machine learning) is math driven in a pure Aristotelian world. Where perfection and harmony is the pillar of thinking. I come from micro-electronics, I only see wires (that are antennas), oscillators, multiplexers, gates and basically a dumb automat I can automatize in respect to time of propagation of signal, and I know that modern computers under the hood are in the physical realm of relativity with approximate answers. I theorize, build, measure, and retheorize ad nauseum until the product is measured to work the way I expect it to in the domain of validity with a good enough confidence and margins of errors are always on my mind to be controled. Security requires a zero margin of ambiguity. Physical world is bound to heisenberg equation and coupling. Purity does not exist and this cannot be mitigated. The postulate of cryptography are wrong from the core. Real world always win at the end. Aristotle way of thinking must die. Math is not science.
deleted 11y ago
[deleted]
ck2 11y ago
If it's not obvious, don't just add that config option, you have to also restart it. Actually wait, is this only affecting the ssh client and not the server/daemon ?
- jlgaddis 11y ago
  Yes, it's client-side only. Server-side, it's disabled (cf. readconf.c).
sharjeel 11y ago
Since this is a client side issue, can this be used to exploit those automated scanners who try to break into your SSH machine?
- deleted 11y ago
  [deleted]
- dsr_ 11y ago
  No. The scanners are looking for password-accessible accounts, not keyed accounts. The scanners won't have useful keys, nor listening ssh daemons.
- Stefan-H 11y ago
  Authenticated scanners that use key auth like Qualys' security appliances could have private keys that are valid across the organization, and if using an affected client version, could leak this information to a malicious system on your network.
protomyth 11y ago
Check me on this, so this is a client-side problem only, so ssh-ing into only know servers shouldn't be an issue and clients cannot cause problems for servers?
- scardine 11y ago
  If an enemy takes control of just one of the hosts you ssh into, he will get your private key and can use it to ssh into any other box where you use RSAAuthentication.
  - protomyth 11y ago
    I already added the mitigation, but I wondering if my servers can be patched on the weekend.
    - drummer32 11y ago
      There is nothing to patch on the server side. You need to ensure the ssh client is updated on the machines you are sshing from.
      - protomyth 11y ago
        ok thanks, that's what I figured, but it I was getting a bit worried that I was missing something.
nulltype 11y ago
If you have a mac (Yosemite), it looks like you want to add " UseRoaming no" under the "Host *" line in /etc/ssh_config (as root). You can test it before and after with this: ssh -v -T git@github.com 2>&1 | grep Roaming debug1: Roaming not allowed by server <- bad ssh -v -T git@github.com 2>&1 | grep Roaming (no output is good)
- super_mario 11y ago
  On El Capitan the file is /etc/ssh/ssh_config
  - snowwrestler 11y ago
    On Mountain Lion it's /private/etc/ssh_config.
    - astrange 11y ago
      /etc points to /private/etc on OS X, so it's all good.
- deleted 11y ago
  [deleted]
- lftl 11y ago
  Interestingly I haven't been able to get the Roaming not allowed message to show on my client with or without the config option set.
  - antoinealb 11y ago
    Did you copy the whole line as "g ssh -v -T git@github.com 2>&1 | grep "Roaming debug1: Roaming not allowed by server"" ? I misread GP's command and made that mistake. The correct command is : ssh -v -T git@github.com 2>&1 | grep "Roaming"
    - nulltype 11y ago
ygjb-dupe 11y ago
While the bug doesn't expose a user to mitm attacks in general, am I correct in thinking that folks using "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" have basically opened the door for that? https://github.com/search?q=-o+UserKnownHostsFile%3D%2Fdev%2Fnull&type=Code&utf8=%E2%9C%93 https://github.com/search?q=-o+UserKnownHostsFile%3D%2Fdev%2...
- tobiasu 11y ago
  Ughhh, that's insane.
  - the_mitsuhiko 11y ago
    > Ughhh, that's insane. Not really. This is for deploy systems which deploy to a trusted environment (for instance through VPN, network security etc.).
    - throwaway2048 11y ago
      Even if you are using some kind of prebaked images to deploy, you should be generating individual keys using the SSH PKI features per machine as part of your individual host configurations. This allows you to verify hosts while having never seen their keys. Just totally shutting off verification is a horrible idea.
      - the_mitsuhiko 11y ago
        If someone can actively MITM in your network you have other problems.
        throwaway2048 11y ago
        That argument has been shown to be nonsense for years, it can apply to any mitigation technology or privilege reduction technique. The point is to reduce the amount of harm that can be done, because "just write software without bugs" isn't a solution at all.
patio11 11y ago
Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places. SSH is designed so that, even if you connect to an evil host, the host only learns your public key, not your private key. This leaks your private key to an evil host. You might think "I only SSH into boxes that I own, so I'm good, right?", but if you in the future lose a box to the enemy, they can then use that box to grab your private key, and then use that key to get into every other box you can access with that keypair. This turns your sysadmin's photo blog hosted on Digital Ocean into a potential vector into your entire infrastructure, since there is a high likelihood they use the same private key for both. If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. You should expect adversaries to add an exploit for this to their rootkits. That would be bad enough since attackers targeting your infrastructure may exist, but this is also plausibly exploitable in a spray-and-pray fashion. Root any server on the Internet, using e.g. a WordPress vulnerability. Install key-stealing malicious SSH server. Forward all keys to your C&C infrastructure. Your infra does a reverse lookup from public key to Github account (trivial -- try "ssh whoami.filippo.io" if you don't believe me) and immediately optimistically tries to use the stolen key to log into the web tier of any site listed in their email/bio/Twitter profile/etc, pinging the attacker when it succeeds.
- rchiniquy 11y ago
  If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate. Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.
axelfontaine 11y ago
We went to other way and kissed SSH goodbye a long time ago in favor of immutable infrastructure and automation: https://boxfuse.com/blog/no-ssh https://boxfuse.com/blog/no-ssh
- feld 11y ago
  you've never had to debug anything in production, have you?
  - axelfontaine 11y ago
    It's not 2006 anymore. For the vast majority of the cases there is no need for SSH when it comes to logging, profiling and debugging.
fixermark 11y ago
So how does the exploit actually work? How can a malicious sshd actually use this to acquire the private key?
- d23 11y ago
  Yet another direct-link post to a cryptic CVE for those of us who aren't in the know on security issues. Upvoting is a great shibboleth to show that some posters "get it", while the rest of us are left in the dark on severity, left to wade through the comments section to try to get a sense of what the real impact is. From what I can tell, if I accidentally SSH to the wrong server (or a compromised one), my private key can be obtained. I have no clue if that's actually the correct interpretation.
  - Estragon 11y ago
    Oh, come on. The issue is very clear. Your interpretation is correct.
hamburglar 11y ago
How does a client bug end up leaking private key material to the server if the private key is in an entirely different process (the agent)? Or is this only an issue if you are loading your identity directly in the client?
- singlow 11y ago
  I would love to hear from someone who knows the answer to this. Does this affect ssh-agent? I am rotating my keys regardless, but I am curious.
  - TimWolla 11y ago
    According to: https://news.ycombinator.com/item?id=10902833 https://news.ycombinator.com/item?id=10902833 it does not: > Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.
joe_bleau 11y ago
Testing the ssh client config workaround: ssh -v user@localhost 2>&1 >/dev/null | grep -i 'roaming' returns "debug1: Roaming not allowed by server" when vulnerable, and nothing when not. YMMV, only tested on a few machines, etc.
- clort 11y ago
  does this require a ssh server running on localhost?
  - pmuk 11y ago
    Yes, that command is connecting to an ssh server on localhost, but you could connect to any ssh server that you trust... ssh -v -T git@github.com 2>&1 | grep -i 'roaming'
ChALkeR 11y ago
Note: this actually means that everyone should regenerate all their key-pairs after updating.
matt_wulfeck 11y ago
This is a surprisingly bad fail. I've never seen or heard of this option before. In fact, my vimlint for ssh_config doesn't even show it as a valid option!
kondbg 11y ago
Details: http://www.openwall.com/lists/oss-security/2016/01/14/7 http://www.openwall.com/lists/oss-security/2016/01/14/7
mjevans 11y ago
For shells where echo is a builtin that does not know about -e: printf 'Host *\nUseRoaming no\n' >> /etc/ssh/ssh_config
devin 11y ago
Am I correct that YubiKey users are unaffected by this vulnerability? As I understand it, the private key never enters into memory.
- feld 11y ago
  I guess it depends on what you mean? I have yubikey 2FA on my servers for SSH, but also have SSH keys. I use SSH keys when I can, and yubikey when I don't have my SSH keys handy. I suppose if you're doing yubikey+password auth and don't have any keys configured for your ssh client you're fine because... you don't have any keys? :)
  - Freaky 11y ago
    You can use a Yubikey NEO to handle key authentication on your computer's behalf: https://blog.habets.se/2013/02/GPG-and-SSH-with-Yubikey-NEO https://blog.habets.se/2013/02/GPG-and-SSH-with-Yubikey-NEO
    - feld 11y ago
      Sure, but this capability isn't unique to Yubikey. The question was just worded confusingly. Really it's just "ssh key on a smart card".
      - devin 11y ago
        Yes, my bad on the wording of the original question.
  - Xylakant 11y ago
    yubikeys support ssh key auth (via gpg-agent in ssh mode). It's sometimes a bit wobbly, but in general works fine.
    - devin 11y ago
      This is what I was referring to, and what I'm using, not 2FA. In this case, the PK stored on my YubiKey Neo never makes it into memory.
Corrado 11y ago
Just a note: the default AWS Linux AMI doesn't seem to have this problem on the server side. Connecting to one of my EC2 instances with verbose on I get the following message: debug1: Roaming not allowed by server Yeah! AWS Linux for the win. :)
- aexaey 11y ago
  This is client-only vulnerability: The matching server code has never been shipped, but the client code was enabled by default and could be tricked by a malicious server into leaking client memory to the server, including private client user keys.[1] [1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-January/034680.html https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...
- deleted 11y ago
  [deleted]
- Skunkleton 11y ago
  OpenSSH server doesn't support roaming. This is a client only issue. The problem is that your connection could be MITM'd by someone looking to exploit this bug.
  - hackuser 11y ago
    > your connection could be MITM'd MITM isn't a risk, if I understand this statement in the undeadly.org announcement: The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers.
    - kardos 11y ago
      Unless it's your first connection to a legit uncompromised server, yes? (AWS instance, etc)
- tomku 11y ago
  That message actually means that your client has roaming enabled and is therefore vulnerable.
ewindisch 11y ago
For the single other ChromeOS user (how many of us are there?), note your SSH client is also based on OpenSSH. Add '-o UseRoaming=no' to your arguments. Watch this repo for updates (or submit a PR?) https://chromium.googlesource.com/chromiumos/platform/assets/+log/master/chromeapps/ssh_client https://chromium.googlesource.com/chromiumos/platform/assets...
aexaey 11y ago
Does anybody else find this eerily reminiscent of heartbleed? Obscure broken feature that nobody uses (or needs) is enabled by default and allow private keys to leak.
- tomjen3 11y ago
  It is OpenSSH, the clown car of bugs in obscure features nobody uses. Does anybody know if LibraSSL has released an SSH client yet? I can't code well enough to make security code safe, so I won't be able to help them out on it, but it sounds more and more like OpenSSH can't either.
  - jldugger 11y ago
    You seem confused. The LibreSSL people released a SSH client years ago, called OpenSSH. Which is not the same as OpenSSL, which suffered Heartbleed.
    - Gigablah 11y ago
      I can't blame him, considering that people have been encouraging this sort of attitude towards the OpenSSL maintainers. It's time they tasted their own medicine.
sly010 11y ago
Is there an easy way to verify that the workaround is effective? ssh -v doesn't display anything related.
- paulannesley 11y ago
  ssh SOMEHOST -v 2>&1 1>/dev/null | grep -i roaming A vulnerable client will show: debug1: Roaming not allowed by server
peterwwillis 11y ago
So, if there was forgotten code in OpenSSH (which, how the fuck does that happen?), where else might there be forgotten code? Also: this is a really great example of why you should never have undocumented features in your code.
sgtnasty 11y ago
Does it affect RHEL 6, using openssh 5.3?
sygma 11y ago
Patch for OS X: echo -e 'Host *\nUseRoaming no' >> ~/.ssh/config
mag00 11y ago
I'm curious how typo and bit squatting would come into play here, and if attacks leveraging them could collect private keys at a dangerously high rate before people can patch their clients. Products like heroku, or the stripe CTF, or other things that come to mind that operate over SSH going rogue a bit scarier. If one were to be compromised it would be a case where mass amounts of private keys could leak. AWS, github, all cloud VPS providers, etc. Multifactor is relevant as a defense with a vulnerability like this.
- deleted 11y ago
  [deleted]
eljimmy 11y ago
Some good news was added to the updates: > The agent will never send a private key over its request channel. Instead, operations that require a private key will be performed by the agent, and the result will be returned to the requester. This way, private keys are not exposed to clients using the agent.
slasaus 11y ago
My hardened ~/.ssh/config on OS X 10.11: Host * Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 HostKeyAlgorithms ssh-ed25519,ssh-rsa ChallengeResponseAuthentication no UseRoaming no If you only connect to newer servers you can further restrict ciphers to only use AEADs (only list the chacha20-poly1305 and aes-gcm ciphers). I assume using AEADs-only makes the MACs keyword obsolete, is this correct? Config is based on tips from https://stribika.github.io/2015/01/04/secure-secure-shell.html https://stribika.github.io/2015/01/04/secure-secure-shell.ht...
- ehPReth 11y ago
  Mozilla also maintains a SSH guide for both clients and servers: https://wiki.mozilla.org/Security/Guidelines/OpenSSH https://wiki.mozilla.org/Security/Guidelines/OpenSSH
- zackelan 11y ago
  > I assume using AEADs-only makes the MACs keyword obsolete, is this correct? Correct. You can test this by running SSH with -v/--verbose and observing these log lines: debug1: kex: server->client aes256-gcm@openssh.com <implicit> none debug1: kex: client->server aes256-gcm@openssh.com <implicit> none When using a cipher that does not support AEAD, the 2nd field will include the MAC instead of <implicit> debug1: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com none debug1: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com none I wasn't sure what the 3rd "none" field is, but after some digging it appears to be the compression algorithm: https://github.com/openssh/openssh-portable/blob/master/kex.c#L818 https://github.com/openssh/openssh-portable/blob/master/kex.... And I confirmed this with ssh -C: debug1: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com zlib@openssh.com debug1: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com zlib@openssh.com
- grawlinson 11y ago
  Funny story, I erroneously threw `UseRoaming no` on my server config and freaked the fuck out when I couldn't get into via SSH. Six rescue sessions later, I figured out it shouldn't have been there ... In the future, I'm definitely going to read documentation & test config files (sshd -t -f /path/to/sshd_config, in this case) EDIT: Moral of the story, I am not a smart person sometimes.
ams6110 11y ago
CentOS/RHEL 6.7 has: $ ssh -V OpenSSH_5.3p1 OpenSSL 1.0.1e-fips 11 Feb 2013 So... not vulnerable? Posted article says: This affects OpenSSH versions 5.4 through 7.1.
kevincox 11y ago
Is there an explanation of why the roaming feature was removed from the server? It sounds like a useful feature to me and I run into this all the time.
betaby 11y ago
Could we say it partially because of mono-culture? Almost no one uses GNU lsh as a client/server, dropbear is mostly used as a embedded system sshd. OpenSSH has to many thing integrated and enabled by default.
nightfox0205 11y ago
After adding in config, Do we need to rota/generate ssh keys also ?
nightfox0205 11y ago
After edit config file , Do we need to rotated/genrate ssh keys also?
Estragon 11y ago
So has anyone run git-blame yet? Who did this?
LinuxBender 11y ago
This wasn't the only feature that was activated without much documentation. MaxSessions has a significant impact on any systems using 2FA (as the default allows bypassing it if you are phished). Just like unrestricted sudo, people have because accustom to the bad default behavior of MaxSessions being 10 and allowing un-authenticated multiplexing. (meaning, you auth once, and my trojan can use your session without authentication)
Jadestuart1 11y ago
Having been through some difficult times when i had been let down by everyone around me, i came across Brandon Reid who is a hacker its actually impossible to put in words how much of a Genius he is and also cant stop thanking him for helping me through my divorce, and also my Nephews grades. He is a Blackhat hacker and very capable of almost and everything, he is actually one of the best out there and also shows you proof before charging you, in my own case the money wasnt the problem and i can gladly say every money spent was so worth it. I have reffered a number of people who will prefer not to be named to Brandon and all have been immensly satisfied with the Top Notch hacking service Brandon offers.To whoever who is lucky enough to read this its a new year and i am only doing this for those genuine people out there who would want the services of a hacker please as i said before be careful how you speak to him his a BigFish. He can be contacted on his services email Brandonreid001@gmail.com or text +1 813 379 2141 . Do mention the name Jade

617 points by moviuro 11y ago

Workaround (yes, it's client-side): # echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config Disclaimer: won't work on all operating systems, shells, etc. YMMV. Consult a doctor before following any advice you get from the Internet. Void where prohibited. Restrictions may apply. Edited per comments below

pmoriarty 11y ago
Would this work too? $ echo "UseRoaming no" >> ~/.ssh/config
- Zr40 11y ago
  Yes, but it would not apply to other users.
- iamtew 11y ago
  Yes, but it'll only affect that user.
  - ralphm 11y ago
    And only the last `Host` entry in that config, if present.
doogle88 11y ago
Presumably you would have to connect to a malicious host to be effected? Or perhaps a MITM on your connection to a legit host can exploit you somehow.
- masklinn 11y ago
  > Presumably you would have to connect to a malicious host to be effected? Malicious or compromised. > roaming code in the ssh client could be tricked by a hostile sshd server, potentially leaking key material.
- amatix 11y ago
  From the updated OP: > The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers.
  - jon-wood 11y ago
    Malicious, compromised, or new servers. Because really, how many people check the host key for a newly spun up EC2 instance?
ralphm 11y ago
Don't ever fix your `ssh_config` by appending stuff to the end of the file. The configuration syntax allows for block constructs without an explicit end marker (like `Match` and `Host`). Appending will cause all kinds of sadness.
- gnur 11y ago
  It's from the linked page.
  - ralphm 11y ago
    Indeed. The page gives bad advise. If your config has `Host` blocks, like often in people's personal configs, or if you have a `Match` block, the new directive only applies to the last of those blocks in the config file.
    - snowwrestler 11y ago
      Sufficient to place it at the top of ~/.ssh/config? Or does it also need a block header of its own, like: Host * UseRoaming no
      - skuhn 11y ago
        I confirmed that it does not need to be placed into a Host block, as long as it is above all of your blocks. For instance: UseRoaming no Host * Blah yes Test with: ssh -v remote.ssh.host.com uptime 2>&1 | grep -i roaming If it returns nothing, the config fix is active. If it isn't active, you'll see 'debug1: Roaming not allowed by server'
    - jldugger 11y ago
      Which is why they're appending a newline and Host *?
      - ralphm 11y ago
        Note that the example was changed as a result of my initial comment.
narsil 11y ago
I'm emailing the one-liner to folks and found it helpful to just include the root subshell: sudo bash -c 'echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config'

moviuro 11y ago

The most terrible thing here is that OpenBSD usually does some full disclosure. This is sure going to be a nasty bug.

praseodym 11y ago
Why is this terrible? In any case, the software is open source so any security fix commit can also be considered as full disclosure.
- kardos 11y ago
  But they didn't disclose the bug yet. The implication is that this bug is /so bad/ that it justified breaking the normal "just disclose it" approach.
- deleted 11y ago
  [deleted]

Aissen 11y ago

Roaming feature has been in OpennSSH code for a while, but is undocumented: http://superuser.com/a/826734/47771 http://superuser.com/a/826734/47771

DrRobinson 11y ago
"Affects all OpenSSH 5.4 - 7.1". 5.4 was released March 2010 it seems
ryanlol 11y ago
AFAIK the roaming feature has been disabled on the server side for a while, but mistakenly left enabled on the client.
- DrRobinson 11y ago
  Seems correct: https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2 https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2

merqurio 11y ago

As far as I know, this will affect any OSX, am I right ?

moviuro 11y ago
s/X//
deleted 11y ago
[deleted]
- ryanlol 11y ago
  This does not affect the sshd.
ryanlol 11y ago
According to Theo, yes.
DrRobinson 11y ago
It affects all operating systems: http://marc.info/?l=openbsd-tech&m=145278077820529&w=2 http://marc.info/?l=openbsd-tech&m=145278077820529&w=2
- merqurio 11y ago
  Is there any way to update openssh on my side, without waiting to apple ? Is this enough ?https://mochtu.de/2015/01/07/updating-openssh-on-mac-os-x-10-10-yosemite/ https://mochtu.de/2015/01/07/updating-openssh-on-mac-os-x-10...
  - DrRobinson 11y ago
    You can patch it with: # echo 'UseRoaming no' >> /etc/ssh/ssh_config or $ echo "UseRoaming no" >> ~/.ssh/config
    - mapgrep 11y ago

ryanlol 11y ago

Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264 https://twitter.com/marver/status/687644904575627264 Edit 1: Here's some relevant commits too https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2 https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2 Edit 2: This mailing list post seems to discuss the vulnerable feature http://www.gossamer-threads.com/lists/openssh/dev/49018?do=post_view_threaded#49018 http://www.gossamer-threads.com/lists/openssh/dev/49018?do=p... Edit 3: Got a better description of actual impact of the bugs: >Experimental roaming code in the ssh client could be tricked by a hostile sshd >server, potentially leaking key material. CVE-2016-077 and CVE-0216-078. >Prevent this problem immediately by adding the line "UseRoaming no" to >/etc/ssh/ssh_config.

masklinn 11y ago
> Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264 https://twitter.com/marver/status/687644904575627264 The subsystem was apparently experimentally (hence the undocumented option) introduced 6 years ago and essentially never used, the server support was never implemented: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/roaming_serv.c http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/roa...
sarciszewski 11y ago
> >server, potentially leaking key material. CVE-2016-077 and CVE-0216-078. Presumably you meant CVE-2016-0777 and CVE-2016-0778
- ryanlol 11y ago
  I'm just quoting OFFICIAL SOURCES here.
kardos 11y ago
So the real question is, can a MITM intercept connections to boxen you frequent to exploit this? Or is it limited to connecting to hostile honeypots?

noir_lord 11y ago

http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/010_ssh.patch.sig http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/010_ss...

fulafel 11y ago
Leaking private keys seems to be all the rage these days.

rnhmjoj 11y ago

Does this affect mosh?

fulafel 11y ago
No, the roaming in Mosh is unrelated to OpenSSH's roaming feature.
- sneak 11y ago
  Irrelevant; it still affects mosh as mosh does an ssh connection first to do auth and key exchange. That ssh session is potentially vulnerable when connecting to a malicious or compromised host.
- TazeTSchnitzel 11y ago
  Doesn't mosh initially establish a connection via SSH, though?
- teamhappy 11y ago
  But Mosh uses OpenSSH for authentication. And if roaming is enabled by default on the client this could still be an issue, no? I'd say it's still a good idea to add "UseRoaming no" to ssh_config.
- rakoo 11y ago
  But authentication in Mosh still relies on OpenSSH, so auth could be intercepted by an attacker after which Mosh is completely open to them: https://mosh.mit.edu/ https://mosh.mit.edu/ > However, in typical usage, Mosh relies on SSH to exchange keys at the beginning of a session, so Mosh will inherit the weaknesses of SSH—at least insofar as they affect the brief SSH session that is used to set up a long-running Mosh session.
- XMPPwocky 11y ago
  Right, but mosh uses ssh for initial key setup. I suspect it is possible to exploit that. (As an occasional Windows user who's using the Mosh Chrome extension, I have no idea how much risk I'm at; can't edit ssh_config, it's not even exposed)

StefanKarpinski 11y ago

Does it strike anyone else as bizarre / poor form for an experimental feature to be enabled by default in OpenSSH, which is normally very conservative with option defaults?

agwa 11y ago
What's more, it was not documented in the ssh_config(5) man page. If it had been, I probably would have disabled it long ago when hardening my SSH config. Time to head to the source to look for other undocumented options... Update: my findings are here (scroll to bottom for the upshot): https://gist.github.com/AGWA/e92d4f5343be1f7a941d https://gist.github.com/AGWA/e92d4f5343be1f7a941d UseRoaming is the only one to be concerned about. There are many other undocumented options, but they're all aliases for a documented option or are deprecated/unsupported.
- noja 11y ago
  Please report back what you find!
- ryanlol 11y ago
  https://github.com/openssh/openssh-portable/blob/e6c85f8889c5c9eb04796fdb76d2807636b9eef5/readconf.c https://github.com/openssh/openssh-portable/blob/e6c85f8889c... Here you go
  - ryanlol 11y ago
    Actually, I quickly generated a list of undocumented config options: http://sprunge.us/QDSE http://sprunge.us/QDSE Edit: Fixed version http://sprunge.us/LVYB http://sprunge.us/LVYB
    - vesinisa 11y ago
      most of those seem to be documented: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/ssh_config.5?query=ssh_config&sec=5 http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/... I did a quick comparison between the OpCodes enum and the latest man page, and `UseRoaming` is the only real undocumented option.

ctstover 11y ago

What was this experimental / undocumented roaming feature even suppose to do?

noja 11y ago
Complete guess: handle changes of ip address on the client side.
- jlgaddis 11y ago
  Allow a mobile user to reconnect to a disconnected session without (user) re-authentication.

jlgaddis 11y ago

ETA: I was way off... you can ignore this... :-) cf. section 3.3.5 [0], which describes "Roaming (Suspend/Resume)". This is documentation for an application by a company called AppGate (later acquired by Cryptzone) that wrote {some|most} of the code in OpenSSH's "roaming_client.c". This gives a hint of what the ramifications may be: basically, a MITM, who observed the initial session negotiation, can disconnect the client and hijack an active session. > Roaming is a feature which allows clients to suspend the connection to the AppGate server and later to resume it again. The user does not need to re-authenticate when reconnecting. Indeed, the entire process can be completely automated and nearly invisible to the user. All established connections will remain alive while roaming. This feature is intended for mobile users who move around between networks. > Technically, roaming is accomplished by closing the TCP tunnel when the connection is suspended. When resuming, a new TCP connection is made to the server and the SSH data stream is continued through this new connection. The user does not need to authenticate again, instead the client authenticates to the server, without user interaction, with the help of a random password which was made up when the user authenticated at the start of the session. In addition to knowing this password, the client must also know the encryption keys and encryption state to be able to reconnect. It is therefore impossible for a third party to break in and take over a suspended session. I particularly like that last sentence. [0]: http://download.cryptzone.com/files/download/AppGate-10.2.3/doc/manual_chunked_html/ch03s03.html http://download.cryptzone.com/files/download/AppGate-10.2.3/...

ryanlol 11y ago
That's not correct though, MITM cannot exploit this. A malicious SSH server can exploit this during a connection though, but only after host key verification.
hultner 11y ago
Oh Cryptzone, funny they offered me a job a few years back. Interesting that this was left in OpenSSH like this, even more as a default.

Aissen 11y ago

Release is here, with many more details, and other security issues fixed: https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-January/034680.html https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...

SFjulie1 11y ago

theo must be mad. Shaming people for leaving useless non essential feature in their code that results in security breach. And now the jewel of his crown has been compromised. The funniest part is now that his jewel has been tarnished, maybe people will understand what he was saying. And maybe too, people that believed privacy can be achieved on the internet will finally look at the problem of believing the 2 general paradox can be solved without at least 2 different constant link on different plans. And the problem is belief is a poor substitute for thinking - critical thinking. And maybe people will discover the sad truth of the internet. Security requires a perfect world, where human beings neither makes mistakes nor are corruptible. Errare humanum est, perseverare diabolicum Oh! Some says that is what 2 factor authentication is. I will answer, my intuition is telling me that 2 factor is good for a fixed amount of time/information and that using it correctly would annoy people to the utmost points. Then people would say well let's accept that fraud exists. Business first. (costs/benefits) Then I say giving 3% of all e-commerce to the bad guys is like admitting organized crime have a strong budget for even more crime ... and that we are fucked. Unless you don't understand that ISIS is basically a startup. A startup that overthrow a state to make even more money and industrialize crime.

marios 11y ago
What Theo (and other OpenBSD developers) have been saying all these years is that it's impossible to /not/ make mistakes, which is why sane design and exploit mitigations are important. Mad ? Maybe, I wouldn't know. On the other hand, I bet he's really glad all that effort to have ASLR by default was made, because it makes it more difficult for an attacker to exploit vulnerabilites such as this one.
- SFjulie1 11y ago
  ASLR makes exploits more difficult as long as you have true randomness. It is just a mitigation ... for another problem. In this case the elephant in the room is stack injection and dynamic libraries. If processes where confined to a well known address space that was self contained ASLR would be useless.. But dependency management would be hellish. Back to the case. OpenSSH has been openly criticize by 9plan teams for exactly the same reason openSSL has been criticized by openSSH team : too much complexity (not to say plan9 came to anything usable and were right (2 wrongs do not make a right)). http://harmful.cat-v.org/software/ssh http://harmful.cat-v.org/software/ssh I do have a feeling as a physicist that the S in CS stands for Shortness of thinking. What is obscure is not profound. And, at the opposite of engineers I don't believe that security is achievable at all on computers. It is like believing there exists a way to avoid triangulation with one strong radio source. Computers leak to much information (especially in the physical world), and C is a map that is taken too much for the territory. Every bugs are exploiting a wrong mapping of concepts to implementation. Modern security is like string theory, admired by everyone because it requires great technical knowledge to master, understood by none because it is way to complex for our "human" brains. We live in an era of belief in solutions. A bit of critical thinking and of distrusts of experts and stuff that you cannot understand without devoting your life to a subject is at my opinion a must. I distrust mathematicians for their capacity of dealing with the real world and its uncertainties, and cryptography (as much as functional programming, big data, algorithmic, IA, machine learning) is math driven in a pure Aristotelian world. Where perfection and harmony is the pillar of thinking. I come from micro-electronics, I only see wires (that are antennas), oscillators, multiplexers, gates and basically a dumb automat I can automatize in respect to time of propagation of signal, and I know that modern computers under the hood are in the physical realm of relativity with approximate answers. I theorize, build, measure, and retheorize ad nauseum until the product is measured to work the way I expect it to in the domain of validity with a good enough confidence and margins of errors are always on my mind to be controled. Security requires a zero margin of ambiguity. Physical world is bound to heisenberg equation and coupling. Purity does not exist and this cannot be mitigated. The postulate of cryptography are wrong from the core. Real world always win at the end. Aristotle way of thinking must die. Math is not science.

deleted 11y ago

[deleted]

ck2 11y ago

If it's not obvious, don't just add that config option, you have to also restart it. Actually wait, is this only affecting the ssh client and not the server/daemon ?

jlgaddis 11y ago
Yes, it's client-side only. Server-side, it's disabled (cf. readconf.c).

sharjeel 11y ago

Since this is a client side issue, can this be used to exploit those automated scanners who try to break into your SSH machine?

deleted 11y ago
[deleted]
dsr_ 11y ago
No. The scanners are looking for password-accessible accounts, not keyed accounts. The scanners won't have useful keys, nor listening ssh daemons.
Stefan-H 11y ago
Authenticated scanners that use key auth like Qualys' security appliances could have private keys that are valid across the organization, and if using an affected client version, could leak this information to a malicious system on your network.

protomyth 11y ago

Check me on this, so this is a client-side problem only, so ssh-ing into only know servers shouldn't be an issue and clients cannot cause problems for servers?

scardine 11y ago
If an enemy takes control of just one of the hosts you ssh into, he will get your private key and can use it to ssh into any other box where you use RSAAuthentication.
- protomyth 11y ago
  I already added the mitigation, but I wondering if my servers can be patched on the weekend.
  - drummer32 11y ago
    There is nothing to patch on the server side. You need to ensure the ssh client is updated on the machines you are sshing from.
    - protomyth 11y ago
      ok thanks, that's what I figured, but it I was getting a bit worried that I was missing something.

nulltype 11y ago

If you have a mac (Yosemite), it looks like you want to add " UseRoaming no" under the "Host *" line in /etc/ssh_config (as root). You can test it before and after with this: ssh -v -T git@github.com 2>&1 | grep Roaming debug1: Roaming not allowed by server <- bad ssh -v -T git@github.com 2>&1 | grep Roaming (no output is good)

super_mario 11y ago
On El Capitan the file is /etc/ssh/ssh_config
- snowwrestler 11y ago
  On Mountain Lion it's /private/etc/ssh_config.
  - astrange 11y ago
    /etc points to /private/etc on OS X, so it's all good.
deleted 11y ago
[deleted]
lftl 11y ago
Interestingly I haven't been able to get the Roaming not allowed message to show on my client with or without the config option set.
- antoinealb 11y ago
  Did you copy the whole line as "g ssh -v -T git@github.com 2>&1 | grep "Roaming debug1: Roaming not allowed by server"" ? I misread GP's command and made that mistake. The correct command is : ssh -v -T git@github.com 2>&1 | grep "Roaming"
  - nulltype 11y ago

ygjb-dupe 11y ago

While the bug doesn't expose a user to mitm attacks in general, am I correct in thinking that folks using "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" have basically opened the door for that? https://github.com/search?q=-o+UserKnownHostsFile%3D%2Fdev%2Fnull&type=Code&utf8=%E2%9C%93 https://github.com/search?q=-o+UserKnownHostsFile%3D%2Fdev%2...

tobiasu 11y ago
Ughhh, that's insane.
- the_mitsuhiko 11y ago
  > Ughhh, that's insane. Not really. This is for deploy systems which deploy to a trusted environment (for instance through VPN, network security etc.).
  - throwaway2048 11y ago
    Even if you are using some kind of prebaked images to deploy, you should be generating individual keys using the SSH PKI features per machine as part of your individual host configurations. This allows you to verify hosts while having never seen their keys. Just totally shutting off verification is a horrible idea.
    - the_mitsuhiko 11y ago
      If someone can actively MITM in your network you have other problems.
      - throwaway2048 11y ago
        That argument has been shown to be nonsense for years, it can apply to any mitigation technology or privilege reduction technique. The point is to reduce the amount of harm that can be done, because "just write software without bugs" isn't a solution at all.

patio11 11y ago

Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places. SSH is designed so that, even if you connect to an evil host, the host only learns your public key, not your private key. This leaks your private key to an evil host. You might think "I only SSH into boxes that I own, so I'm good, right?", but if you in the future lose a box to the enemy, they can then use that box to grab your private key, and then use that key to get into every other box you can access with that keypair. This turns your sysadmin's photo blog hosted on Digital Ocean into a potential vector into your entire infrastructure, since there is a high likelihood they use the same private key for both. If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. You should expect adversaries to add an exploit for this to their rootkits. That would be bad enough since attackers targeting your infrastructure may exist, but this is also plausibly exploitable in a spray-and-pray fashion. Root any server on the Internet, using e.g. a WordPress vulnerability. Install key-stealing malicious SSH server. Forward all keys to your C&C infrastructure. Your infra does a reverse lookup from public key to Github account (trivial -- try "ssh whoami.filippo.io" if you don't believe me) and immediately optimistically tries to use the stolen key to log into the web tier of any site listed in their email/bio/Twitter profile/etc, pinging the attacker when it succeeds.

rchiniquy 11y ago
If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive. As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate. Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

axelfontaine 11y ago

We went to other way and kissed SSH goodbye a long time ago in favor of immutable infrastructure and automation: https://boxfuse.com/blog/no-ssh https://boxfuse.com/blog/no-ssh

feld 11y ago
you've never had to debug anything in production, have you?
- axelfontaine 11y ago
  It's not 2006 anymore. For the vast majority of the cases there is no need for SSH when it comes to logging, profiling and debugging.

fixermark 11y ago

So how does the exploit actually work? How can a malicious sshd actually use this to acquire the private key?

d23 11y ago
Yet another direct-link post to a cryptic CVE for those of us who aren't in the know on security issues. Upvoting is a great shibboleth to show that some posters "get it", while the rest of us are left in the dark on severity, left to wade through the comments section to try to get a sense of what the real impact is. From what I can tell, if I accidentally SSH to the wrong server (or a compromised one), my private key can be obtained. I have no clue if that's actually the correct interpretation.
- Estragon 11y ago
  Oh, come on. The issue is very clear. Your interpretation is correct.

hamburglar 11y ago

How does a client bug end up leaking private key material to the server if the private key is in an entirely different process (the agent)? Or is this only an issue if you are loading your identity directly in the client?

singlow 11y ago
I would love to hear from someone who knows the answer to this. Does this affect ssh-agent? I am rotating my keys regardless, but I am curious.
- TimWolla 11y ago
  According to: https://news.ycombinator.com/item?id=10902833 https://news.ycombinator.com/item?id=10902833 it does not: > Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

joe_bleau 11y ago

Testing the ssh client config workaround: ssh -v user@localhost 2>&1 >/dev/null | grep -i 'roaming' returns "debug1: Roaming not allowed by server" when vulnerable, and nothing when not. YMMV, only tested on a few machines, etc.

clort 11y ago
does this require a ssh server running on localhost?
- pmuk 11y ago
  Yes, that command is connecting to an ssh server on localhost, but you could connect to any ssh server that you trust... ssh -v -T git@github.com 2>&1 | grep -i 'roaming'

ChALkeR 11y ago

Note: this actually means that everyone should regenerate all their key-pairs after updating.

matt_wulfeck 11y ago

This is a surprisingly bad fail. I've never seen or heard of this option before. In fact, my vimlint for ssh_config doesn't even show it as a valid option!

kondbg 11y ago

Details: http://www.openwall.com/lists/oss-security/2016/01/14/7 http://www.openwall.com/lists/oss-security/2016/01/14/7

mjevans 11y ago

For shells where echo is a builtin that does not know about -e: printf 'Host *\nUseRoaming no\n' >> /etc/ssh/ssh_config

devin 11y ago

Am I correct that YubiKey users are unaffected by this vulnerability? As I understand it, the private key never enters into memory.

feld 11y ago
I guess it depends on what you mean? I have yubikey 2FA on my servers for SSH, but also have SSH keys. I use SSH keys when I can, and yubikey when I don't have my SSH keys handy. I suppose if you're doing yubikey+password auth and don't have any keys configured for your ssh client you're fine because... you don't have any keys? :)
- Freaky 11y ago
  You can use a Yubikey NEO to handle key authentication on your computer's behalf: https://blog.habets.se/2013/02/GPG-and-SSH-with-Yubikey-NEO https://blog.habets.se/2013/02/GPG-and-SSH-with-Yubikey-NEO
  - feld 11y ago
    Sure, but this capability isn't unique to Yubikey. The question was just worded confusingly. Really it's just "ssh key on a smart card".
    - devin 11y ago
      Yes, my bad on the wording of the original question.
- Xylakant 11y ago
  yubikeys support ssh key auth (via gpg-agent in ssh mode). It's sometimes a bit wobbly, but in general works fine.
  - devin 11y ago
    This is what I was referring to, and what I'm using, not 2FA. In this case, the PK stored on my YubiKey Neo never makes it into memory.

Corrado 11y ago

Just a note: the default AWS Linux AMI doesn't seem to have this problem on the server side. Connecting to one of my EC2 instances with verbose on I get the following message: debug1: Roaming not allowed by server Yeah! AWS Linux for the win. :)

aexaey 11y ago
This is client-only vulnerability: The matching server code has never been shipped, but the client code was enabled by default and could be tricked by a malicious server into leaking client memory to the server, including private client user keys.[1] [1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-January/034680.html https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...
deleted 11y ago
[deleted]
Skunkleton 11y ago
OpenSSH server doesn't support roaming. This is a client only issue. The problem is that your connection could be MITM'd by someone looking to exploit this bug.
- hackuser 11y ago
  > your connection could be MITM'd MITM isn't a risk, if I understand this statement in the undeadly.org announcement: The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers.
  - kardos 11y ago
    Unless it's your first connection to a legit uncompromised server, yes? (AWS instance, etc)
tomku 11y ago
That message actually means that your client has roaming enabled and is therefore vulnerable.

ewindisch 11y ago

For the single other ChromeOS user (how many of us are there?), note your SSH client is also based on OpenSSH. Add '-o UseRoaming=no' to your arguments. Watch this repo for updates (or submit a PR?) https://chromium.googlesource.com/chromiumos/platform/assets/+log/master/chromeapps/ssh_client https://chromium.googlesource.com/chromiumos/platform/assets...

aexaey 11y ago

Does anybody else find this eerily reminiscent of heartbleed? Obscure broken feature that nobody uses (or needs) is enabled by default and allow private keys to leak.

tomjen3 11y ago
It is OpenSSH, the clown car of bugs in obscure features nobody uses. Does anybody know if LibraSSL has released an SSH client yet? I can't code well enough to make security code safe, so I won't be able to help them out on it, but it sounds more and more like OpenSSH can't either.
- jldugger 11y ago
  You seem confused. The LibreSSL people released a SSH client years ago, called OpenSSH. Which is not the same as OpenSSL, which suffered Heartbleed.
  - Gigablah 11y ago
    I can't blame him, considering that people have been encouraging this sort of attitude towards the OpenSSL maintainers. It's time they tasted their own medicine.

sly010 11y ago

Is there an easy way to verify that the workaround is effective? ssh -v doesn't display anything related.

paulannesley 11y ago
ssh SOMEHOST -v 2>&1 1>/dev/null | grep -i roaming A vulnerable client will show: debug1: Roaming not allowed by server

peterwwillis 11y ago

So, if there was forgotten code in OpenSSH (which, how the fuck does that happen?), where else might there be forgotten code? Also: this is a really great example of why you should never have undocumented features in your code.

sgtnasty 11y ago

Does it affect RHEL 6, using openssh 5.3?

sygma 11y ago

Patch for OS X: echo -e 'Host *\nUseRoaming no' >> ~/.ssh/config

mag00 11y ago

I'm curious how typo and bit squatting would come into play here, and if attacks leveraging them could collect private keys at a dangerously high rate before people can patch their clients. Products like heroku, or the stripe CTF, or other things that come to mind that operate over SSH going rogue a bit scarier. If one were to be compromised it would be a case where mass amounts of private keys could leak. AWS, github, all cloud VPS providers, etc. Multifactor is relevant as a defense with a vulnerability like this.

deleted 11y ago
[deleted]

eljimmy 11y ago

Some good news was added to the updates: > The agent will never send a private key over its request channel. Instead, operations that require a private key will be performed by the agent, and the result will be returned to the requester. This way, private keys are not exposed to clients using the agent.

slasaus 11y ago

My hardened ~/.ssh/config on OS X 10.11: Host * Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 HostKeyAlgorithms ssh-ed25519,ssh-rsa ChallengeResponseAuthentication no UseRoaming no If you only connect to newer servers you can further restrict ciphers to only use AEADs (only list the chacha20-poly1305 and aes-gcm ciphers). I assume using AEADs-only makes the MACs keyword obsolete, is this correct? Config is based on tips from https://stribika.github.io/2015/01/04/secure-secure-shell.html https://stribika.github.io/2015/01/04/secure-secure-shell.ht...

ehPReth 11y ago
Mozilla also maintains a SSH guide for both clients and servers: https://wiki.mozilla.org/Security/Guidelines/OpenSSH https://wiki.mozilla.org/Security/Guidelines/OpenSSH
zackelan 11y ago
> I assume using AEADs-only makes the MACs keyword obsolete, is this correct? Correct. You can test this by running SSH with -v/--verbose and observing these log lines: debug1: kex: server->client aes256-gcm@openssh.com <implicit> none debug1: kex: client->server aes256-gcm@openssh.com <implicit> none When using a cipher that does not support AEAD, the 2nd field will include the MAC instead of <implicit> debug1: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com none debug1: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com none I wasn't sure what the 3rd "none" field is, but after some digging it appears to be the compression algorithm: https://github.com/openssh/openssh-portable/blob/master/kex.c#L818 https://github.com/openssh/openssh-portable/blob/master/kex.... And I confirmed this with ssh -C: debug1: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com zlib@openssh.com debug1: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com zlib@openssh.com
grawlinson 11y ago
Funny story, I erroneously threw `UseRoaming no` on my server config and freaked the fuck out when I couldn't get into via SSH. Six rescue sessions later, I figured out it shouldn't have been there ... In the future, I'm definitely going to read documentation & test config files (sshd -t -f /path/to/sshd_config, in this case) EDIT: Moral of the story, I am not a smart person sometimes.

ams6110 11y ago

CentOS/RHEL 6.7 has: $ ssh -V OpenSSH_5.3p1 OpenSSL 1.0.1e-fips 11 Feb 2013 So... not vulnerable? Posted article says: This affects OpenSSH versions 5.4 through 7.1.

kevincox 11y ago

Is there an explanation of why the roaming feature was removed from the server? It sounds like a useful feature to me and I run into this all the time.

betaby 11y ago

Could we say it partially because of mono-culture? Almost no one uses GNU lsh as a client/server, dropbear is mostly used as a embedded system sshd. OpenSSH has to many thing integrated and enabled by default.

nightfox0205 11y ago

After adding in config, Do we need to rota/generate ssh keys also ?

nightfox0205 11y ago

After edit config file , Do we need to rotated/genrate ssh keys also?

Estragon 11y ago

So has anyone run git-blame yet? Who did this?

LinuxBender 11y ago

This wasn't the only feature that was activated without much documentation. MaxSessions has a significant impact on any systems using 2FA (as the default allows bypassing it if you are phished). Just like unrestricted sudo, people have because accustom to the bad default behavior of MaxSessions being 10 and allowing un-authenticated multiplexing. (meaning, you auth once, and my trojan can use your session without authentication)

Jadestuart1 11y ago

Having been through some difficult times when i had been let down by everyone around me, i came across Brandon Reid who is a hacker its actually impossible to put in words how much of a Genius he is and also cant stop thanking him for helping me through my divorce, and also my Nephews grades. He is a Blackhat hacker and very capable of almost and everything, he is actually one of the best out there and also shows you proof before charging you, in my own case the money wasnt the problem and i can gladly say every money spent was so worth it. I have reffered a number of people who will prefer not to be named to Brandon and all have been immensly satisfied with the Top Notch hacking service Brandon offers.To whoever who is lucky enough to read this its a new year and i am only doing this for those genuine people out there who would want the services of a hacker please as i said before be careful how you speak to him his a BigFish. He can be contacted on his services email Brandonreid001@gmail.com or text +1 813 379 2141 . Do mention the name Jade