6 ms·

If you look at the diagram of privsep, the authentication process is part of the privileged binary, which is where this RCE lives http://www.citi.umich.edu/u/p

by treasy 3y ago

If you look at the diagram of privsep, the authentication process is part of the privileged binary, which is where this RCE lives http://www.citi.umich.edu/u/provos/ssh/priv.jpg http://www.citi.umich.edu/u/provos/ssh/priv.jpg

cryptonector 3y ago
The signature validation could be moved into an unprivileged process forked from that one.
- littlestymaar 3y ago
  That's an easy thing to say after the fact indeed but yes. In fact after such a disastrous backdoor I wouldn't be surprised if OpenSSH moved all code calling external libraries to unprivileged processes to make sure such an attack can never have such a dramatic effect (an auth bypass would still likely be possible, but that's still way better than a root RCE…). At this point “All libraries could be malicious” is a threat model that must be considered for something as security critical as OpenSSH.
  - asveikau 3y ago
    I don't think that's a threat model that OpenSSH should waste too much time on. Ultimately this is malicious code in the build machine compiling a critical system library. That's not reasonable to defend against. Keep in mind that upstream didn't even link to liblzma. Debian patched it to do so. OpenSSH should defend against that too?
    - CanaryLayout 3y ago
      any one of us if we sat on the OSSH team would flip the middle finger. What code is the project supposed to write when nothing on main dyn loaded liblzma. It was brought in from a patch they don't have realistic control over. This is a Linux problem, and the problem is systemd, which is who brought the lib into memory and init'd it.
      - shzhdbi09gv8ioi 3y ago
        > This is a Linux problem, and the problem is systemd, which is who brought the lib into memory and init'd it. Not at all, it is a distro issue because a few distros such as Debian chose to patch openssh to bring in systemd support [1]. Other systemd-based distros like Arch Linux remains unaffected because they don't carry this patch. 1: https://sources.debian.org/src/openssh/1%3A9.7p1-2/debian/patches/systemd-readiness.patch/ https://sources.debian.org/src/openssh/1%3A9.7p1-2/debian/pa...
        b112 3y ago
        Yet Redhat and others applied this patch, as systemd is so incapable of reliably launching processes, that it kept killing sshd without it. What a complete failure of an init system's job, and the patch was applied due to systemd not resolving the issue in another way. This is the problem with systemd. Way, way way too much complexity.
        streb-lo 3y ago
        systemd is just fine on Arch without this patch. It's not a systemd issue, it's a larger cultural issue with packagers increasing attack surface to make their lives easier.
        b112 3y ago
        It absolutely, positively, 100% was added due to issues with systemd. Both Redhat and Debian and others applied this patch as a result. People didn't do it "just cause". https://bugzilla.redhat.com/show_bug.cgi?id=1381997 https://bugzilla.redhat.com/show_bug.cgi?id=1381997 Jakub Jelen 2016-10-12 08:40:44 UTC <snip> > Why do you want to avoid sd_notify()? Why to avoid this? Well, it is adding more systemd-specific bits and new build dependency to something that always worked well under other inits without any problems for years.
        streb-lo 3y ago
    - astrange 3y ago
      It is possible to prevent libraries from patching functions in other libraries; make those VM regions unwritable, don't let anyone make them writable, and adopt PAC or similar hardware protection so the kernel can't overwrite them either.
      - rwmj 3y ago
        That's already done, but in this case the attack happened in a glibc ifunc and those run before the patching protection is enabled (since an ifunc has to patch the PLT).
        astrange 3y ago
        Sounds like libraries should only get to patch themselves. (Some difficulty with this one though. For instance you probably have to ban running arbitrary code at load time, but you should do this anyway because it will stop people from writing C++.)
        rwmj 3y ago
        They're all in a single address space, there's nothing you can do to stop one part of a binary patching any other part.
        astrange 3y ago
        I already said the way to do it in my previous comment. Don't let arbitrary code run before page permissions are locked down.
        rwmj 3y ago
        If you're running in the binary you can call mprotect(2), and even if that is blocked you can cause all kinds of mischief. The original motivation for rings of protection on i286 was so that libraries could run in a different ring from the binary (usually library in ring 2 and program in ring 3), using a call gate (a very controlled type of call) to dispatch calls from the binary to the library, which stops the binary from modifying the library and IIRC libraries from touching each other. But x86-64 got rid of the middle rings.
  - jethro_tell 3y ago
    Openssh has refused this patch that enables this in Debian multiple times as acrisk of preauth code execution as root. So maybe just don't patch sshd?
- mjg59 3y ago
  It wouldn't matter in this case, since the exploit could simply rewrite the function that calls out to the unprivileged process. If you already have malicious code in your privileged parent process there's no way to recover from that.
  - CanaryLayout 3y ago
    Exactly. The attack came in by hitching a ride on to systemd. sshd is not the problem. the ldd/monolith architecture surrounding systemd is. What if I duplicated this attack but instead targeted dbus or any other thing that systemd is managing?
    - saagarjha 3y ago
      No, the problem is that someone had access to backdoor code that runs in a privileged process.
      - eto-san 3y ago
        Tell us all, please, how the starting vector of this attack would affect statically compiled dropbear binary even with systemd's libsystemd pwnage? I am very cruious about your reasoning. The fact, that the whole reason this library is even being pulled into the sshd daemon process, is some stupid stuff like readiness notification, which itself is utterly broken on systemd, by design (and thus is forever unfixable), and makes this even more tragic. Don't put your head into the sand, just because of the controversial nature of the topic. Systemd was VERY accommodating in this whole fiasco. Saddest part of all this is, that we know how to to do better. At least since Bernstein, OpenBSD and supervision community (runit/s6) guys solved it. Yet somehow we see same mistakes repeated again and again. If you want to read about notification shenanigans and systemd's dubious (at best) decisions and implementation quality, read here: https://jdebp.uk/FGA/unix-daemon-readiness-protocol-problems.html https://jdebp.uk/FGA/unix-daemon-readiness-protocol-problems... I believe this is how you do readiness notification properly: http://skarnet.org/software/s6/notifywhenup.html http://skarnet.org/software/s6/notifywhenup.html I.e. you fork and run little helper to write, or directly write a single byte(!), to notify supervisor over supervisor provided fd. It allows you to even privseparate your notifier stuff or do all the cute SELinux magic you need. But that would be too simple, I guess, so instead we link like 10 completely unrelated libraries into sshd, liblzma being one of them, one of the most crucial processes on the machine. To notify supervisor that it's ready. Sounds about right, linux distros (and very specific ones at that). Sshd should be sacred, nothing more than libc and some base cryptolibs (I don't remember whether it still needs <any>ssl even) it needs. Another great spot to break sshd is PAM, which has no place doing there either. Unfortunately it's hard dep. on most linux distros. Maybe sshd should adopt kernel taint approach: as soon as any weird libraries (ie everything not libc and cryptolibs) are detected in sshd proces it should consider itself tainted. Maybe even seppuku itself. The exploit could be, probably, somehow doable without systemd. But it would be much, much harder though. Don't try to obfuscate that very fact from the discussion.

by treasy 3y ago

cryptonector 3y ago

The signature validation could be moved into an unprivileged process forked from that one.

littlestymaar 3y ago
That's an easy thing to say after the fact indeed but yes. In fact after such a disastrous backdoor I wouldn't be surprised if OpenSSH moved all code calling external libraries to unprivileged processes to make sure such an attack can never have such a dramatic effect (an auth bypass would still likely be possible, but that's still way better than a root RCE…). At this point “All libraries could be malicious” is a threat model that must be considered for something as security critical as OpenSSH.
- asveikau 3y ago
  I don't think that's a threat model that OpenSSH should waste too much time on. Ultimately this is malicious code in the build machine compiling a critical system library. That's not reasonable to defend against. Keep in mind that upstream didn't even link to liblzma. Debian patched it to do so. OpenSSH should defend against that too?
  - CanaryLayout 3y ago
    any one of us if we sat on the OSSH team would flip the middle finger. What code is the project supposed to write when nothing on main dyn loaded liblzma. It was brought in from a patch they don't have realistic control over. This is a Linux problem, and the problem is systemd, which is who brought the lib into memory and init'd it.
    - shzhdbi09gv8ioi 3y ago
      > This is a Linux problem, and the problem is systemd, which is who brought the lib into memory and init'd it. Not at all, it is a distro issue because a few distros such as Debian chose to patch openssh to bring in systemd support [1]. Other systemd-based distros like Arch Linux remains unaffected because they don't carry this patch. 1: https://sources.debian.org/src/openssh/1%3A9.7p1-2/debian/patches/systemd-readiness.patch/ https://sources.debian.org/src/openssh/1%3A9.7p1-2/debian/pa...
      - b112 3y ago
        Yet Redhat and others applied this patch, as systemd is so incapable of reliably launching processes, that it kept killing sshd without it. What a complete failure of an init system's job, and the patch was applied due to systemd not resolving the issue in another way. This is the problem with systemd. Way, way way too much complexity.
        streb-lo 3y ago
        systemd is just fine on Arch without this patch. It's not a systemd issue, it's a larger cultural issue with packagers increasing attack surface to make their lives easier.
        b112 3y ago
        It absolutely, positively, 100% was added due to issues with systemd. Both Redhat and Debian and others applied this patch as a result. People didn't do it "just cause". https://bugzilla.redhat.com/show_bug.cgi?id=1381997 https://bugzilla.redhat.com/show_bug.cgi?id=1381997 Jakub Jelen 2016-10-12 08:40:44 UTC <snip> > Why do you want to avoid sd_notify()? Why to avoid this? Well, it is adding more systemd-specific bits and new build dependency to something that always worked well under other inits without any problems for years.
        streb-lo 3y ago
  - astrange 3y ago
    It is possible to prevent libraries from patching functions in other libraries; make those VM regions unwritable, don't let anyone make them writable, and adopt PAC or similar hardware protection so the kernel can't overwrite them either.
    - rwmj 3y ago
      That's already done, but in this case the attack happened in a glibc ifunc and those run before the patching protection is enabled (since an ifunc has to patch the PLT).
      - astrange 3y ago
        Sounds like libraries should only get to patch themselves. (Some difficulty with this one though. For instance you probably have to ban running arbitrary code at load time, but you should do this anyway because it will stop people from writing C++.)
        rwmj 3y ago
        They're all in a single address space, there's nothing you can do to stop one part of a binary patching any other part.
        astrange 3y ago
        I already said the way to do it in my previous comment. Don't let arbitrary code run before page permissions are locked down.
        rwmj 3y ago
        If you're running in the binary you can call mprotect(2), and even if that is blocked you can cause all kinds of mischief. The original motivation for rings of protection on i286 was so that libraries could run in a different ring from the binary (usually library in ring 2 and program in ring 3), using a call gate (a very controlled type of call) to dispatch calls from the binary to the library, which stops the binary from modifying the library and IIRC libraries from touching each other. But x86-64 got rid of the middle rings.
- jethro_tell 3y ago
  Openssh has refused this patch that enables this in Debian multiple times as acrisk of preauth code execution as root. So maybe just don't patch sshd?
mjg59 3y ago
It wouldn't matter in this case, since the exploit could simply rewrite the function that calls out to the unprivileged process. If you already have malicious code in your privileged parent process there's no way to recover from that.
- CanaryLayout 3y ago
  Exactly. The attack came in by hitching a ride on to systemd. sshd is not the problem. the ldd/monolith architecture surrounding systemd is. What if I duplicated this attack but instead targeted dbus or any other thing that systemd is managing?
  - saagarjha 3y ago
    No, the problem is that someone had access to backdoor code that runs in a privileged process.
    - eto-san 3y ago
      Tell us all, please, how the starting vector of this attack would affect statically compiled dropbear binary even with systemd's libsystemd pwnage? I am very cruious about your reasoning. The fact, that the whole reason this library is even being pulled into the sshd daemon process, is some stupid stuff like readiness notification, which itself is utterly broken on systemd, by design (and thus is forever unfixable), and makes this even more tragic. Don't put your head into the sand, just because of the controversial nature of the topic. Systemd was VERY accommodating in this whole fiasco. Saddest part of all this is, that we know how to to do better. At least since Bernstein, OpenBSD and supervision community (runit/s6) guys solved it. Yet somehow we see same mistakes repeated again and again. If you want to read about notification shenanigans and systemd's dubious (at best) decisions and implementation quality, read here: https://jdebp.uk/FGA/unix-daemon-readiness-protocol-problems.html https://jdebp.uk/FGA/unix-daemon-readiness-protocol-problems... I believe this is how you do readiness notification properly: http://skarnet.org/software/s6/notifywhenup.html http://skarnet.org/software/s6/notifywhenup.html I.e. you fork and run little helper to write, or directly write a single byte(!), to notify supervisor over supervisor provided fd. It allows you to even privseparate your notifier stuff or do all the cute SELinux magic you need. But that would be too simple, I guess, so instead we link like 10 completely unrelated libraries into sshd, liblzma being one of them, one of the most crucial processes on the machine. To notify supervisor that it's ready. Sounds about right, linux distros (and very specific ones at that). Sshd should be sacred, nothing more than libc and some base cryptolibs (I don't remember whether it still needs <any>ssl even) it needs. Another great spot to break sshd is PAM, which has no place doing there either. Unfortunately it's hard dep. on most linux distros. Maybe sshd should adopt kernel taint approach: as soon as any weird libraries (ie everything not libc and cryptolibs) are detected in sshd proces it should consider itself tainted. Maybe even seppuku itself. The exploit could be, probably, somehow doable without systemd. But it would be much, much harder though. Don't try to obfuscate that very fact from the discussion.