8 ms·

> However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as roo

> However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root. With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. However, applying a sandbox to an SSH daemon effectively is very difficult.

junon 3y ago
Right, though if I'm understanding correctly, this is targeting openssl, not just sshd. So there's a larger set of circumstances where this could have been exploited. I'm not sure if it's yet been confirmed that this is confined only to sshd.
- jeroenhd 3y ago
  The exploit, as currently found, seems to target OpenSSH specifically. It's possible that everything involving xz has been compromised, but I haven't read any reports that there is a path to malware execution outside of OpenSSH. A quote from the first analysis that I know of (https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4): > Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked. This appears to be part of some countermeasures to make analysis harder. > a) TERM environment variable is not set > b) argv[0] needs to be /usr/sbin/sshd > c) LD_DEBUG, LD_PROFILE are not set > d) LANG needs to be set > e) Some debugging environments, like rr, appear to be detected. Plain gdb appears to be detected in some situations, but not others
onedognight 3y ago
> With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. Please review this commit[0] where the sandbox detection was “improved”. [0] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...
- ronsor 3y ago
  Well, the definition of "improve" depends on one's goals.
- Denvercoder9 3y ago
  That one's a separate attack vector, which is seemingly unused in the sshd attack. It only disables sandboxing of the xzdec(2) utility, which is not used in the sshd attack.
  - formerly_proven 3y ago
    Which strongly suggests that they planned and/or executed more backdoors via Jia Tan’s access.
    - pja 3y ago
      I guess xzdec was supposed to sandbox itself where possible so they disabled the sandbox feature check in the build system so that future payload exploits passed to xzdec wouldn’t have to escape the sandbox in order to do anything useful? Sneaky.
- glandium 3y ago
  Oh, that one is interesting, because it only breaks it in cmake.
  - sn 3y ago
    I wonder if there is anything else cmake related that should be looked at. Wasn't cmake support originally added to xz to use with Windows and MSVC?
    - glandium 3y ago
      But that's a check for a Linux feature. So the more interesting question would be, what in the Linux world might be building xz-utils with cmake, I guess using ExternalProject_Add or something similar.
hellcow 3y ago
Another reason to adopt OpenBSD style pledge/unveil in Linux.
- somat 3y ago
  Would that help? sshd, by design, opens shells. the backdoor payload was basically to open a shell. that is, the very thing that sshd has to do. The pledge/unvail system is pretty great, but my understanding is that it do not do anything that the linux equivalent interfaces(seccomp i think) cannot do. It is just a simplified/saner interface to the same problem of "how can a program notify the kernel what it's scope is?" The main advantage to pledge/unveil bring to the table is that they are easy to use and cannot be turned off, optional security isn't.
- jeroenhd 3y ago
  By design, OpenSSH will start an interactive shell with either the capabilities to escalate to root or direct root permissions. I don't think pledge/unveil will work any better than seccomp already does. I do like the pledge/unveil API, but I don't think it would've made much of a difference.
semiquaver 3y ago
Could you explain how SELinux could ever sandbox against RCE in sshd? Its purpose is to grant login shells to arbitrary users, after all.
- ajross 3y ago
  It's possible to spawn a sshd as an unprivileged or partially-capabilitized process. Such as sandbox isn't the default deployment, but it's done often enough and would work as designed to prevent privilege elevation above the sshd process.
  - admax88qqq 3y ago
    How can sshd spawn interactive sessions for other users if it's sandboxed?
    - ajross 3y ago
      Plausibly by having set-user-ID capability but not others an attacker might need. But in the more common case it just doesn't: you have an sshd running on a dedicated port for the sole purpose of running some service or another under a specific sandboxed UID. That's basically the github business model, for example.
      - dotancohen 3y ago
        I need full filesystem access, VIM, ls, cd, grep, awk, df, du at the very least. Sometimes perl, find, ncdu, and other utilities are necessary as well. Are you suggesting that each tool have its own SSH process wrapping it? Maybe write a shell to coordinate between them? It should support piping and output redirection, please.
        ajross 3y ago
        Sigh. I'm not saying there's a sandboxed sshd setup that has equivalent functionality to the default one in your distro. I'm not even saying that there's one appropriate for your app. I'm saying, as a response to the point above, that sandboxing sshd is absolutely a valid defense-in-depth technique for privilege isolation, that it would work against attacks like this one to prevent whole-system exploitation, and that it's very commonly deployed in practice (c.f. running a git/ssh server a-la github).
deleted 3y ago
[deleted]
quotemstr 3y ago
Doesn't matter. This is a supply chain attack, not a vulnerability arising from a bug. All sandboxing the certificate parsing code would have done is make the author of the backdoor do a little bit more work to hijack the necessarily un-sandboxed supervisor process. Applying the usual exploit mitigations to supply chain attacks won't do much good. What will? Kill distribution tarballs. Make every binary bit for bit reproducible from a known git hash. Minimize dependencies. Run whole programs with minimal privileges. Oh, and finally support SHA2 in git to forever forestall some kind of preimage attack against a git commit hash.
- nurple 3y ago
  Oh boy, do I have the packaging system for you!
- LtWorf 3y ago
  Malware can be in commits as well…
  - quotemstr 3y ago
    But it's harder to hide there
    - LtWorf 3y ago
      Just do a "fix style" that deletes and re-addds every line-
- int_19h 3y ago
  ... and stop adding random patches to upstream software, especially when we're talking about security-critical stuff that must absolutely not be released without a very thorough security review.
cryptonector 3y ago
This is what PrivSep was supposed to do. sshd could fork an unprivileged and restricted process to do the signature validation, I suppose.
dhx 3y ago
sshd is probably the softest target on most systems. It is generally expected (and setup by default) so that people can gain a root shell that provides unrestricted access. sshd.service will typically score 9.6/10 for "systemd-analyze security sshd.service" where 10 is the worst score. When systemd starts a process, it does so by using systemd-nspawn to setup a (usually) restricted namespace and apply seccomp filters before the process is then executed. seccomp filters are inherited by child processes, which can then only further restrict privileges but not expand upon the inherited privileges. openssh-portable on Linux does apply seccomp filters to child processes but this is useless in this attack scenario because sshd is backdoored by the xz library, and the backdoored library can just disable/change those seccomp filters before sshd is executed. sshd is particularly challenging to sandbox because if you were to restrict the namespace and apply strict seccomp filters via systemd-nspawn, a user gaining a root shell via sshd (or wanting to sudo/su as root) is then perhaps prevented from remotely debugging applications, accessing certain filesystems, interacting with network interfaces, etc depending on what level of sandboxing is applied from systemd-nspawn. This choice is highly user dependent and there are probably only limited sane defaults for someone who has already decided they want to use sshd. For example, sane defaults could include creating dedicated services with sandboxing tailored just for read-only sftp user filesystem access, a separate service for read/write sftp user filesystem access, sshd tunneling, unprivileged remote shell access, etc.
- nottorp 3y ago
  So for all practical purposes you can't sandbox ssh on a developer's machine much.

junon 3y ago

Right, though if I'm understanding correctly, this is targeting openssl, not just sshd. So there's a larger set of circumstances where this could have been exploited. I'm not sure if it's yet been confirmed that this is confined only to sshd.

jeroenhd 3y ago
The exploit, as currently found, seems to target OpenSSH specifically. It's possible that everything involving xz has been compromised, but I haven't read any reports that there is a path to malware execution outside of OpenSSH. A quote from the first analysis that I know of (https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4): > Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked. This appears to be part of some countermeasures to make analysis harder. > a) TERM environment variable is not set > b) argv[0] needs to be /usr/sbin/sshd > c) LD_DEBUG, LD_PROFILE are not set > d) LANG needs to be set > e) Some debugging environments, like rr, appear to be detected. Plain gdb appears to be detected in some situations, but not others

onedognight 3y ago

> With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. Please review this commit[0] where the sandbox detection was “improved”. [0] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2...

ronsor 3y ago
Well, the definition of "improve" depends on one's goals.
Denvercoder9 3y ago
That one's a separate attack vector, which is seemingly unused in the sshd attack. It only disables sandboxing of the xzdec(2) utility, which is not used in the sshd attack.
- formerly_proven 3y ago
  Which strongly suggests that they planned and/or executed more backdoors via Jia Tan’s access.
  - pja 3y ago
    I guess xzdec was supposed to sandbox itself where possible so they disabled the sandbox feature check in the build system so that future payload exploits passed to xzdec wouldn’t have to escape the sandbox in order to do anything useful? Sneaky.
glandium 3y ago
Oh, that one is interesting, because it only breaks it in cmake.
- sn 3y ago
  I wonder if there is anything else cmake related that should be looked at. Wasn't cmake support originally added to xz to use with Windows and MSVC?
  - glandium 3y ago
    But that's a check for a Linux feature. So the more interesting question would be, what in the Linux world might be building xz-utils with cmake, I guess using ExternalProject_Add or something similar.

hellcow 3y ago

Another reason to adopt OpenBSD style pledge/unveil in Linux.

somat 3y ago
Would that help? sshd, by design, opens shells. the backdoor payload was basically to open a shell. that is, the very thing that sshd has to do. The pledge/unvail system is pretty great, but my understanding is that it do not do anything that the linux equivalent interfaces(seccomp i think) cannot do. It is just a simplified/saner interface to the same problem of "how can a program notify the kernel what it's scope is?" The main advantage to pledge/unveil bring to the table is that they are easy to use and cannot be turned off, optional security isn't.
jeroenhd 3y ago
By design, OpenSSH will start an interactive shell with either the capabilities to escalate to root or direct root permissions. I don't think pledge/unveil will work any better than seccomp already does. I do like the pledge/unveil API, but I don't think it would've made much of a difference.

semiquaver 3y ago

Could you explain how SELinux could ever sandbox against RCE in sshd? Its purpose is to grant login shells to arbitrary users, after all.

ajross 3y ago
It's possible to spawn a sshd as an unprivileged or partially-capabilitized process. Such as sandbox isn't the default deployment, but it's done often enough and would work as designed to prevent privilege elevation above the sshd process.
- admax88qqq 3y ago
  How can sshd spawn interactive sessions for other users if it's sandboxed?
  - ajross 3y ago
    Plausibly by having set-user-ID capability but not others an attacker might need. But in the more common case it just doesn't: you have an sshd running on a dedicated port for the sole purpose of running some service or another under a specific sandboxed UID. That's basically the github business model, for example.
    - dotancohen 3y ago
      I need full filesystem access, VIM, ls, cd, grep, awk, df, du at the very least. Sometimes perl, find, ncdu, and other utilities are necessary as well. Are you suggesting that each tool have its own SSH process wrapping it? Maybe write a shell to coordinate between them? It should support piping and output redirection, please.
      - ajross 3y ago
        Sigh. I'm not saying there's a sandboxed sshd setup that has equivalent functionality to the default one in your distro. I'm not even saying that there's one appropriate for your app. I'm saying, as a response to the point above, that sandboxing sshd is absolutely a valid defense-in-depth technique for privilege isolation, that it would work against attacks like this one to prevent whole-system exploitation, and that it's very commonly deployed in practice (c.f. running a git/ssh server a-la github).

deleted 3y ago

[deleted]

quotemstr 3y ago

Doesn't matter. This is a supply chain attack, not a vulnerability arising from a bug. All sandboxing the certificate parsing code would have done is make the author of the backdoor do a little bit more work to hijack the necessarily un-sandboxed supervisor process. Applying the usual exploit mitigations to supply chain attacks won't do much good. What will? Kill distribution tarballs. Make every binary bit for bit reproducible from a known git hash. Minimize dependencies. Run whole programs with minimal privileges. Oh, and finally support SHA2 in git to forever forestall some kind of preimage attack against a git commit hash.

nurple 3y ago
Oh boy, do I have the packaging system for you!
LtWorf 3y ago
Malware can be in commits as well…
- quotemstr 3y ago
  But it's harder to hide there
  - LtWorf 3y ago
    Just do a "fix style" that deletes and re-addds every line-
int_19h 3y ago
... and stop adding random patches to upstream software, especially when we're talking about security-critical stuff that must absolutely not be released without a very thorough security review.

cryptonector 3y ago

This is what PrivSep was supposed to do. sshd could fork an unprivileged and restricted process to do the signature validation, I suppose.

dhx 3y ago

sshd is probably the softest target on most systems. It is generally expected (and setup by default) so that people can gain a root shell that provides unrestricted access. sshd.service will typically score 9.6/10 for "systemd-analyze security sshd.service" where 10 is the worst score. When systemd starts a process, it does so by using systemd-nspawn to setup a (usually) restricted namespace and apply seccomp filters before the process is then executed. seccomp filters are inherited by child processes, which can then only further restrict privileges but not expand upon the inherited privileges. openssh-portable on Linux does apply seccomp filters to child processes but this is useless in this attack scenario because sshd is backdoored by the xz library, and the backdoored library can just disable/change those seccomp filters before sshd is executed. sshd is particularly challenging to sandbox because if you were to restrict the namespace and apply strict seccomp filters via systemd-nspawn, a user gaining a root shell via sshd (or wanting to sudo/su as root) is then perhaps prevented from remotely debugging applications, accessing certain filesystems, interacting with network interfaces, etc depending on what level of sandboxing is applied from systemd-nspawn. This choice is highly user dependent and there are probably only limited sane defaults for someone who has already decided they want to use sshd. For example, sane defaults could include creating dedicated services with sandboxing tailored just for read-only sftp user filesystem access, a separate service for read/write sftp user filesystem access, sshd tunneling, unprivileged remote shell access, etc.

nottorp 3y ago
So for all practical purposes you can't sandbox ssh on a developer's machine much.