5 ms·

I love this case, because every time someone cites it as a reason packagers shouldn't patch upstream software I get to point out that they had to go back over a

by ris 2y ago

I love this case, because every time someone cites it as a reason packagers shouldn't patch upstream software I get to point out that they had to go back over a decade to find an example of it going bad. Soon it's going to be two decades.

tredre3 2y ago
The xz backdoor was caused by packagers patching OpenSSH. Just because it was caught you don't get to pretend it doesn't count.
- asveikau 2y ago
  That was one factor of multiple, and the malicious actor chose to exploit it that way. liblzma is in enough critical stuff that they could have chosen to exploit something different had it not been for that.
- mistrial9 2y ago
  this is such a spin! that bug was two years worth of James Bond-level insertions into a situation that "was caused" by systemd ! if you want to get creative in the rewriting of fact
  - hxelk1 2y ago
    This isn't about systemd. OpenSSH is one of the most (if not the most) security-critical program in the distribution. Many systems run with just ssh enabled. That's why you don't mess with it. Which library pulled the vulnerability in is mostly irrelevant.
    - b112 2y ago
      When the init system won't reliably start openssh, and insists the only fix is to patch, then blame the horrible init system. And that was what happened with systemd.
      - amaranth 2y ago
        sd_notify is for additional (useful) functionality, it would work fine without it. You can tell because it works on arch.
        b112 2y ago
        You'd think it woukd work fine, after all, init systems for half a century have worked fine without it. But no. Newer versions of systemd have issues, and this was what systemd pushed. Just why do you think all these distros had the sane patch? For fun? Arch would have ended up with it eventually. It wasn't Arch being prescient, Arch wasn't using the same systend version as Debian Unstable, and other distros bleeding edge branches.
- dgrunwald 2y ago
  You're confused there. The xz backdoor made use of a Debian OpenSSH patch, but it wasn't "caused" by it. Without the patch, the malicious xz maintainer could have written a different backdoor without making use of the OpenSSH patch -- for example, since debian packages are compressed with xz, the backdoor could have modified the sshd binary while unpacking the next OpenSSH security update. That would have been slower (attacker might have needed to wait a long time for a security update), and more discoverable since the modified file would be persisted to disk; but it also wouldn't have caused the performance issues that ended up in the discovery of the backdoor.
  - kemotep 2y ago
    It would be discoverable but only if you ran an additional hash to check the final binary after updating and checking with an out of band source what the hash of the binary should be. How many people double check that apt actually updated the package to the right version, if it’s output is compromised?
- arp242 2y ago
  Do you really think there was no other avenue? There are tons and tons of things that link against liblzma, including stuff that commonly gets run as root such as apt, udev, and grub.
hxelk1 2y ago
I believe the xz backdoor relied on Debian patching OpenSSH with libsystemd to work.
- cassianoleal 2y ago
  _sigh_ the backdoor was found because Debian also made those patches. Nearly all major distros were affected. The reason why Debian made the news is because the researcher who found the issue was using Debian. Had he been using Ubuntu, Arch, Fedora... those would have been in the news instead.
  - Latty 2y ago
    According to Arch: "openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma", so at least one of your examples is wrong. That specific vulnerability was not in Arch. The xz package was potentially vulnerable (although not in reality because "the build script was configured to only inject the bad code in Debian/Fedora based package build environments", while this was a choice by the attacker, it's still true the vulnerability wasn't there), but patching OpenSSH made OpenSSH specifically vulnerable when used with a malicious xz install. https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore...
    - tetha 2y ago
      >"openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma", so at least one of your examples is wrong. That specific vulnerability was not in Arch. This is such a weird formulation though, because "other distributions" apparently included insignificant parts of the linux landscape like Fedora (i.e. the testing variant of the RedHat world) and SUSE. And if the three largest upstream distris in the linux world have this mistake, calling that "Well some distris, but screw mostly Debian" doesn't sound like a strong point.
CJefferson 2y ago
Two open source programs I work on have had bad patches added by packagers. This one was special because it lead to a massive security issue, rather than just annoyed users who had bugs only because of packaging patches.

by ris 2y ago

tredre3 2y ago

The xz backdoor was caused by packagers patching OpenSSH. Just because it was caught you don't get to pretend it doesn't count.

asveikau 2y ago
That was one factor of multiple, and the malicious actor chose to exploit it that way. liblzma is in enough critical stuff that they could have chosen to exploit something different had it not been for that.
mistrial9 2y ago
this is such a spin! that bug was two years worth of James Bond-level insertions into a situation that "was caused" by systemd ! if you want to get creative in the rewriting of fact
- hxelk1 2y ago
  This isn't about systemd. OpenSSH is one of the most (if not the most) security-critical program in the distribution. Many systems run with just ssh enabled. That's why you don't mess with it. Which library pulled the vulnerability in is mostly irrelevant.
  - b112 2y ago
    When the init system won't reliably start openssh, and insists the only fix is to patch, then blame the horrible init system. And that was what happened with systemd.
    - amaranth 2y ago
      sd_notify is for additional (useful) functionality, it would work fine without it. You can tell because it works on arch.
      - b112 2y ago
        You'd think it woukd work fine, after all, init systems for half a century have worked fine without it. But no. Newer versions of systemd have issues, and this was what systemd pushed. Just why do you think all these distros had the sane patch? For fun? Arch would have ended up with it eventually. It wasn't Arch being prescient, Arch wasn't using the same systend version as Debian Unstable, and other distros bleeding edge branches.
dgrunwald 2y ago
You're confused there. The xz backdoor made use of a Debian OpenSSH patch, but it wasn't "caused" by it. Without the patch, the malicious xz maintainer could have written a different backdoor without making use of the OpenSSH patch -- for example, since debian packages are compressed with xz, the backdoor could have modified the sshd binary while unpacking the next OpenSSH security update. That would have been slower (attacker might have needed to wait a long time for a security update), and more discoverable since the modified file would be persisted to disk; but it also wouldn't have caused the performance issues that ended up in the discovery of the backdoor.
- kemotep 2y ago
  It would be discoverable but only if you ran an additional hash to check the final binary after updating and checking with an out of band source what the hash of the binary should be. How many people double check that apt actually updated the package to the right version, if it’s output is compromised?
arp242 2y ago
Do you really think there was no other avenue? There are tons and tons of things that link against liblzma, including stuff that commonly gets run as root such as apt, udev, and grub.

hxelk1 2y ago

I believe the xz backdoor relied on Debian patching OpenSSH with libsystemd to work.

cassianoleal 2y ago
_sigh_ the backdoor was found because Debian also made those patches. Nearly all major distros were affected. The reason why Debian made the news is because the researcher who found the issue was using Debian. Had he been using Ubuntu, Arch, Fedora... those would have been in the news instead.
- Latty 2y ago
  According to Arch: "openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma", so at least one of your examples is wrong. That specific vulnerability was not in Arch. The xz package was potentially vulnerable (although not in reality because "the build script was configured to only inject the bad code in Debian/Fedora based package build environments", while this was a choice by the attacker, it's still true the vulnerability wasn't there), but patching OpenSSH made OpenSSH specifically vulnerable when used with a malicious xz install. https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore...
  - tetha 2y ago
    >"openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma. Arch does not directly link openssh to liblzma", so at least one of your examples is wrong. That specific vulnerability was not in Arch. This is such a weird formulation though, because "other distributions" apparently included insignificant parts of the linux landscape like Fedora (i.e. the testing variant of the RedHat world) and SUSE. And if the three largest upstream distris in the linux world have this mistake, calling that "Well some distris, but screw mostly Debian" doesn't sound like a strong point.

CJefferson 2y ago

Two open source programs I work on have had bad patches added by packagers. This one was special because it lead to a massive security issue, rather than just annoyed users who had bugs only because of packaging patches.