5 ms·

FAQ on the xz-utils backdoor

141 points by jethro_tell 3y ago

TillE 3y ago
> The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. This doesn't seem really vital to the attack, but it's a silly outdated convention. Nobody is manually downloading and compiling tarballs these days, and those who do can figure out how to use autotools.
- IshKebab 3y ago
  Yeah terrible idea. Either you should commit the output of autoconf (and enforce keeping it up to date via CI), or much much better - stop using autoconf! It belongs in the 90s!
  - knorker 3y ago
    It's another case of that "yes, it sucks. But the alternatives just plain don't work, or work in WAY fewer circumstances". If I have a strange architecture or install, autoconf is often the only one that works.
    - IshKebab 3y ago
      What strange architecture are you talking about? I don't think the cost of supporting dead architectures is worth it. CMake is a clear better choice (it also sucks but it sucks waaaay less).
      - knorker 3y ago
        I didn't want to start shitting on alternatives, but cmake is in my experience clearly the worst of all options. It seems that more often than not, cmake setups work on the developer's own machine and nothing else. And the choice of undocumented "-DSOMETHING" is about as user hostile as it gets.
- frankjr 3y ago
  Reminds of the note at the bottom of Fish releases. It's there because the build system cannot determine the current version for some reason. Hopefully that will go away now that they have switched to a different language / build system. The custom tarball is used by Arch Linux at the very least. https://github.com/fish-shell/fish-shell/releases/tag/3.7.1 https://github.com/fish-shell/fish-shell/releases/tag/3.7.1 https://github.com/fish-shell/fish-shell/issues/7772#issuecomment-793982192 https://github.com/fish-shell/fish-shell/issues/7772#issueco... https://gitlab.archlinux.org/archlinux/packaging/packages/fish/-/blob/661c4ae6a91a7ead1a6bdb45b186fe6d0e698fac/PKGBUILD#L22 https://gitlab.archlinux.org/archlinux/packaging/packages/fi...
- k8svet 3y ago
  Yes, and in fact, it's quite a huge pain even in the packaging side. Everytime I have to deal with autotools in nixpkgs, I end up having to rewrite the derivation to build from actual freaking source instead of a pre-mangled non-release tarball. In fact, this makes me want to RFC *banning* release tarball usage.
  - deleted 3y ago
    [deleted]
- binary132 3y ago
  you mean like LLVM’s source release tars? yeah, who would do that, haha?
deleted 3y ago
[deleted]
pdw 3y ago
Flagged: How is this "via a project dev"? Which project? The guy doesn't seem to have anything to do with the discovery of this backdoor, and he just summarizes Andres Freund's announcement.
- bhaney 3y ago
  The author of this is a gentoo maintainer who focuses on security work. Most distros have a couple of point people handling this issue for them at this point, and they are all likely pretty well informed. If you want clarification on something, you can just ask instead of announcing that you've flagged it.
- jethro_tell 3y ago
  Its from the Gentoo maintainer of the package and libarchive project dev that submits changes, patches, is involved in the mail list and has ops on the groups IRC chan. It's included in Lasse Collin's FAQ now that he has been found, but in his absence, Sam was speaking for the project on IRC chan and filed the gentoo bug also linked at the bottom of in Collin's FAQ: https://tukaani.org/xz-backdoor/ https://tukaani.org/xz-backdoor/ You could have just asked and I could have elaborated.
choeger 3y ago
So the attacker put the payload into the tests folder of the compression library and injected it upon build via some arcane build script? Kudos. That's quite a smart way to hide your backdoor. Nevertheless, they got caught. So big respect for the analysis. Amazing work. That being said, what does this attack tell you about the future of software security? Free software could easily be compromised by having a highly paid team create such a payload and then convince some maintainer, one way or another, to deploy it. How do we back off from that cliff?
- cassianoleal 3y ago
  Security is hard. In this case, it seems like there's a lot that could be done to reduce the chance of being exploited. Why are obscure binaries in the repository at all in the first place, even if in the test dir? Surely those binaries can be generated as a setup step prior to the tests actually running. Furthermore, why are there scripts present in the release package that have not been added to the source code? Also, if I understood it correctly, the compromised binaries are also present in the release tarball. Why are they part of the release if they are supposedly test fixtures rather than part of the application code?
  - cesarb 3y ago
    > Why are they part of the release if they are supposedly test fixtures rather than part of the application code? Test fixtures should be distributed together with the application code, so that the actually built application can be tested. Ideally, the application should be tested every time it's built, to ensure that for instance a change in the compiler being used didn't break it.
    - cassianoleal 3y ago
      Thanks, that makes sense. I think in my mind the whole chain was still source code->compile->test->distribute from a single repo which is obviously not the case. It’s more like source->package->test->ship to distros->compile->test->package->distribute
- 3y ago
pseudo0 3y ago
Main discussion thread here: https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810 Backdoor in upstream xz/liblzma leading to SSH server compromise (openwall.com) 1848 points by rkta 5 hours ago | 654 comments
AshamedCaptain 3y ago
> The payload only activates if the running program has the process name /usr/bin/sshd. This means that systems that put sshd in /usr/sbin or another folder are not vulnerable. This further suspects targeting systemd systems due to their usrmerge initiative putting all binaries in /usr/bin. This is highly likely wrong. The hack targets rpm and deb distros and as far as I know _all of them_ put sshd in /usr/sbin. The original oss-security report specifically said arg0 had to be /usr/sbin/sshd and I can reproduce his findings about increased runtime if you do so. This /usr/bin vs /usr/sbin also has nothing do with usrmerge. This type of mistake casts doubt in the rest of analysis.
- redserk 3y ago
  There is no rational reason to justify throwing out the entire analysis over one mistake.
  - AshamedCaptain 3y ago
    I said "Casts doubt". In the same way a typo in the headline, a logic error in the first bullet point, a misconception in the introduction, etc. casts doubt on the rest of a publication. And this was no simple typo, as there was an accompanying and erroneous justification (which btw also hints to a lack of knowledge in the topic). This bullet was practically the easiest to verify out of the entire article and was already showing a contradiction. You might enjoy publications where fact-checking is not high in the list of priorities of the author (this article contains a disclaimer saying as much), but most "rational" readers should be interested in knowing how light the fact-checking indeed was, specially in a topic as important as this.
aimonster2 3y ago
So wait, if I did a find / -n 'liblzma*' and saw 5.4, I'm good? This is only 5.6 and 5.6.1, right?
- xorcist 3y ago
  From what is publicly known at the moment, yes. Note however that xzutils home page says that "versions 5.2.12, 5.4.3 and later have been signed with Jia Tan's OpenPGP key" so there would have been plenty more opportunities. We may just have seen the beginning. Whoever did this played the long game. Also note that there was proposed patches by this compromised project maintainer to oss-fuzz and valgrind to avoid the detection of this backdoor.
- botanical 3y ago
  The attacker had 750 previous commits; maybe it was all for this or maybe there are more vulnerabilities.
xorcist 3y ago
People are prematurely celebrating this did not reach a stable distribution. But we haven't seen the full fallout yet. How many debian-developer's machines have been compromised, for example?
- threeseed 3y ago
  I don't think that's the biggest concern. There are 2 years of commits that potentially have backdoors.
timattrn 3y ago
so this backdoor is a package which installs a fake sshd? what is selinux for?

141 points by jethro_tell 3y ago

> The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. This doesn't seem really vital to the attack, but it's a silly outdated convention. Nobody is manually downloading and compiling tarballs these days, and those who do can figure out how to use autotools.

IshKebab 3y ago
Yeah terrible idea. Either you should commit the output of autoconf (and enforce keeping it up to date via CI), or much much better - stop using autoconf! It belongs in the 90s!
- knorker 3y ago
  It's another case of that "yes, it sucks. But the alternatives just plain don't work, or work in WAY fewer circumstances". If I have a strange architecture or install, autoconf is often the only one that works.
  - IshKebab 3y ago
    What strange architecture are you talking about? I don't think the cost of supporting dead architectures is worth it. CMake is a clear better choice (it also sucks but it sucks waaaay less).
    - knorker 3y ago
      I didn't want to start shitting on alternatives, but cmake is in my experience clearly the worst of all options. It seems that more often than not, cmake setups work on the developer's own machine and nothing else. And the choice of undocumented "-DSOMETHING" is about as user hostile as it gets.
frankjr 3y ago
Reminds of the note at the bottom of Fish releases. It's there because the build system cannot determine the current version for some reason. Hopefully that will go away now that they have switched to a different language / build system. The custom tarball is used by Arch Linux at the very least. https://github.com/fish-shell/fish-shell/releases/tag/3.7.1 https://github.com/fish-shell/fish-shell/releases/tag/3.7.1 https://github.com/fish-shell/fish-shell/issues/7772#issuecomment-793982192 https://github.com/fish-shell/fish-shell/issues/7772#issueco... https://gitlab.archlinux.org/archlinux/packaging/packages/fish/-/blob/661c4ae6a91a7ead1a6bdb45b186fe6d0e698fac/PKGBUILD#L22 https://gitlab.archlinux.org/archlinux/packaging/packages/fi...
k8svet 3y ago
Yes, and in fact, it's quite a huge pain even in the packaging side. Everytime I have to deal with autotools in nixpkgs, I end up having to rewrite the derivation to build from actual freaking source instead of a pre-mangled non-release tarball. In fact, this makes me want to RFC *banning* release tarball usage.
- deleted 3y ago
  [deleted]
binary132 3y ago
you mean like LLVM’s source release tars? yeah, who would do that, haha?

deleted 3y ago

[deleted]

pdw 3y ago

Flagged: How is this "via a project dev"? Which project? The guy doesn't seem to have anything to do with the discovery of this backdoor, and he just summarizes Andres Freund's announcement.

bhaney 3y ago
The author of this is a gentoo maintainer who focuses on security work. Most distros have a couple of point people handling this issue for them at this point, and they are all likely pretty well informed. If you want clarification on something, you can just ask instead of announcing that you've flagged it.
jethro_tell 3y ago
Its from the Gentoo maintainer of the package and libarchive project dev that submits changes, patches, is involved in the mail list and has ops on the groups IRC chan. It's included in Lasse Collin's FAQ now that he has been found, but in his absence, Sam was speaking for the project on IRC chan and filed the gentoo bug also linked at the bottom of in Collin's FAQ: https://tukaani.org/xz-backdoor/ https://tukaani.org/xz-backdoor/ You could have just asked and I could have elaborated.

choeger 3y ago

So the attacker put the payload into the tests folder of the compression library and injected it upon build via some arcane build script? Kudos. That's quite a smart way to hide your backdoor. Nevertheless, they got caught. So big respect for the analysis. Amazing work. That being said, what does this attack tell you about the future of software security? Free software could easily be compromised by having a highly paid team create such a payload and then convince some maintainer, one way or another, to deploy it. How do we back off from that cliff?

cassianoleal 3y ago
Security is hard. In this case, it seems like there's a lot that could be done to reduce the chance of being exploited. Why are obscure binaries in the repository at all in the first place, even if in the test dir? Surely those binaries can be generated as a setup step prior to the tests actually running. Furthermore, why are there scripts present in the release package that have not been added to the source code? Also, if I understood it correctly, the compromised binaries are also present in the release tarball. Why are they part of the release if they are supposedly test fixtures rather than part of the application code?
- cesarb 3y ago
  > Why are they part of the release if they are supposedly test fixtures rather than part of the application code? Test fixtures should be distributed together with the application code, so that the actually built application can be tested. Ideally, the application should be tested every time it's built, to ensure that for instance a change in the compiler being used didn't break it.
  - cassianoleal 3y ago
    Thanks, that makes sense. I think in my mind the whole chain was still source code->compile->test->distribute from a single repo which is obviously not the case. It’s more like source->package->test->ship to distros->compile->test->package->distribute
3y ago

pseudo0 3y ago

Main discussion thread here: https://news.ycombinator.com/item?id=39865810 https://news.ycombinator.com/item?id=39865810 Backdoor in upstream xz/liblzma leading to SSH server compromise (openwall.com) 1848 points by rkta 5 hours ago | 654 comments

AshamedCaptain 3y ago

> The payload only activates if the running program has the process name /usr/bin/sshd. This means that systems that put sshd in /usr/sbin or another folder are not vulnerable. This further suspects targeting systemd systems due to their usrmerge initiative putting all binaries in /usr/bin. This is highly likely wrong. The hack targets rpm and deb distros and as far as I know _all of them_ put sshd in /usr/sbin. The original oss-security report specifically said arg0 had to be /usr/sbin/sshd and I can reproduce his findings about increased runtime if you do so. This /usr/bin vs /usr/sbin also has nothing do with usrmerge. This type of mistake casts doubt in the rest of analysis.

redserk 3y ago
There is no rational reason to justify throwing out the entire analysis over one mistake.
- AshamedCaptain 3y ago
  I said "Casts doubt". In the same way a typo in the headline, a logic error in the first bullet point, a misconception in the introduction, etc. casts doubt on the rest of a publication. And this was no simple typo, as there was an accompanying and erroneous justification (which btw also hints to a lack of knowledge in the topic). This bullet was practically the easiest to verify out of the entire article and was already showing a contradiction. You might enjoy publications where fact-checking is not high in the list of priorities of the author (this article contains a disclaimer saying as much), but most "rational" readers should be interested in knowing how light the fact-checking indeed was, specially in a topic as important as this.

aimonster2 3y ago

So wait, if I did a find / -n 'liblzma*' and saw 5.4, I'm good? This is only 5.6 and 5.6.1, right?

xorcist 3y ago
From what is publicly known at the moment, yes. Note however that xzutils home page says that "versions 5.2.12, 5.4.3 and later have been signed with Jia Tan's OpenPGP key" so there would have been plenty more opportunities. We may just have seen the beginning. Whoever did this played the long game. Also note that there was proposed patches by this compromised project maintainer to oss-fuzz and valgrind to avoid the detection of this backdoor.
botanical 3y ago
The attacker had 750 previous commits; maybe it was all for this or maybe there are more vulnerabilities.

xorcist 3y ago

People are prematurely celebrating this did not reach a stable distribution. But we haven't seen the full fallout yet. How many debian-developer's machines have been compromised, for example?

threeseed 3y ago
I don't think that's the biggest concern. There are 2 years of commits that potentially have backdoors.

timattrn 3y ago

so this backdoor is a package which installs a fake sshd? what is selinux for?