12 ms·

Xz/liblzma: Bash-stage Obfuscation Explained

535 points by ecliptik 3y ago

deleted 3y ago
[deleted]
politelemon 3y ago
Thanks the simplified explanation and noisy image comparison is quite appreciated. It gives me a good grasp of what people mean by the sophistication involved. I also saw a comment on reddit mentioning that the "sandboxing" method was sabotaged with a dot. It's on the line just after "#include <sys/prctl.h>" you can see a dot all the way on the left. https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2... https://old.reddit.com/r/linux/comments/1brhlur/xz_utils_backdoor/kx9b9wc/ https://old.reddit.com/r/linux/comments/1brhlur/xz_utils_bac...
- temp12237792 3y ago
  OMG that's evil. The diff just shows: + +. + and the dot goes unnoticed
  - cqqxo4zV46cp 3y ago
    like a more (ostensibly) malicious “goto fail”
  - emmelaich 3y ago
    I wonder why they didn't use a non-breaking space or similar. I guess it's possible a nbsp would stand out even more.
    - jetpks 3y ago
      the extra dot is easily hand waved away as a mistake. a non breaking space looks intentional.
    - IshKebab 3y ago
      They could have just misspelt one of the constants. Even less obvious and more deniable. There's multiple things like this in this backdoor that seems like they've been super sneaky (using a compile check to disable Landlock is genius) but then half-assed the last step.
    - account42 3y ago
      Plausible deniability probably. A dot could be a typo, a NBSP is less likely.
- ihsoy 3y ago
  This is very likely just a mistake and not deliberate. a) absolutely nobody uses cmake to build this packet b) if you try to build the packet with cmake and -DENABLE_SANDBOX=landlock, the build just fails: https://i.imgur.com/7xbeWFx.png https://i.imgur.com/7xbeWFx.png The "." does not disable sandboxing, it just makes it impossible to build with cmake. If anyone had ever actually tried building it with cmake, they would get the error and realize that something is wrong. It makes absolutely no sense that this would be malicious attempt to reduce security.
  - everybackdoor 3y ago
    [dead]
- gravescale 3y ago
  I really hate writing these compile/build time conditional things. It's hard to have tests that it's enabled when it should be and disabled when it isn't, especially if it's in the build system where there's no unit test framework. And that's with the failure party being just accidentally borking it so the test always fails or always succeeds when it shouldn't. You can see why it's a juicy target for malicious actions.
sega_sai 3y ago
Did anyone search github yet for similar head | tail tricks ? I doubt it was invented just for this.
- nabakin 3y ago
  Opportunity to write a paper
  - rmast 3y ago
    Maybe some analysis of odd patterns in entropy of binary files committed to repositories could pick out some to look at a bit deeper?
- saagarjha 3y ago
  It’s clever but not entirely novel, this is kind of the intended usecase for these
  - rmast 3y ago
    The use of head/tail for deobfuscation also isn’t visible as plain text in the repository or release tarball, which makes searching for its use in other repositories more difficult (unless a less obfuscated version was tested elsewhere).
- oogali 3y ago
  I've generally seen this with Unix installers from commercial software vendors. You get a giant .sh file that displays a license, asks you to accept, then upon acceptance, cats itself, pipes through head/tail, into cpio to extract the actual assets.
jijijijij 3y ago
How are the binary files passed to those stage-0 commands?
- viraptor 3y ago
  They already exist in the source. They're split in the compression test files themselves. (Unless you meant some other binaries?)
  - jijijijij 3y ago
    Yeah, but how exactly are they passed to those commands? I don't see/understand that part. I don't see the "take this file here" part.
    - chaosite 3y ago
      That's for 2 reasons: 1. It might not be there in the place where you're looking. It exists in the m4 in the release tarballs, not in the git repo. 2. It's highly obfuscated.
      - jijijijij 3y ago
        No, as far as I understand the binary files must be pointed at here: '$gl_am_configmake' ... But I don't see how. This: 'gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/`' seem to match the '####Hello####', but, as far as I can see, that's supposed to be the already converted script?! I presumed the binary files not to contain human readable strings, maybe that's the whole confusion.
        credulousperson 3y ago
        Opening bad-3-corrupt_lzma2.xz in an editor reveals it indeed has the string ####Hello####. I don't know enough about lzma compression streams to explain how this appears in the "compressed" version of the payload, but it does.
senoralligator 3y ago
https://github.com/tukaani-project/.github/issues/2 https://github.com/tukaani-project/.github/issues/2
- mappu 3y ago
  That's quite funny - yes, not only is this a horrible wilful backdoor, it is also a GPL violation since the backdoor is a derived work without included source / distributed not in "the preferred form for modification".
  - lucb1e 3y ago
    Sadly, it looks like xz-utils is actually public domain. Only some of the scripts (like xzgrep) were GPL. So it is and remains only a joke, not an actual violation, hilarious as that would have been to enforce
Martinussen 3y ago
How on earth did any of this make it through a code review and get merged in? It seems absurdly careless, unless I am missing something.
- asveikau 3y ago
  The commit messages for the test files claim they used an RNG to generate them. The guy making the release tarball then put the final line in the right place without checking it in.
  - sgerenser 3y ago
    What is the reason distros are still building from release tarballs rather than a git checkout that can be verified against a public git repo?
    - dvhh 3y ago
      code repository are not necessarily git based. Plus you would need to put the effort in monitoring the activity of the repository for changes. Until last month, would you refuse a tar package from the official maintainer, I wouldn't, especially when there was a mention of a bugfix that might have been tripping our build system For example nginx is using mercurial (with admittedly a github mirror for convenience), and a lot of OSS are still using subversion and CVS, and my guess is that there are some project which might run with less free source control software ( most likely for historical purpose, or use case that might be the strong point of that software). Other than that, why wouldn't the user be the one to build their own software package.
    - asveikau 3y ago
      I think a lot of it is probably historical. When debian or red hat infrastructure came up there was no git; projects were still often in source control during development but tarballs were still the major distribution mechanism to normal people. Though before git they'd sometimes have packages that would be based on an SVN or cvs snapshot back in the day, in absence of releases. I believe what happens in debian is that they host their own mirror of source tarballs, since going to some random website or git repo means it could be taken down from under them. So I guess if the package is built straight from a repo they'd probably make a tarball of it anyway.
marco_patino 3y ago
Now the GitHub repo has been disabled by GitHub due to violation of GitHub's terms. https://github.com/tukaani-project/xz https://github.com/tukaani-project/xz
1letterunixname 3y ago
Never allow complexity in code or so-called engineers who ask to merge tons of shitty code. Get rid of that shit and don't trust committers blindly. Anyone who enables this crap is also a liability.
- plg94 3y ago
  You do realize that "that shit" was part of the obfuscated and xz-compressed backdoor hidden as binary test file, right? It was never committed in plain sight. You can go to https://git.tukaani.org/xz.git https://git.tukaani.org/xz.git and look at the commits yourself – while the commits of the attacker are not prime examples of "good commits", they don't have glaringly obvious red flags either. This backdoor was very sophisticated and well-hidden, so your comment misses the point completely.
  - johnisgood 3y ago
    > It was never committed in plain sight. It was though. I have seen those two test files being added by a commit on GitHub. Unfortunately it has been disabled by now, so I cannot give you a working link.
    - fullstop 3y ago
      It really wasn't, though. commit 74b138d2a6529f2c07729d7c77b1725a8e8b16f1 Author: Jia Tan <jiat0218@gmail.com> Date: Sat Mar 9 10:18:29 2024 +0800 Tests: Update two test files. The original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files. diff --git a/tests/files/bad-3-corrupt_lzma2.xz b/tests/files/bad-3-corrupt_lzma2.xz index 926f95b0..f9ec69a2 100644 Binary files a/tests/files/bad-3-corrupt_lzma2.xz and b/tests/files/bad-3-corrupt_lzma2.xz differ diff --git a/tests/files/good-large_compressed.lzma b/tests/files/good-large_compressed.lzma index 8450fea8..878991f3 100644 Binary files a/tests/files/good-large_compressed.lzma and b/tests/files/good-large_compressed.lzma differ Would you bat an eye at this? If it were from a trusted developer and the code was part of a test case? If you looked at strings contained within the bad file, you might notice that this was not random: 7zXZ ####Hello#### 7zXZ w,( 7zXZ ####World#### But, again, this was a test case.
mrkramer 3y ago
The whole XZ drama reminds me of this[1], in another words, verify the identity of open source maintainer/s and question their motive for joining the open source project. Also reminded me of the relevant XKCD meme[2]. Speaking of obfuscation; I'm not a programmer but I did some research in Windows malware RE and what stuck with me is that every code that is obfuscated or every code that is unused is automatically suspicious. There is no purpose for obfuscated code in the open source non-profit software project and there is no purpose for extra code that is unused. Extra/redundant code is most likely junk code meant to confuse the reverse engineer when s/he is debugging the binary. [1] https://lwn.net/Articles/846272/ https://lwn.net/Articles/846272/ [2] https://xkcd.com/2347/ https://xkcd.com/2347/
- dboreham 3y ago
  > verify the identity of open source maintainer/s and question their motive for joining the open source project. This kind of goes against the whole "free" thing.
  - mrkramer 3y ago
    Anybody is free to contribute if s/he is contributing in a good will but what happens if you don't know who they are and what are their motives? You can look at the their track record for example, that's another way to determining their credibility. In another words you need to establish trust somehow. Idk if this specific individual that backdoored XZ had a track record of contributing to other open source projects(in a good will) or if s/he just out of the blue starting contributing to this project. I read somewhere that somebody else recommended him or vouched for him. Somebody needs to fill me in with the details.
    - chii 3y ago
      Just because you know the identity of an individual, doesn't mean they are trustworthy. They might be compromised, or they might be willfully doing it for their own personal gain, regardless of their existing reputation (or even, leveraging their existing reputation - bernie madoff was a well known and well respected investment banker).
vodou 3y ago
A disturbing thought here is that unit tests opened up an attack vector. Without the tests this would have been much harder to hide.
- oogali 3y ago
  Furthermore, the attacker covered their tracks on the initial payload with an innocuous paragraph in the README. ("Nothing to see here!") bad-3-corrupt_lzma2.xz has three Streams in it. The first and third streams are valid xz Streams. The middle Stream has a correct Stream Header, Block Header, Index and Stream Footer. Only the LZMA2 data is corrupt. This file should decompress if --single-stream is used. The strings of `####Hello####` and `####World####` are there so that if you actually follow the instructions in the README, you get a seemingly valid result. $ cat tests/files/bad-3-corrupt_lzma2.xz | xz -d --single-stream ####Hello#### They're shell comments so it won't interfere with payload execution. And lastly, they act as a marker that can be used by a later regex to locate the file _without_ referencing it by name directly nor using the actual Hello and World strings. $ gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null` $ echo $gl_am_configmake ./tests/files/bad-3-corrupt_lzma2.xz
- creato 3y ago
  For security critical projects, it seems like it would make sense to try to set up the build infrastructure to error (or at least warn!) when binary files are being included in the build. This should be done transitively, so when linux distros attempted to update to this new version of liblzma, the build would fail (or warn) about this new binary dependency. I don't know how common this practice is in the linux distro builds. Obviously if it's common, it would take a lot of work to clean up to make this possible, even if it's even possible in the first place. It seems like something that would be doable with bazel, but I'm not sure about other build systems.
  - bicolao 3y ago
    seems easily worked around with base64 and friends.
deleted 3y ago
[deleted]
zzzeek 3y ago
can we start considering binary files committed to a repo, even as data for tests, to be a huge red flag, and that the binary files themselves should instead, to the greatest extent possible, be generated at testing time by source code that's stated as reviewable cleartext (though I think this might be very difficult for some situations). This would make it much harder (though of course we can never really say "impossible") to embed a substantial payload in this way. when binary files are part of a test suite, they are typically trying to illustrate some element of the program being tested, in this case a file that was incorrectly xz-encoded. Binary files like these weren't typed by hand, they will always ultimately come from something plaintext source, modulo whatever "real world" data came in, like randomly generated numbers, audio or visual data, etc. Here's an example! My own SQLAlchemy repository has a few binary files in it! https://github.com/sqlalchemy/sqlalchemy/blob/main/test/binary_data_one.dat https://github.com/sqlalchemy/sqlalchemy/blob/main/test/bina... oh noes. Why are those files there? well in this case I just wanted to test that I can send large binary BLOBs into the database driver and I was lazy. This is actually pretty dumb, the two binary files here add 35K of useless crap to the source, and I could just as easily generate this binary data on the fly using a two liner that spits out random bytes. Anyone could see that two liner and know that it isn't embedding a malicious payload. If I wanted to generate a poorly formed .xz file, I'd illustrate source code that generates random data, runs it through .xz, then applies "corruption" to it, like zeroing out the high bit of every byte. The process by which this occurs would be all reviewable in source code. Where I might be totally off here is if you're an image processing library and you want to test filters on an image, and you have the "before" and "after" images, or something similar for audio information, or other scientifically-generated real world datapoints that have some known meaning. That might be difficult to generate programmatically, and I guess even if said data were valid, payloads could be applied steganographically. So I don't know! But just like nobody would ever accept a PR that has a "curl https://some_rando_url/myfile.zip https://some_rando_url/myfile.zip" inside of it, we should not accept PRs that have non-cleartext binary data in them, or package them, without really vetting the contents of those binary files. The simple presence of a binary file in a PR can certainly be highlighted, github could put a huge red banner BINARY FILES IN THIS PR. Downstream packagers for distros like Debian, Redhat etc. would ideally be similarly skeptical of new binary files that appear in the source downloads, and tooling can be applied to highlight the appearance of such files. Packagers would be on the hook to confirm the source of these binary files, or ensure they are deleted (even disabling tests if necessary) before the build process is performed.
notnmeyer 3y ago
i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it? i live in a different part of the dev world, but could this be written to be less obtuse so it’s more obvious what’s happening? i get that a maintainer can still get malicious code in without the same rigor as an unaffiliated contributor, but surely there’s a better way than piles of “concise” (inadvertently obfuscated?) code?
- MPSimmons 3y ago
  >this convoluted mess of bash is a smell isn’t it At a glance, I don't think so. At least, not the fact that the bash looks like a convoluted mess. Sometimes that's how a tight bash script looks (whether it SHOULD be tight or not is a different argument). For me, the thing that looks suspicious is the repeated tr calls. If I see that, I assume someone is trying to be clever, where 'clever' here is a pejorative. If I were a maintainer and someone checked that in, I'd ask them to walk me through what it was doing, because there's almost always a better solution to avoid chaining things like that. The real problem here is that there wasn't another maintainer to look at this code being brought in. A critical piece of the stack relied on a single person who, in this case, was malicious.
- duped 3y ago
  The shell is generated, not written. There are mountains of generated shell configuration code out there due to the prevalence of autoconf, which relies on these M4 (a macro preprocessor) scripts to generate thousands of lines of unreadable shell script (which has to be portable). This is how a non negligible number of your tools are built.
- semiquaver 3y ago
  > inadvertently The whole point is that it’s intentionally obfuscated!
- ekidd 3y ago
  > i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it? It's a very old smell, basically. The central problem is that back in the 80s and 90s, there were scads of different Unix-like systems, each with their own warts and missing features. And software authors wanted to minimize their build dependencies. So many communities standardized on automating builds using shell scripts, which worked everywhere. But shell scripts were a pain to write, so people would generate shell scripts using tools like the M4 macro preprocessor. And this is why many projects have a giant mass of opaque shell scripts, just in case someone wants to run the code on AIX or some broken ancient Unix. If you wanted to get rid of these impenetrable thickets of shell, you could: 1. Sharply limit the number of platforms you support. 2. You could standardize on much cleaner build tools. 3. You could build more key infrastructure in languages which don't require shell to build portably. But this would be a massive undertaking, and a ton of key C libraries are maintained by one or two unpaid volunteers. And dropping support for "Obscurnix-1997" tends to be a fairly controversial decision. So much of our key infrastructure remains surrounded by a morass of mysterious machine-generated shell scripts.
minimax 3y ago
> if this was found by accident, how many things still remain undiscovered. This, to me, is the most important question. There is no way Andres Freund just happened to find the _only_ backdoored popular open source project out there. There must be like a dozen of these things in the wild?
- roflmaostc 3y ago
  Maybe. But maybe there is not many (critical) ones out there. Otherwise, I believe we would encounter more often those kind of situations.
- mistrial9 3y ago
  "(B)urn (A)ll (S)atanic (H)elper-scripts" ? get the kindling?
  - paulmd 3y ago
    thou shalt not make a machine in the likeness of a human mind
    - supersuryan 3y ago
      Human minds shine bright, but machines lend their light
      - slater 3y ago
        Human minds shine bright, and machines steal their light
- whywhywhywhy 3y ago
  It wasn’t found accidentally, he felt it. There’s a difference. Next one will definitely be less careless with adding time to it’s execution.
benlivengood 3y ago
Deterministic/repeatable builds can help with issues like this: once the binaries exist from checksummed code repository and are hashed, the tests can do whatever they want but if the final binaries change from the recorded hashes they shouldn't get packaged. This is in general a problem with traditional permission models. Pure capabilities would never leave binaries writable by anything other than the compiler/linker, shrinking the attack surface.
- edflsafoiewq 3y ago
  Running the tests does not modify the binary. The build script was modified to create a malicious .o file which was linked into the final binary by the linker as normal. Tests were only involved in that the malicious .o was hidden inside binary test files.
  - paulmd 3y ago
    letting dist builds be linked against test resources is a design defect to begin with, and the fact that this is easy/trivial/widely-accepted is a general indication of the engineering culture problems around C. Nobody in Java word is linking against test resources in prod, and a sanely designed build system should in fact make this extremely difficult. That shit went away in the maven/gradle days - which is for a reason, ant is basically makefiles for Java, gradle/maven are a build system not a pile of scripts. And that transition happened 20 years ago! If you can’t even prevent a test resource being linked into a final build you are not serious. I don’t care about legacy whatever, that’s an obvious baseline metric for security culture / build engineering. Maybe not “prevent” but like, tooling should absolutely make it blindingly obvious that you’re violating best-practices by disabling scoping rules or including unusual source/resource roots, etc. C has never moved past the 1970 mindset of build being a pile of scripts with a superstructure bolted on. Just like Ant. Even the attempts to fix C’s build are just better ways to programmatically generate better bash scripts to keep you going off the rails.
    - malkia 3y ago
      Good point, bot sure if enforced there, but systems like bazel (buck?) and others have ways to mark build nodes as "testonl"
anthk 3y ago
Using LTS distros can shield you a bit. Slackware uses lzma (tar.xz) for it's packages I think, and beside of -current, the last stable release didn't have that issue. Also, if you want a step up on the freedom ladder, Hyperbola GNU neither had that issue. EDIT: Also, Slackware -current neither doesn't link sshd against xz, nor uses systemd.
deleted 3y ago
[deleted]
ds9soft 3y ago
The whole C land, including build tools, old unix utils, is a security mess waiting to be exploited, and it's going to be exploited. Just look how easy it's to break everything with a single dot. It's time people realize we can't bet the world's security on C. Please use Ada or Rust with modern tooling.
- brenoguim 3y ago
  If you add the dot to Rust it doesnt break?

535 points by ecliptik 3y ago

deleted 3y ago

[deleted]

Thanks the simplified explanation and noisy image comparison is quite appreciated. It gives me a good grasp of what people mean by the sophistication involved. I also saw a comment on reddit mentioning that the "sandboxing" method was sabotaged with a dot. It's on the line just after "#include <sys/prctl.h>" you can see a dot all the way on the left. https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2... https://old.reddit.com/r/linux/comments/1brhlur/xz_utils_backdoor/kx9b9wc/ https://old.reddit.com/r/linux/comments/1brhlur/xz_utils_bac...

temp12237792 3y ago
OMG that's evil. The diff just shows: + +. + and the dot goes unnoticed
- cqqxo4zV46cp 3y ago
  like a more (ostensibly) malicious “goto fail”
- emmelaich 3y ago
  I wonder why they didn't use a non-breaking space or similar. I guess it's possible a nbsp would stand out even more.
  - jetpks 3y ago
    the extra dot is easily hand waved away as a mistake. a non breaking space looks intentional.
  - IshKebab 3y ago
    They could have just misspelt one of the constants. Even less obvious and more deniable. There's multiple things like this in this backdoor that seems like they've been super sneaky (using a compile check to disable Landlock is genius) but then half-assed the last step.
  - account42 3y ago
    Plausible deniability probably. A dot could be a typo, a NBSP is less likely.
ihsoy 3y ago
This is very likely just a mistake and not deliberate. a) absolutely nobody uses cmake to build this packet b) if you try to build the packet with cmake and -DENABLE_SANDBOX=landlock, the build just fails: https://i.imgur.com/7xbeWFx.png https://i.imgur.com/7xbeWFx.png The "." does not disable sandboxing, it just makes it impossible to build with cmake. If anyone had ever actually tried building it with cmake, they would get the error and realize that something is wrong. It makes absolutely no sense that this would be malicious attempt to reduce security.
- everybackdoor 3y ago
  [dead]
gravescale 3y ago
I really hate writing these compile/build time conditional things. It's hard to have tests that it's enabled when it should be and disabled when it isn't, especially if it's in the build system where there's no unit test framework. And that's with the failure party being just accidentally borking it so the test always fails or always succeeds when it shouldn't. You can see why it's a juicy target for malicious actions.

sega_sai 3y ago

Did anyone search github yet for similar head | tail tricks ? I doubt it was invented just for this.

nabakin 3y ago
Opportunity to write a paper
- rmast 3y ago
  Maybe some analysis of odd patterns in entropy of binary files committed to repositories could pick out some to look at a bit deeper?
saagarjha 3y ago
It’s clever but not entirely novel, this is kind of the intended usecase for these
- rmast 3y ago
  The use of head/tail for deobfuscation also isn’t visible as plain text in the repository or release tarball, which makes searching for its use in other repositories more difficult (unless a less obfuscated version was tested elsewhere).
oogali 3y ago
I've generally seen this with Unix installers from commercial software vendors. You get a giant .sh file that displays a license, asks you to accept, then upon acceptance, cats itself, pipes through head/tail, into cpio to extract the actual assets.

jijijijij 3y ago

How are the binary files passed to those stage-0 commands?

viraptor 3y ago
They already exist in the source. They're split in the compression test files themselves. (Unless you meant some other binaries?)
- jijijijij 3y ago
  Yeah, but how exactly are they passed to those commands? I don't see/understand that part. I don't see the "take this file here" part.
  - chaosite 3y ago
    That's for 2 reasons: 1. It might not be there in the place where you're looking. It exists in the m4 in the release tarballs, not in the git repo. 2. It's highly obfuscated.
    - jijijijij 3y ago
      No, as far as I understand the binary files must be pointed at here: '$gl_am_configmake' ... But I don't see how. This: 'gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/`' seem to match the '####Hello####', but, as far as I can see, that's supposed to be the already converted script?! I presumed the binary files not to contain human readable strings, maybe that's the whole confusion.
      - credulousperson 3y ago
        Opening bad-3-corrupt_lzma2.xz in an editor reveals it indeed has the string ####Hello####. I don't know enough about lzma compression streams to explain how this appears in the "compressed" version of the payload, but it does.

senoralligator 3y ago

https://github.com/tukaani-project/.github/issues/2 https://github.com/tukaani-project/.github/issues/2

mappu 3y ago
That's quite funny - yes, not only is this a horrible wilful backdoor, it is also a GPL violation since the backdoor is a derived work without included source / distributed not in "the preferred form for modification".
- lucb1e 3y ago
  Sadly, it looks like xz-utils is actually public domain. Only some of the scripts (like xzgrep) were GPL. So it is and remains only a joke, not an actual violation, hilarious as that would have been to enforce

Martinussen 3y ago

How on earth did any of this make it through a code review and get merged in? It seems absurdly careless, unless I am missing something.

asveikau 3y ago
The commit messages for the test files claim they used an RNG to generate them. The guy making the release tarball then put the final line in the right place without checking it in.
- sgerenser 3y ago
  What is the reason distros are still building from release tarballs rather than a git checkout that can be verified against a public git repo?
  - dvhh 3y ago
    code repository are not necessarily git based. Plus you would need to put the effort in monitoring the activity of the repository for changes. Until last month, would you refuse a tar package from the official maintainer, I wouldn't, especially when there was a mention of a bugfix that might have been tripping our build system For example nginx is using mercurial (with admittedly a github mirror for convenience), and a lot of OSS are still using subversion and CVS, and my guess is that there are some project which might run with less free source control software ( most likely for historical purpose, or use case that might be the strong point of that software). Other than that, why wouldn't the user be the one to build their own software package.
  - asveikau 3y ago
    I think a lot of it is probably historical. When debian or red hat infrastructure came up there was no git; projects were still often in source control during development but tarballs were still the major distribution mechanism to normal people. Though before git they'd sometimes have packages that would be based on an SVN or cvs snapshot back in the day, in absence of releases. I believe what happens in debian is that they host their own mirror of source tarballs, since going to some random website or git repo means it could be taken down from under them. So I guess if the package is built straight from a repo they'd probably make a tarball of it anyway.

marco_patino 3y ago

Now the GitHub repo has been disabled by GitHub due to violation of GitHub's terms. https://github.com/tukaani-project/xz https://github.com/tukaani-project/xz

1letterunixname 3y ago

Never allow complexity in code or so-called engineers who ask to merge tons of shitty code. Get rid of that shit and don't trust committers blindly. Anyone who enables this crap is also a liability.

plg94 3y ago
You do realize that "that shit" was part of the obfuscated and xz-compressed backdoor hidden as binary test file, right? It was never committed in plain sight. You can go to https://git.tukaani.org/xz.git https://git.tukaani.org/xz.git and look at the commits yourself – while the commits of the attacker are not prime examples of "good commits", they don't have glaringly obvious red flags either. This backdoor was very sophisticated and well-hidden, so your comment misses the point completely.
- johnisgood 3y ago
  > It was never committed in plain sight. It was though. I have seen those two test files being added by a commit on GitHub. Unfortunately it has been disabled by now, so I cannot give you a working link.
  - fullstop 3y ago
    It really wasn't, though. commit 74b138d2a6529f2c07729d7c77b1725a8e8b16f1 Author: Jia Tan <jiat0218@gmail.com> Date: Sat Mar 9 10:18:29 2024 +0800 Tests: Update two test files. The original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files. diff --git a/tests/files/bad-3-corrupt_lzma2.xz b/tests/files/bad-3-corrupt_lzma2.xz index 926f95b0..f9ec69a2 100644 Binary files a/tests/files/bad-3-corrupt_lzma2.xz and b/tests/files/bad-3-corrupt_lzma2.xz differ diff --git a/tests/files/good-large_compressed.lzma b/tests/files/good-large_compressed.lzma index 8450fea8..878991f3 100644 Binary files a/tests/files/good-large_compressed.lzma and b/tests/files/good-large_compressed.lzma differ Would you bat an eye at this? If it were from a trusted developer and the code was part of a test case? If you looked at strings contained within the bad file, you might notice that this was not random: 7zXZ ####Hello#### 7zXZ w,( 7zXZ ####World#### But, again, this was a test case.

mrkramer 3y ago

The whole XZ drama reminds me of this[1], in another words, verify the identity of open source maintainer/s and question their motive for joining the open source project. Also reminded me of the relevant XKCD meme[2]. Speaking of obfuscation; I'm not a programmer but I did some research in Windows malware RE and what stuck with me is that every code that is obfuscated or every code that is unused is automatically suspicious. There is no purpose for obfuscated code in the open source non-profit software project and there is no purpose for extra code that is unused. Extra/redundant code is most likely junk code meant to confuse the reverse engineer when s/he is debugging the binary. [1] https://lwn.net/Articles/846272/ https://lwn.net/Articles/846272/ [2] https://xkcd.com/2347/ https://xkcd.com/2347/

dboreham 3y ago
> verify the identity of open source maintainer/s and question their motive for joining the open source project. This kind of goes against the whole "free" thing.
- mrkramer 3y ago
  Anybody is free to contribute if s/he is contributing in a good will but what happens if you don't know who they are and what are their motives? You can look at the their track record for example, that's another way to determining their credibility. In another words you need to establish trust somehow. Idk if this specific individual that backdoored XZ had a track record of contributing to other open source projects(in a good will) or if s/he just out of the blue starting contributing to this project. I read somewhere that somebody else recommended him or vouched for him. Somebody needs to fill me in with the details.
  - chii 3y ago
    Just because you know the identity of an individual, doesn't mean they are trustworthy. They might be compromised, or they might be willfully doing it for their own personal gain, regardless of their existing reputation (or even, leveraging their existing reputation - bernie madoff was a well known and well respected investment banker).

vodou 3y ago

A disturbing thought here is that unit tests opened up an attack vector. Without the tests this would have been much harder to hide.

oogali 3y ago
Furthermore, the attacker covered their tracks on the initial payload with an innocuous paragraph in the README. ("Nothing to see here!") bad-3-corrupt_lzma2.xz has three Streams in it. The first and third streams are valid xz Streams. The middle Stream has a correct Stream Header, Block Header, Index and Stream Footer. Only the LZMA2 data is corrupt. This file should decompress if --single-stream is used. The strings of `####Hello####` and `####World####` are there so that if you actually follow the instructions in the README, you get a seemingly valid result. $ cat tests/files/bad-3-corrupt_lzma2.xz | xz -d --single-stream ####Hello#### They're shell comments so it won't interfere with payload execution. And lastly, they act as a marker that can be used by a later regex to locate the file _without_ referencing it by name directly nor using the actual Hello and World strings. $ gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null` $ echo $gl_am_configmake ./tests/files/bad-3-corrupt_lzma2.xz
creato 3y ago
For security critical projects, it seems like it would make sense to try to set up the build infrastructure to error (or at least warn!) when binary files are being included in the build. This should be done transitively, so when linux distros attempted to update to this new version of liblzma, the build would fail (or warn) about this new binary dependency. I don't know how common this practice is in the linux distro builds. Obviously if it's common, it would take a lot of work to clean up to make this possible, even if it's even possible in the first place. It seems like something that would be doable with bazel, but I'm not sure about other build systems.
- bicolao 3y ago
  seems easily worked around with base64 and friends.

deleted 3y ago

[deleted]

zzzeek 3y ago

can we start considering binary files committed to a repo, even as data for tests, to be a huge red flag, and that the binary files themselves should instead, to the greatest extent possible, be generated at testing time by source code that's stated as reviewable cleartext (though I think this might be very difficult for some situations). This would make it much harder (though of course we can never really say "impossible") to embed a substantial payload in this way. when binary files are part of a test suite, they are typically trying to illustrate some element of the program being tested, in this case a file that was incorrectly xz-encoded. Binary files like these weren't typed by hand, they will always ultimately come from something plaintext source, modulo whatever "real world" data came in, like randomly generated numbers, audio or visual data, etc. Here's an example! My own SQLAlchemy repository has a few binary files in it! https://github.com/sqlalchemy/sqlalchemy/blob/main/test/binary_data_one.dat https://github.com/sqlalchemy/sqlalchemy/blob/main/test/bina... oh noes. Why are those files there? well in this case I just wanted to test that I can send large binary BLOBs into the database driver and I was lazy. This is actually pretty dumb, the two binary files here add 35K of useless crap to the source, and I could just as easily generate this binary data on the fly using a two liner that spits out random bytes. Anyone could see that two liner and know that it isn't embedding a malicious payload. If I wanted to generate a poorly formed .xz file, I'd illustrate source code that generates random data, runs it through .xz, then applies "corruption" to it, like zeroing out the high bit of every byte. The process by which this occurs would be all reviewable in source code. Where I might be totally off here is if you're an image processing library and you want to test filters on an image, and you have the "before" and "after" images, or something similar for audio information, or other scientifically-generated real world datapoints that have some known meaning. That might be difficult to generate programmatically, and I guess even if said data were valid, payloads could be applied steganographically. So I don't know! But just like nobody would ever accept a PR that has a "curl https://some_rando_url/myfile.zip https://some_rando_url/myfile.zip" inside of it, we should not accept PRs that have non-cleartext binary data in them, or package them, without really vetting the contents of those binary files. The simple presence of a binary file in a PR can certainly be highlighted, github could put a huge red banner BINARY FILES IN THIS PR. Downstream packagers for distros like Debian, Redhat etc. would ideally be similarly skeptical of new binary files that appear in the source downloads, and tooling can be applied to highlight the appearance of such files. Packagers would be on the hook to confirm the source of these binary files, or ensure they are deleted (even disabling tests if necessary) before the build process is performed.

notnmeyer 3y ago

i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it? i live in a different part of the dev world, but could this be written to be less obtuse so it’s more obvious what’s happening? i get that a maintainer can still get malicious code in without the same rigor as an unaffiliated contributor, but surely there’s a better way than piles of “concise” (inadvertently obfuscated?) code?

MPSimmons 3y ago
>this convoluted mess of bash is a smell isn’t it At a glance, I don't think so. At least, not the fact that the bash looks like a convoluted mess. Sometimes that's how a tight bash script looks (whether it SHOULD be tight or not is a different argument). For me, the thing that looks suspicious is the repeated tr calls. If I see that, I assume someone is trying to be clever, where 'clever' here is a pejorative. If I were a maintainer and someone checked that in, I'd ask them to walk me through what it was doing, because there's almost always a better solution to avoid chaining things like that. The real problem here is that there wasn't another maintainer to look at this code being brought in. A critical piece of the stack relied on a single person who, in this case, was malicious.
duped 3y ago
The shell is generated, not written. There are mountains of generated shell configuration code out there due to the prevalence of autoconf, which relies on these M4 (a macro preprocessor) scripts to generate thousands of lines of unreadable shell script (which has to be portable). This is how a non negligible number of your tools are built.
semiquaver 3y ago
> inadvertently The whole point is that it’s intentionally obfuscated!
ekidd 3y ago
> i don’t have a better answer, but this convoluted mess of bash is a smell isn’t it? It's a very old smell, basically. The central problem is that back in the 80s and 90s, there were scads of different Unix-like systems, each with their own warts and missing features. And software authors wanted to minimize their build dependencies. So many communities standardized on automating builds using shell scripts, which worked everywhere. But shell scripts were a pain to write, so people would generate shell scripts using tools like the M4 macro preprocessor. And this is why many projects have a giant mass of opaque shell scripts, just in case someone wants to run the code on AIX or some broken ancient Unix. If you wanted to get rid of these impenetrable thickets of shell, you could: 1. Sharply limit the number of platforms you support. 2. You could standardize on much cleaner build tools. 3. You could build more key infrastructure in languages which don't require shell to build portably. But this would be a massive undertaking, and a ton of key C libraries are maintained by one or two unpaid volunteers. And dropping support for "Obscurnix-1997" tends to be a fairly controversial decision. So much of our key infrastructure remains surrounded by a morass of mysterious machine-generated shell scripts.

minimax 3y ago

> if this was found by accident, how many things still remain undiscovered. This, to me, is the most important question. There is no way Andres Freund just happened to find the _only_ backdoored popular open source project out there. There must be like a dozen of these things in the wild?

roflmaostc 3y ago
Maybe. But maybe there is not many (critical) ones out there. Otherwise, I believe we would encounter more often those kind of situations.
mistrial9 3y ago
"(B)urn (A)ll (S)atanic (H)elper-scripts" ? get the kindling?
- paulmd 3y ago
  thou shalt not make a machine in the likeness of a human mind
  - supersuryan 3y ago
    Human minds shine bright, but machines lend their light
    - slater 3y ago
      Human minds shine bright, and machines steal their light
whywhywhywhy 3y ago
It wasn’t found accidentally, he felt it. There’s a difference. Next one will definitely be less careless with adding time to it’s execution.

benlivengood 3y ago

Deterministic/repeatable builds can help with issues like this: once the binaries exist from checksummed code repository and are hashed, the tests can do whatever they want but if the final binaries change from the recorded hashes they shouldn't get packaged. This is in general a problem with traditional permission models. Pure capabilities would never leave binaries writable by anything other than the compiler/linker, shrinking the attack surface.

edflsafoiewq 3y ago
Running the tests does not modify the binary. The build script was modified to create a malicious .o file which was linked into the final binary by the linker as normal. Tests were only involved in that the malicious .o was hidden inside binary test files.
- paulmd 3y ago
  letting dist builds be linked against test resources is a design defect to begin with, and the fact that this is easy/trivial/widely-accepted is a general indication of the engineering culture problems around C. Nobody in Java word is linking against test resources in prod, and a sanely designed build system should in fact make this extremely difficult. That shit went away in the maven/gradle days - which is for a reason, ant is basically makefiles for Java, gradle/maven are a build system not a pile of scripts. And that transition happened 20 years ago! If you can’t even prevent a test resource being linked into a final build you are not serious. I don’t care about legacy whatever, that’s an obvious baseline metric for security culture / build engineering. Maybe not “prevent” but like, tooling should absolutely make it blindingly obvious that you’re violating best-practices by disabling scoping rules or including unusual source/resource roots, etc. C has never moved past the 1970 mindset of build being a pile of scripts with a superstructure bolted on. Just like Ant. Even the attempts to fix C’s build are just better ways to programmatically generate better bash scripts to keep you going off the rails.
  - malkia 3y ago
    Good point, bot sure if enforced there, but systems like bazel (buck?) and others have ways to mark build nodes as "testonl"

anthk 3y ago

Using LTS distros can shield you a bit. Slackware uses lzma (tar.xz) for it's packages I think, and beside of -current, the last stable release didn't have that issue. Also, if you want a step up on the freedom ladder, Hyperbola GNU neither had that issue. EDIT: Also, Slackware -current neither doesn't link sshd against xz, nor uses systemd.

deleted 3y ago

[deleted]

ds9soft 3y ago

The whole C land, including build tools, old unix utils, is a security mess waiting to be exploited, and it's going to be exploited. Just look how easy it's to break everything with a single dot. It's time people realize we can't bet the world's security on C. Please use Ada or Rust with modern tooling.

brenoguim 3y ago
If you add the dot to Rust it doesnt break?