12 ms·

Reflections on Distrusting xz

286 points by edward 3y ago

berkes 3y ago
Full on tinfoil hat here. But warranted and practical. I'm wondering what fallout we'll see from this backdoor in the coming weeks, months or years. Was the backdoor used on obscure build servers or obscure pieces of build infrastructure somewhere? Lying dormant for a moment in future to start injecting code into built packages maybe? Are distro's going to go full-on tinfoil-hat and lock down their distribution, halting progress for long time? Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress?
- skywhopper 3y ago
  IIRC Debian has wiped and is rebuilding all their build hosts, so yes. But while I understand what you mean, I would not call improving the security of a piece of software “halting progress”. Security improvements are progress, too. Plus, revisiting processes and assumptions can also give opportunities to improve efficiency elsewhere. Maintenance can be an opportunity if you approach it properly.
  - berkes 3y ago
    What I meant with "halting progress" is what commonly happens when a piece of software is "rewritten from scratch". Users (or clients or customers) see no improvements for years or weeks, while the business is burning money like mad. The main reason why I am firmly opposed to "rewrite from scratch" or "we'll need some weeks to refactor¹". Removing upstream dependencies and replacing them with other deps, with no-code, or with self-written code² is a task that takes long time during which stakeholders see no value added other than "we reduced the risk"; in case of e.g. SAAS that's not even risk these stakeholders are exposed to, so they then see "nothing improving". I'm certain a lot of managers, CTOs and developers suddenly realize that, wow, dependencies really are a liability. ¹ I am not against refactoring, just very much against refactoring as standalone, large task. Sometimes it's unavoidable because of poor/hard choices made in the past, but it's always a bad option. The good option would be "refactor on touch" - refactoring as part of our daily jobs of writing software. ² Too often do I see dependencies that are redicoulously simple. Left-pad being the posterchild. Or dependencies that bring everything and the kitchen-sink, but all we use is this one tiny bit that would've cost us less than 100 lines to write. Or dependencies to solve -- nothing really? Just that no-one took the time to go through it and remove it. And so forth and so on.
- pdimitar 3y ago
  It's impossible to insure against in any practical terms. The way forward is to invest heavily in a much more security-oriented kernel(s) and make sure that each program has the bare minimum to achieve what it offers as a value-add. The human aspect of vetting seems like an impossibly difficult game of whack-a-mole. Though realistically I doubt that the bad actors have infinite agents everywhere, this also has to be said. So maybe a "sweep" could eliminate 90% of them, though I'd be skeptical.
  - galangalalgol 3y ago
    Agreed, as a developer: minimize your dependencies while providing your core function. Don't grant dependencies permissions they don't need. Be granular about it. Austral lets you select what filesystem, network, etc. access each library gets. Also, in big organizations, risk assessment is more about making sure there is someone to point the finger at, than actual security. Treating libfubar as golden because it ships with something you paid another company money for makes sense in that light. But not from an actual security mindset.
    - adql 3y ago
      > Agreed, as a developer: minimize your dependencies while providing your core function. Don't grant dependencies permissions they don't need. Be granular about it. Austral lets you select what filesystem, network, etc. access each library gets. Still wouldn't help for this particular exploit.
      - pdimitar 3y ago
        Since I didn't keep as current as I wanted to be (work and life happen a lot lately), what could have prevented it?
      - galangalalgol 3y ago
        If systemd could deny liblzma any syacall or filesystem access, that would have prevented it. It is only used to compress a data stream, it only needs read access from one buffer, and write access to another. I realize there is no current mechanism for these granular permissions, that is what I was proposing be addressed.
- gammalost 3y ago
  You can't really remove dependencies in open source. It is so intertwined at this point that doing it would be too expensive for most companies. I think the solution is to containerize, containerize and then containerize some more times and make it all with zero trust in mind.
  - funcDropShadow 3y ago
    Docker containers are not really a security measure.
    - gammalost 3y ago
      It is a security measure. Sure it doesn't secure anything in the container itself. But it secures the container from other containers. Code can (as proven) not be trusted, but the area of effect can be reduced.
      - 65a 3y ago
        Only with additional hardening between the container and the kernel and hardware itself.
  - rwmj 3y ago
    Containerizing is entirely the worst response here. Containers, as deployed in the real world, are basically massive binary blobs of completely uncertain origin, usually hard to reproduce, that easily permit the addition of unaudited invisible changes. (Yes yes, I know there are some systems which try to mitigate this, but I say as deployed in the real world.)
    - adql 3y ago
      Pretty much; updating say libssl in a "traditional" system running app, or maybe 2-3 dependent apps fixes the bug. Put all of them in containers and now every single one needs to be rebuilt with the dep fixed and instead of having one team (ops) responsible, you now need to coordinate half of the company to do so. It's not impossible but in general much more complex, despise containers promising "simpler" operations. ...that being said I don't miss playing whack-a-mole game with developers that do not know what their apps need to be deployed on production and for some retarded reason tested their app on unstable ubuntu while all of the servers run some flavour of stable linux with a bit older libs...
- constantcrying 3y ago
  >Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress? How is that even a possibility? xz was very useful software, which can only exist if people with significant knowledge put effort into it. Not every OSS project has the ability or resources to duplicate that. The same goes for many, many other dependencies. I believe that there is essentially nothing you can do to prevent these attacks with the current software creation model. The problem here is that it is relatively simple for a committed actor to make significant contributions to a publicly developed project, but this is also the greatest asset of that development model. It is extremely hard to judge the motivation of such an individual, for most benign contributors it is interest in the project, which they project onto their co contributors.
  - TeMPOraL 3y ago
    Agreed. More than that, there's not much of a way to preven these kinds of attacks, period, whether in software or otherwise, if the perpetrator is some intelligence agency or such. For threats lesser than a black op, the standard way of mitigating supply chain attacks in the civilized world is through contracts, courts, and law enforcement. I could, in theory, get a job at a local food manufacturer, and over the course of year or two, reach the point I could start adding poison to the products. But you're relatively confident that this won't happen, because should it ever did, the manufacturer will be smeared and sued and they'll be very quick to find me and hand me over to the police. That's how it works for pretty much everything; that's how trust is established at scale. Two key components of that: having responsibility over quality/fitness for use of your product, and being able to pass the blame up to your suppliers, should you be the victim too. In other words: warranty and being an easily identifiable legal entity. Exactly the two components that Open Source development does away with. Software made by a random mix of potentially pseudoanonymous people, offered with zero warranties. This is, of course, also the reason OSS is so successful. Rapid, unstructured evolution. Can't have one without the other. Or in short: the only way I see to properly mitigate these kinds of threats is to ditch OSS and make all software commercial again (and legally force vendors to stop with the "no warranty" clause in licensing). That doesn't seem like a worthwhile trade-off to me, though.
- ProblemFactory 3y ago
  > Was the backdoor used on obscure build servers or obscure pieces of build infrastructure somewhere? And developer machines. The backdoor was live for ~1 month on testing releases of Debian and Fedora, which are likely to be used by developers. Their computers can be scraped for passwords, access keys and API credentials for the next attack.
- coretx 3y ago
  Not fallout but increased vigilance is the expected most significant outcome. Some 5+ years ago or so I listened to a Debian guy his talk about reproducible builds and security, he was stressing the audience to be aware of just happened in a very detailed manner. One of the details he mentioned was glowies having moved their focal point to individual developers and their tooling & build systems. At least some people who matter have been working on these threats for many years already, maybe more people will start to listen to them; in such case this entire debacle could have a net positive effect on the long run.
- Macha 3y ago
  > Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress? We've been here before, with e.g. event-stream and colors on npm. So I don't think it will change much. Except maybe people will stop blaming it on JS devs being script kiddies in their mind, when they realise that even the traditional world of C codebases and distro packages are not immune.
throwaway63467 3y ago
Seems like a sensible thing to do, assuming this is a state-level threat actor there’s really no easy way to prove that their contributions are free of back doors. Seems not worthwhile risking the security of a large part of the Internet over a few thousand lines of code.
- afc 3y ago
  But why would the entire behind this submit all their attacks through the same single identity? Removing all this code could just be removing 1% of their harmful code. How do you deal with the rest? How do you discover the other identities?
  - rwmj 3y ago
    You start with what you know about, and you investigate other projects carefully at the same time. There's no easy answer here, you do what you can.
Macha 3y ago
Yeah, it's left me a little disappointed in Arch in particular that they didn't follow the lead of Debian and Fedora and revert to a much older version, instead just building 5.6.1 from the git repo and basically defended it as "the hacked build script checked for dpkg/rpm anyway".
- antoinealb 3y ago
  Arch Linux is not vulnerable to this specific attack, which requires sshd to be linked to liblzma. This link is provided by out-of-sshd patches, that Arch does not apply to their build.
  - skywhopper 3y ago
    But as this article points out, liblzma is used in other crucial processes, and is generally trusted, often probably being run as root. The known bad actor contributed lots of code to xz that isn’t involved in the SSH backdoor. To assume it’s all innocuous would be truly foolish.
  - cowsandmilk 3y ago
    The point here is there is uncertainty in all commits by Jia Tan, Arch’s focus is on this specific hack, but are there other vulnerabilities in the hundreds of commits to the git repo from the same author?
- diggan 3y ago
  Is this what you're referring to? > Regarding sshd authentication bypass/code execution > Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command: > However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist. https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore... I'm not finding anything from Arch mentioning dpkg/rpm, the linked article above is the latest article about the xz compromise from the Arch homepage.
skywhopper 3y ago
my gut instinct is that xz needs to be rolled back to its pre-attack state, but obviously that would probably also reintroduce some bugs and likely break some things. Still, very curious to see some analysis on the impact of doing so, because this article points out, xz is in a critical path for lots of system level processes.
- fullstop 3y ago
  There were symbol changes in recent releases, and things like apt link to liblzma. If liblzma were downgraded without also updating apt at the same time you could be left with a non-functioning apt.
fsflover 3y ago
> What if xz contains a hidden buffer overflow or other vulnerability, that can be exploited by the xz file it's decompressing? If you generalize this problem further, to all packages, then the only reliable solution is security through compartmentalization. On Qubes OS, any file I open, including .jpg and .avi, can't have the access to my private data or attack the admin account for the whole computer. This is ensured by hardware-assisted virtualization.
- JoshTriplett 3y ago
  > the only reliable solution is security through compartmentalization I hope we get there eventually. Not just for standalone processes, but for individual libraries. A decompression library could run inside a WebAssembly sandbox, with the compressed file as input, the uncompressed file as output, and no other capabilities.
  - funcDropShadow 3y ago
    What does this have to do with WebAssembly? That is another runtime that adds complexity. Apple has been sandboxing codecs for a long time. They run in a sandboxed process that is only communicating through stdin and stdout or something similar, if I remember correctly. You can ran native code directly. Adding a runtime with a JIT-compiler makes it harder to understand what is going on.
    - 01HNNWZ0MV43FF 3y ago
      There's AOT wasm too. Firefox uses it to sandbox some stuff. https://hacks.mozilla.org/2021/12/webassembly-and-back-again-fine-grained-sandboxing-in-firefox-95/ https://hacks.mozilla.org/2021/12/webassembly-and-back-again...
    - JoshTriplett 3y ago
      WebAssembly can be run in-process rather than requiring a process switch, and it can be easier to port library code to run inside a WebAssembly sandbox than a completely separate process. Also, sandbox mechanisms for separate processes are not always as robust, since they have to give access to any direct syscalls the process makes, whereas WebAssembly completely insulates a library from any native surface area.
irdc 3y ago
One thing that comes to mind is that “Jia Tan” might be more accurately seen as a “sleeper” of some sort: a foot soldier who infiltrates a juicy open source project and waits for further instructions; backdooring sshd might not have been part of the original plan. Which raises the concerning question of how much more sleeper maintainers there are.
- berkes 3y ago
  I once got a (probably scam) offer for adding a cryptominer to a library that I maintained at that time. And a more serious offer to add trackers to a popular >1M installs app. Both cases I obviously ignored it. But it made me aware of a nasty attack vector: someone who's thanklessly building a wordpress-plugin, pip, npm, or whatever software, thanklessly dealing with issues, PRs, support, maintainence, often for no pay, suddenly gets offered three figure sums to add a few lines of "affiliate stuff" or such. There are many places in the world, or people in situations where this amount of money really makes a compelling case.
  - irdc 3y ago
    “Given enough underfunded maintainers, all security is shallow.”[0] 0. https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law
    - lenerdenator 3y ago
      I wouldn't say "funding" is necessarily the problem. Most maintainers do it because they like doing it. Their main limiting factor is time. I can drop a million dollars an hour into a maintainer's lap; that doesn't mean they can dedicate every waking moment to a project. They still have human needs that money can't buy like sleep, family obligations, and health concerns. And that's making the assumption that the maintainer uses that million/hr to quit their job. No, the problem is a lack of trustworthy candidates for maintainership and a lack of time. There are components of a GNU userland that are now too complex for a single human to both maintain and enhance at the same time. We now need to target multiple distros (really, more than are necessary, strictly speaking) and ISAs. Most are written in systems programming languages like C that are more complex than the average software engineer in 2024 works with. We need consolidation, simplification, maintainer redundancy, and a trust/governance framework for packages.
fsniper 3y ago
It's like security 101. If a system has been infiltrated, you can't trust any part of it. So it's better to discard any part that has been reached or possibly affected. Perhaps it's the correct action to distrust xz/lzma or any source code this team has control over and switch to alternatives. If there are no alternatives, to start ones.
- lrem 3y ago
  But this suggests reimplementing xz/lzma. Which would cost money. Hence, won't be done.
  - frankjr 3y ago
    But there are alternatives, most notably zstd.
    - WesolyKubeczek 3y ago
      It's a different algorithm made for a different purpose.
    - fsniper 3y ago
      a half-arsed search resulted in this half-baked rust library: https://github.com/gendx/lzma-rs https://github.com/gendx/lzma-rs
    - kzrdude 3y ago
      sadly, the zstd cli tool links to lzma right now (as installed by some distros) :/
  - dspillett 3y ago
    > But this suggests reimplementing xz/lzma. If there is a known good copy of the repo from before the attacker had sufficient access to alter history, then that is an acceptable starting point. From there you look at each update since and assess what they do to decide if you want to keep (as they are valid improvements/fixes) or discard them. If some are discarded, then later ones that are valid may need further work to integrate them into the now changed codebase. Similar to Debian assessing upstream security patches to the latest version to possibly back-port them to the version they have in stable, when there is significant disparity (due to a project being much faster moving than Debian:Stable). As xz/xzutils is a relatively stable package, with very few recent changes, this should be quite practical. A full rewrite shouldn't be needed at all here.
logro 3y ago
Trusting anything from that actor is full on ignorant, let alone "a new decoder". It's insane.
- iso8859-1 3y ago
  Trusting people in general is inadvisable. I haven't trusted anyone for years and I am richer than ever.
  - codetrotter 3y ago
    You probably still rely on trusting others a lot more than you realize. If you really, really, didn’t rely on trusting anyone, I don’t even see how it would be possible to exist on earth.
  - logro 3y ago
    Those two topic don't have much in common, trusting a state level hacker actor vs. trusting people in general.
  - frankjr 3y ago
    You're trusting millions of people just to be able to write this comment.
rwmj 3y ago
I think Joey's right that we should all go back to the "pre-Jia-Tan" xz, and I've raised this with Red Hat too. It's actually not a big deal as xz and liblzma is relatively stable and the version from 2 years ago is fine, although I understand that Debian's dpkg uses some new API(s) from liblzma which makes this a problem albeit a minor one. (Unfortunately the Debian bug report that Joey filed got derailed with a lot of useless comments early on.)
- iso8859-1 3y ago
  How do you know what 'pre' means given that pseudo-anonymous identities are free and Tan is already suspected of having some (e.g. Hans Jansen and Jigar Kumar: https://research.swtch.com/xz-timeline https://research.swtch.com/xz-timeline)
  - rwmj 3y ago
    I mean we go back before all possible sockpuppets. We do have a reasonably good idea of when the attempt started.
    - glitchcrab 3y ago
      The point that the person you replied to is trying to make is how do you know when the repo is clean? How can you ever be sure that someone hasn't introduced a backdoor at some point? It's bigger than just what has been discovered.
      - rwmj 3y ago
        How do you know anything has not been compromised? You go and look at the commits and the code. It's hard work with no easy answers despite what many think.
      - rrr_oh_man 3y ago
        By not letting the perfect be the enemy of the good-enough-for-now
ducktective 3y ago
I'm here wondering why big tech companies that have Too Much to Lose didn't already massively fund a project that freaking sshd depended on it (through systemd). Like how does it hurt Google to assign 100 people to review and investigate commits of some project as basic and fundamental as a compression tool
- junon 3y ago
  sshd didn't depend on it, to be fair. Not officially at least.
- WesolyKubeczek 3y ago
  Poke at any large company at all, and you'll find that their in-house critical fundamental infrastructure thing is chronically underfunded, understaffed, bug-ridden, everyone is worried but no budget is ever approved.
  - ghaff 3y ago
    Within a given part of an organization, just about everyone thinks they're underresourced and understaffed.
- 01HNNWZ0MV43FF 3y ago
  Very shrimple https://en.wikipedia.org/wiki/Tragedy_of_the_commons https://en.wikipedia.org/wiki/Tragedy_of_the_commons
- dlandau 3y ago
  That's exactly what they do, though: https://cloud.google.com/assured-open-source-software/docs https://cloud.google.com/assured-open-source-software/docs
  - arp242 3y ago
    Google also does code reviews for some commonly used projects (or maybe that's part of the same thing? I don't know). I went through that last year with one of my Go libraries. The idea is good, but the entire process is so bureaucracy-heavy and time-consuming that I found it both frustrating and entertaining in equal parts; like something out of Brazil (the film, not the country). So many emails, so many video meetings, so many people involved, so much talking. And all for looking at a 4,500 line Go library. "Here's the code; just clone and look at it, and let me know if you find something"... It's not like you need my permission to do any of this *shrug*.
rfoo 3y ago
Nit-picking but, eh, png does not use lzma at all. > PNG compression method 0 (the only compression method presently defined for PNG) specifies deflate/inflate compression with a sliding window of at most 32768 bytes. Deflate compression is an LZ77 derivative used in zip, gzip, pkzip, and related programs.
- jwilk 3y ago
  I don't think joeyh wanted to imply that PNG uses liblzma. PNG is just a convenient place to put opaque binary stuff that'd trigger an xz compression bug.
  - rfoo 3y ago
    I still don't understand how would that work. The post said: > Let's say they want to target gcc. Well, gcc contains a lot of documentation, which includes png images. So they spend a while getting accepted as a documentation contributor on that project, and get added to it a png file that is specially constructed, it has additional binary data appended that exploits the buffer overflow. And instructs xz to modify the source code that comes later when decompressing gcc.tar.xz. It says "when decompressing", and I would imagine that such a bug needs specifically constructed lzma stream to trigger. If you want to do it by changing a source file (a png here) you need to make "second-order" bugs: i.e. the compressor needs to output a broken lzma stream which when later decompressed would exploit (not simply cause) a memory corruption bug. This is too brittle [1] and are very likely to be detected. [1] Disclaimer: I'm not an expert in writing backdoors. I consider myself reasonably competent for writing exploits, and I've written deliberately buggy programs (for CTFs) before.
    - deleted 3y ago
      [deleted]
    - j16sdiz 3y ago
      Consider this backdoor: - if the decompressed stream contain a magic keyword, run the rest of the file as x86-64 binary. Now you just need any opaque binary file to host the payload. A PNG works fine, because most decoder don't care extra bytes at the end.
moomin 3y ago
> Hopefully, Lasse Collin will consider these possibilities and address them in his response to the attack. Here's the thing: Lasse Collin was overloaded back in 2021. I've no particular reason to believe that isn't still the case. He needs help. Dealing with this solo is an incredible amount of work. Also, he needs help from a verifiably trustworthy source, verifiable in a way that doesn't require a lot of effort. In practice, that almost certainly means help from a major open source company. I seriously doubt that's going to happen, because the people who really need to learn this lesson won't, because it's probably not in their financial interest to realize that supply chain problems start with them doing things on the cheap. While we keep on running the world's infrastructure like XKCD 2347 every so often everything's going to topple over.
- j16sdiz 3y ago
  ... but `xz` is pretty much feature complete to me. Lasse Collin was doing bug-fix-only release just fine.
  - adql 3y ago
    The start of the attack was few fake accounts trying to shame the maintainer for not "developing" it constantly and so give maintainer rights to someone else. And there wasn't really anyone to say "nope, it's fine, fuck off"
    - FdbkHb 3y ago
      > And there wasn't really anyone to say "nope, it's fine, fuck off" That's because for people for whom the project is doing fine and they haven't experienced any bug, why would they go to the mailing list, forum or whatever other form of communication channel the project has? One has to understand and keep in mind that places that can gather feedback will invariably attract more of the negative kind than the positive kind because people who are happy are not motivated to say they're happy. Those people wouldn't even know people were complaining about xz. There's thousands of libraries/independent software projects installed on any computer. No one has the time to check the place of all those software projects and go there just to say "hey, I'm happy, no need to change anything, thanks?", right? People who are discontent with something on the other hand are sure to be vocal about it. But just because they're the most vocal doesn't mean they are the majority of your users.
Maledictus 3y ago
Is XZ embedded affected by any of this?
- squarefoot 3y ago
  Good question. Embedded systems are harder, sometimes impossible, to upgrade, and there are chances a backdoor in a small board inside an appliance that doesn't offer easy physical access could take years before it is found and removed.
KyleSanderson 3y ago
There seems to be a fundamental misunderstanding with a lot of these writeups. Are they 100% sure history was not rewritten at any point? Going back in time on the repo prior to listed involvement doesn't do anything as the attacker had full control. Starting from the last signed release prior to their involvement is the only way to actually move this forward (history may be fully lost at this point), the rest is posturing.
- AtNightWeCode 3y ago
  People think git is immutable. It is not.
  - fl7305 3y ago
    Can you elaborate? Are you thinking of intentional SHA-1 has collisions? Would that work in practice?
    - AtNightWeCode 3y ago
      The history. Every time something like this attack happens people think they can read the complete git history in the repo.
      - fl7305 3y ago
        If some commits are signed by people you trust, can the chain before that still be compromised?
  - Lichtso 3y ago
    Yes and no. A local GIT repo can be changed (including its history) however you please. But once you have shared it with others you can't take that back. If you try to, then others will notice that the hashes mismatch and that their HEAD diffs uncleanly. I know the term is infamous here, but GIT is essentially a blockchain. Each commit has a hash, which is based on the hashes of previous commits, forming a linked list (+ some DAG branching).
meinersbur 3y ago
The greater concern should be how many other sleeper contributors are out there. Anonymous contributions are accepted every day, and we know of cases with malicious intent such by "James Bond" (https://lore.kernel.org/lkml/20200809221453.10235-1-jameslouisebond@gmail.com/ https://lore.kernel.org/lkml/20200809221453.10235-1-jameslou...). I am not specifically worried about other contributions by "Jia Tan", those are being extensively looked at right now. They and other sleepers may just as well have contributed to any project with a different name and therefore "Jia Tan" does not pose more danger than any other contribution whose submitter cannot be held responsible.
- rkta 3y ago
  What's malicious about that patch? From reading the thread it looks like an attempt to fix a FP from some tooling.
  - meinersbur 3y ago
    One of the patches that the University of Minnesota was banned for from contributing to the Linux kernel. They were trying to introduce a use-after-free (Fig. 9 in their paper). https://news.ycombinator.com/item?id=26887670 https://news.ycombinator.com/item?id=26887670
    - meinersbur 3y ago
      I just had to think about how ironic it would be if "Jia Tan" turned out to be a Post-Doc from the University of Minnesota continuing that research on hypocrite commits.
lispm 3y ago
Why not get rid of xz completely? Hoe about using a simpler piece of software, which could be maintained by more people?
- deleted 3y ago
  [deleted]
- adql 3y ago
  > Why not get rid of xz completely? ... you mean compression package used by most big distros to make their packages? Do you really need to ask > Hoe about using a simpler piece of software, which could be maintained by more people? It's not a complex piece of software. Lib itself is ~15k lines of code. It does it's job well and it needs little work. It didn't had any outstanding bugs lingering unfixed for years Complexity have nothing to do with the problem, it's just... uninteresting enough that there is no reason for contribute.
  - imjonse 3y ago
    Complexity is one of the root causes of the problem. The script was added via messy autoconf scripts inscrutable to most people. That is decades old tech which has alternatives.
  - simoncion 3y ago
    > ... you mean compression package used by most big distros to make their packages? Do you really need to ask Because of the existence of gzip, zlib, bzip2, and many others, it's trivial to drop xz. So yeah, why not get rid of xz completely?
  - lispm 3y ago
    I've seen for example here a lot of issues raised: https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
deleted 3y ago
[deleted]
loftsy 3y ago
I took a look at the diff linked in the article with code that "we are all running". The top of the diff certainly looks interesting. They remove the bounds check in dict_put() and add a safe version dict_put_safe(). This kind of change is difficult to make without mistakes because it silently changes the assumptions made when code calling dict_put() was originally written. ALL call sites would need to be audited to ensure they are not overflowing the dictionary size. The diff I am referring to is here: https://git.tukaani.org/?p=xz.git;a=commitdiff;h=de5c5e417645ad8906ef914bc059d08c1462fc29 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=de5c5e41764...
- logro 3y ago
  This is 100% malicious or novice coder. And we surely know it's not the latter. If you need an unsafe call, you add a dict_put_unsafe(). That again should of course be rejected in a code review.
- kmfpl 3y ago
  I agree, this looks extremely sketchy. Especially because the code is just writing a fully controlled byte in the buffer and incrementing its index. This would give you a controlled relative write primitive if you can repeatedly call this function in a loop and going OOB.
- liendolucas 3y ago
  I think at this point is clear that everybody has to assume that XZ is completely rotten and can no longer be trusted. Is it XZ easy to replace with some other compression tool? Or has it been so widely adopted that is going to take huge effort moving out of it?
  - kzrdude 3y ago
    Huge effort, because it is the default .deb compressor in Debian for example
    - rthnbgrredf 3y ago
      Arch Linux has replaced it with zstd in 2020 already. It's doable for the next major release of Debian.
deleted 3y ago
[deleted]
rpigab 3y ago
What if over 80% of all open source projects are secretly sleeper agents for various malicious actors, states, terrorists and whatnot, and they pretend to give use precious software updates for free so they can attack later? What if proprietary software is running the same way, except they don't even give you free updates and you can't audit the source code and have to trust them when they push updates? What if your mother gave birth to you just so she can slap you in the face when you're 30? Yeah, we can go very far, but in this moment, xz is under so much scrutiny that in 2-4 weeks, I'd trust it with my life unless the big orgs looking at it issue more reverts (hence, the delay). So if there are issues, they're everywhere else.
codezero 3y ago
Consider “Jia Tan” started working on xz because they already found a critical vulnerability and wanted to maintain it, or more tin foil, they burned xz to get upstreams to use another compression that is also already backdoored. When dealing with state actors there’s really no limit to how complex the situation can be.
- alt227 3y ago
  This is something I also wondered but havent seen discussed anywhere. This could all be a smokescreen to get distros to switch to the next best compression library which already contains malicious code. Hopefully maintainers of any upstream compression libraries are all looking hard at their code bases right now.
nomilk 3y ago
It seems detecting holes in jia’s code could be extremely difficult. Given the stakes, as a precaution, would it be viable to simply wipe and rewrite (from scratch) the last ~2 years of commits to xz?
imjonse 3y ago
I am wondering if the person who sent the patch to disable systemd reliance on lzma shortly after the release of the backdoored xz knew about the plan. Maybe an agent of a competing entity?
cletus 3y ago
The US spends $30-60 billion a year on agriculture and subisides [1]. This is controversial for all the obvious reasons. Without subsidies we'd waste less food but overproduction of food is an intentional objective of these programs. Why? Because if there's a major drought or crops are lost to ice or snow or flooding, Americans won't starve. It's why we have things like the US government having a reserve of over a billion pounds of cheese [2]. People not starving is a national security interest. It's getting to the point where the software we rely on is also a national security interest. The US government should be paying to maintain and improve this software. Security risks in Linux and core packages threaten to shut down key infrastructure. Paying developers to maintain open source projects that are actually used could be an incredibly effective use of tax dollars. [1]: https://usafacts.org/topics/agriculture/#581290c9-a960-49aa-a4f9-ab52dec5bbdd https://usafacts.org/topics/agriculture/#581290c9-a960-49aa-... [2]: https://www.deseret.com/2022/2/14/22933326/1-4-billion-pounds-of-cheese-stored-in-a-cave-underneath-springfield-missouri-jimmy-carter-reagan/ https://www.deseret.com/2022/2/14/22933326/1-4-billion-pound...
- chjj 3y ago
  I'm not sure why people keep misidentifying the problem as "lack of funding". Lasse Collin was doing fine as a maintainer up until Jia Tan showed up. He was psy-op'd into believing there was a crowd of angry people eagerly awaiting a new release when there wasn't. No real person was unhappy with the way he'd been maintaining xz.
  - Cthulhu_ 3y ago
    Funding aside, single individuals being responsible for software is not a good thing, see bus factor.
    - rebolek 3y ago
      Fortunately there was Jia Tan to help him. /s
    - Gormo 3y ago
      In fact, there was no issue with Lasse Collin maintaining xz as a single individual, and creating a false impression to the contrary was the primary tactic used by the antagonist to gain access to the project.
bilekas 3y ago
> It feels good to not need to worry about dpkg and tar. I only plan to maintain this fork minimally, eg security fixes. This is exactly the problem in the first place, lack of support for maintainers. OP themselves say "I will only minimally maintain this fork". Okay, but it's so easy in hindsight to criticize what has happened. > Hopefully Lasse Collin will consider these possibilities and address them in his response to the attack. I can't even imagine how he's feeling these days.
- dmitrygr 3y ago
  > I can't even imagine how he's feeling these days. None of this is his problem or fault. I see no reason he should feel anything about this. Keeping everyone safe was never his job. He wrote code and gave it away for free. That should be enough.
  - bilekas 3y ago
    > I see no reason he should feel anything about this. Absolutely agree, but from the sounds of the emails at least, he was going through a bad time then, and nobody feels good when they realise they were taken advantage of.
  - david_draco 3y ago
    anyone who has ever been pickpocketed or robbed or worse will know reason and feelings are different things
sergioisidoro 3y ago
It's a bit ironic that after a trust attack this person ends the article sayin > I do have a xz-unscathed fork which I've carefully constructed to avoid all "Jia Tan" involved commits. He may be fully legitimate, and perhaps a famous person in OSS (which I was unfamiliar with), but still ironic :)
WesolyKubeczek 3y ago
From reading comments here alone, I can predict the following (unhealthy) effects on software at large: first, huge scaremongering like this writeup. It all hinges on a notion that who knows who is writing walls of who knows what code, and the code and its purpose is absolutely unpenetrable, inscrutable by anyone else at all. It's not true, as evidenced by analyses of this hack alone and by reverse engineering community at large. Of course, it requires doing what Anders has done, meaning, rolling up your sleeves and actually reading the source and trying to understand it, and not just hope someone else will do it. Whatever one person tangles, another can always untangle. Then, there is going to be a witch hunt and people jumping on any innocent change with pitchforks (already happening, satanic panic over ibus is in the nearby discussion). The code review theater will be in full swing. Previously, people were conserving their brain energy (or masking their incompetence) by skimming walls of changes and stamping LGTM on them based on how well-formed they had been, how long they had known the author, and how the builds were not broken. Now it will become extremely hard to get any changes done at all, because the new mental shortcuts will be: too long, didn't approve; explanations too complex; just plain Reviewer Says No; and so on. Any sloppy PR denial will follow by patting oneself on the back, look at me, I just have thwarted a KGB agent. Some people thinking they know a lot without basing such assertions in reality will try to become overnight „wonder experts in security”, barking on every shell script they don't understand, or every piece of generated text, like generated Makefiles. Vulnerabilities akin to CVE 2022-3786 and CVE 2022-3602 — which got introduced by writing a whole new email address parser from scratch (which IIRC also got checked in as a whole wall of code at once, and I read someone blaming exactly this as the culprit) — will lead to questioning by police at least once. Automated codebase scans with bogus reports like Daniel Stenberg wrote about in [0] will be more abound. Everyone will just jump on any "unsafe" function call without actually understanding its context, and will keep pestering authors to "make a fix" because potentially something (gasp!) may happen. Later everyone will get tired of this charade, and everything will come back to "normal", probably with added processes and red tape to make FOSS maintainership even more of a liability than it is today. Nothing will be done at all to make it less of a burden. [0] https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...
deleted 3y ago
[deleted]
lakomen 3y ago
I'm disappointed, to put it very mildly how archlinux handled the matter. They still use version 5.6.1 and assume that switching from github to the one hosted by Lasse fixes the issue. They say "our sshd isn't compromised". But what the author of this article wrote, who knows what else might be affected. There's a forum post on the archlinux forum, which was closed by ewaller, an administrator account, with a weird reason, that this thread was only to inform people, but when people started calling out the malpractice of the archlinux maintainers the thread got locked. To me, this is very suspicious.
geggo98 3y ago
That’s a good start. In the long run probably three things are necessary: 1) wiring critical software in a language that protects better against such exploits. Might be Rust, Go, perhaps also C# and Nim. 2) Making reproducible builds the norm, that start from the original source code repositories (e.g., based on a Git hash) 3) making maintainers more resilient against social attacks. This means more appreciation, less demands, and zero tolerance against abuse. If the maintainer can be pressured, I am at risk. The last one is probably the most difficult.
JetSpiegel 3y ago
Why not add another layer to the tinfoil hat? How do we know "Jia Tan" is not a Facebook op to "nudge" people to switch to Zstandard?

286 points by edward 3y ago

Full on tinfoil hat here. But warranted and practical. I'm wondering what fallout we'll see from this backdoor in the coming weeks, months or years. Was the backdoor used on obscure build servers or obscure pieces of build infrastructure somewhere? Lying dormant for a moment in future to start injecting code into built packages maybe? Are distro's going to go full-on tinfoil-hat and lock down their distribution, halting progress for long time? Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress?

skywhopper 3y ago
IIRC Debian has wiped and is rebuilding all their build hosts, so yes. But while I understand what you mean, I would not call improving the security of a piece of software “halting progress”. Security improvements are progress, too. Plus, revisiting processes and assumptions can also give opportunities to improve efficiency elsewhere. Maintenance can be an opportunity if you approach it properly.
- berkes 3y ago
  What I meant with "halting progress" is what commonly happens when a piece of software is "rewritten from scratch". Users (or clients or customers) see no improvements for years or weeks, while the business is burning money like mad. The main reason why I am firmly opposed to "rewrite from scratch" or "we'll need some weeks to refactor¹". Removing upstream dependencies and replacing them with other deps, with no-code, or with self-written code² is a task that takes long time during which stakeholders see no value added other than "we reduced the risk"; in case of e.g. SAAS that's not even risk these stakeholders are exposed to, so they then see "nothing improving". I'm certain a lot of managers, CTOs and developers suddenly realize that, wow, dependencies really are a liability. ¹ I am not against refactoring, just very much against refactoring as standalone, large task. Sometimes it's unavoidable because of poor/hard choices made in the past, but it's always a bad option. The good option would be "refactor on touch" - refactoring as part of our daily jobs of writing software. ² Too often do I see dependencies that are redicoulously simple. Left-pad being the posterchild. Or dependencies that bring everything and the kitchen-sink, but all we use is this one tiny bit that would've cost us less than 100 lines to write. Or dependencies to solve -- nothing really? Just that no-one took the time to go through it and remove it. And so forth and so on.
pdimitar 3y ago
It's impossible to insure against in any practical terms. The way forward is to invest heavily in a much more security-oriented kernel(s) and make sure that each program has the bare minimum to achieve what it offers as a value-add. The human aspect of vetting seems like an impossibly difficult game of whack-a-mole. Though realistically I doubt that the bad actors have infinite agents everywhere, this also has to be said. So maybe a "sweep" could eliminate 90% of them, though I'd be skeptical.
- galangalalgol 3y ago
  Agreed, as a developer: minimize your dependencies while providing your core function. Don't grant dependencies permissions they don't need. Be granular about it. Austral lets you select what filesystem, network, etc. access each library gets. Also, in big organizations, risk assessment is more about making sure there is someone to point the finger at, than actual security. Treating libfubar as golden because it ships with something you paid another company money for makes sense in that light. But not from an actual security mindset.
  - adql 3y ago
    > Agreed, as a developer: minimize your dependencies while providing your core function. Don't grant dependencies permissions they don't need. Be granular about it. Austral lets you select what filesystem, network, etc. access each library gets. Still wouldn't help for this particular exploit.
    - pdimitar 3y ago
      Since I didn't keep as current as I wanted to be (work and life happen a lot lately), what could have prevented it?
    - galangalalgol 3y ago
      If systemd could deny liblzma any syacall or filesystem access, that would have prevented it. It is only used to compress a data stream, it only needs read access from one buffer, and write access to another. I realize there is no current mechanism for these granular permissions, that is what I was proposing be addressed.
gammalost 3y ago
You can't really remove dependencies in open source. It is so intertwined at this point that doing it would be too expensive for most companies. I think the solution is to containerize, containerize and then containerize some more times and make it all with zero trust in mind.
- funcDropShadow 3y ago
  Docker containers are not really a security measure.
  - gammalost 3y ago
    It is a security measure. Sure it doesn't secure anything in the container itself. But it secures the container from other containers. Code can (as proven) not be trusted, but the area of effect can be reduced.
    - 65a 3y ago
      Only with additional hardening between the container and the kernel and hardware itself.
- rwmj 3y ago
  Containerizing is entirely the worst response here. Containers, as deployed in the real world, are basically massive binary blobs of completely uncertain origin, usually hard to reproduce, that easily permit the addition of unaudited invisible changes. (Yes yes, I know there are some systems which try to mitigate this, but I say as deployed in the real world.)
  - adql 3y ago
    Pretty much; updating say libssl in a "traditional" system running app, or maybe 2-3 dependent apps fixes the bug. Put all of them in containers and now every single one needs to be rebuilt with the dep fixed and instead of having one team (ops) responsible, you now need to coordinate half of the company to do so. It's not impossible but in general much more complex, despise containers promising "simpler" operations. ...that being said I don't miss playing whack-a-mole game with developers that do not know what their apps need to be deployed on production and for some retarded reason tested their app on unstable ubuntu while all of the servers run some flavour of stable linux with a bit older libs...
constantcrying 3y ago
>Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress? How is that even a possibility? xz was very useful software, which can only exist if people with significant knowledge put effort into it. Not every OSS project has the ability or resources to duplicate that. The same goes for many, many other dependencies. I believe that there is essentially nothing you can do to prevent these attacks with the current software creation model. The problem here is that it is relatively simple for a committed actor to make significant contributions to a publicly developed project, but this is also the greatest asset of that development model. It is extremely hard to judge the motivation of such an individual, for most benign contributors it is interest in the project, which they project onto their co contributors.
- TeMPOraL 3y ago
  Agreed. More than that, there's not much of a way to preven these kinds of attacks, period, whether in software or otherwise, if the perpetrator is some intelligence agency or such. For threats lesser than a black op, the standard way of mitigating supply chain attacks in the civilized world is through contracts, courts, and law enforcement. I could, in theory, get a job at a local food manufacturer, and over the course of year or two, reach the point I could start adding poison to the products. But you're relatively confident that this won't happen, because should it ever did, the manufacturer will be smeared and sued and they'll be very quick to find me and hand me over to the police. That's how it works for pretty much everything; that's how trust is established at scale. Two key components of that: having responsibility over quality/fitness for use of your product, and being able to pass the blame up to your suppliers, should you be the victim too. In other words: warranty and being an easily identifiable legal entity. Exactly the two components that Open Source development does away with. Software made by a random mix of potentially pseudoanonymous people, offered with zero warranties. This is, of course, also the reason OSS is so successful. Rapid, unstructured evolution. Can't have one without the other. Or in short: the only way I see to properly mitigate these kinds of threats is to ditch OSS and make all software commercial again (and legally force vendors to stop with the "no warranty" clause in licensing). That doesn't seem like a worthwhile trade-off to me, though.
ProblemFactory 3y ago
> Was the backdoor used on obscure build servers or obscure pieces of build infrastructure somewhere? And developer machines. The backdoor was live for ~1 month on testing releases of Debian and Fedora, which are likely to be used by developers. Their computers can be scraped for passwords, access keys and API credentials for the next attack.
coretx 3y ago
Not fallout but increased vigilance is the expected most significant outcome. Some 5+ years ago or so I listened to a Debian guy his talk about reproducible builds and security, he was stressing the audience to be aware of just happened in a very detailed manner. One of the details he mentioned was glowies having moved their focal point to individual developers and their tooling & build systems. At least some people who matter have been working on these threats for many years already, maybe more people will start to listen to them; in such case this entire debacle could have a net positive effect on the long run.
Macha 3y ago
> Are software developers (finally?) going to remove dependencies(now proven to be liabilities!), causing months of refactoring and rewriting, without any other progress? We've been here before, with e.g. event-stream and colors on npm. So I don't think it will change much. Except maybe people will stop blaming it on JS devs being script kiddies in their mind, when they realise that even the traditional world of C codebases and distro packages are not immune.

throwaway63467 3y ago

Seems like a sensible thing to do, assuming this is a state-level threat actor there’s really no easy way to prove that their contributions are free of back doors. Seems not worthwhile risking the security of a large part of the Internet over a few thousand lines of code.

afc 3y ago
But why would the entire behind this submit all their attacks through the same single identity? Removing all this code could just be removing 1% of their harmful code. How do you deal with the rest? How do you discover the other identities?
- rwmj 3y ago
  You start with what you know about, and you investigate other projects carefully at the same time. There's no easy answer here, you do what you can.

Macha 3y ago

Yeah, it's left me a little disappointed in Arch in particular that they didn't follow the lead of Debian and Fedora and revert to a much older version, instead just building 5.6.1 from the git repo and basically defended it as "the hacked build script checked for dpkg/rpm anyway".

antoinealb 3y ago
Arch Linux is not vulnerable to this specific attack, which requires sshd to be linked to liblzma. This link is provided by out-of-sshd patches, that Arch does not apply to their build.
- skywhopper 3y ago
  But as this article points out, liblzma is used in other crucial processes, and is generally trusted, often probably being run as root. The known bad actor contributed lots of code to xz that isn’t involved in the SSH backdoor. To assume it’s all innocuous would be truly foolish.
- cowsandmilk 3y ago
  The point here is there is uncertainty in all commits by Jia Tan, Arch’s focus is on this specific hack, but are there other vulnerabilities in the hundreds of commits to the git repo from the same author?
diggan 3y ago
Is this what you're referring to? > Regarding sshd authentication bypass/code execution > Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command: > However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist. https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore... I'm not finding anything from Arch mentioning dpkg/rpm, the linked article above is the latest article about the xz compromise from the Arch homepage.

skywhopper 3y ago

my gut instinct is that xz needs to be rolled back to its pre-attack state, but obviously that would probably also reintroduce some bugs and likely break some things. Still, very curious to see some analysis on the impact of doing so, because this article points out, xz is in a critical path for lots of system level processes.

fullstop 3y ago
There were symbol changes in recent releases, and things like apt link to liblzma. If liblzma were downgraded without also updating apt at the same time you could be left with a non-functioning apt.

fsflover 3y ago

> What if xz contains a hidden buffer overflow or other vulnerability, that can be exploited by the xz file it's decompressing? If you generalize this problem further, to all packages, then the only reliable solution is security through compartmentalization. On Qubes OS, any file I open, including .jpg and .avi, can't have the access to my private data or attack the admin account for the whole computer. This is ensured by hardware-assisted virtualization.

JoshTriplett 3y ago
> the only reliable solution is security through compartmentalization I hope we get there eventually. Not just for standalone processes, but for individual libraries. A decompression library could run inside a WebAssembly sandbox, with the compressed file as input, the uncompressed file as output, and no other capabilities.
- funcDropShadow 3y ago
  What does this have to do with WebAssembly? That is another runtime that adds complexity. Apple has been sandboxing codecs for a long time. They run in a sandboxed process that is only communicating through stdin and stdout or something similar, if I remember correctly. You can ran native code directly. Adding a runtime with a JIT-compiler makes it harder to understand what is going on.
  - 01HNNWZ0MV43FF 3y ago
    There's AOT wasm too. Firefox uses it to sandbox some stuff. https://hacks.mozilla.org/2021/12/webassembly-and-back-again-fine-grained-sandboxing-in-firefox-95/ https://hacks.mozilla.org/2021/12/webassembly-and-back-again...
  - JoshTriplett 3y ago
    WebAssembly can be run in-process rather than requiring a process switch, and it can be easier to port library code to run inside a WebAssembly sandbox than a completely separate process. Also, sandbox mechanisms for separate processes are not always as robust, since they have to give access to any direct syscalls the process makes, whereas WebAssembly completely insulates a library from any native surface area.

irdc 3y ago

One thing that comes to mind is that “Jia Tan” might be more accurately seen as a “sleeper” of some sort: a foot soldier who infiltrates a juicy open source project and waits for further instructions; backdooring sshd might not have been part of the original plan. Which raises the concerning question of how much more sleeper maintainers there are.

berkes 3y ago
I once got a (probably scam) offer for adding a cryptominer to a library that I maintained at that time. And a more serious offer to add trackers to a popular >1M installs app. Both cases I obviously ignored it. But it made me aware of a nasty attack vector: someone who's thanklessly building a wordpress-plugin, pip, npm, or whatever software, thanklessly dealing with issues, PRs, support, maintainence, often for no pay, suddenly gets offered three figure sums to add a few lines of "affiliate stuff" or such. There are many places in the world, or people in situations where this amount of money really makes a compelling case.
- irdc 3y ago
  “Given enough underfunded maintainers, all security is shallow.”[0] 0. https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law
  - lenerdenator 3y ago
    I wouldn't say "funding" is necessarily the problem. Most maintainers do it because they like doing it. Their main limiting factor is time. I can drop a million dollars an hour into a maintainer's lap; that doesn't mean they can dedicate every waking moment to a project. They still have human needs that money can't buy like sleep, family obligations, and health concerns. And that's making the assumption that the maintainer uses that million/hr to quit their job. No, the problem is a lack of trustworthy candidates for maintainership and a lack of time. There are components of a GNU userland that are now too complex for a single human to both maintain and enhance at the same time. We now need to target multiple distros (really, more than are necessary, strictly speaking) and ISAs. Most are written in systems programming languages like C that are more complex than the average software engineer in 2024 works with. We need consolidation, simplification, maintainer redundancy, and a trust/governance framework for packages.

fsniper 3y ago

It's like security 101. If a system has been infiltrated, you can't trust any part of it. So it's better to discard any part that has been reached or possibly affected. Perhaps it's the correct action to distrust xz/lzma or any source code this team has control over and switch to alternatives. If there are no alternatives, to start ones.

lrem 3y ago
But this suggests reimplementing xz/lzma. Which would cost money. Hence, won't be done.
- frankjr 3y ago
  But there are alternatives, most notably zstd.
  - WesolyKubeczek 3y ago
    It's a different algorithm made for a different purpose.
  - fsniper 3y ago
    a half-arsed search resulted in this half-baked rust library: https://github.com/gendx/lzma-rs https://github.com/gendx/lzma-rs
  - kzrdude 3y ago
    sadly, the zstd cli tool links to lzma right now (as installed by some distros) :/
- dspillett 3y ago
  > But this suggests reimplementing xz/lzma. If there is a known good copy of the repo from before the attacker had sufficient access to alter history, then that is an acceptable starting point. From there you look at each update since and assess what they do to decide if you want to keep (as they are valid improvements/fixes) or discard them. If some are discarded, then later ones that are valid may need further work to integrate them into the now changed codebase. Similar to Debian assessing upstream security patches to the latest version to possibly back-port them to the version they have in stable, when there is significant disparity (due to a project being much faster moving than Debian:Stable). As xz/xzutils is a relatively stable package, with very few recent changes, this should be quite practical. A full rewrite shouldn't be needed at all here.

logro 3y ago

Trusting anything from that actor is full on ignorant, let alone "a new decoder". It's insane.

iso8859-1 3y ago
Trusting people in general is inadvisable. I haven't trusted anyone for years and I am richer than ever.
- codetrotter 3y ago
  You probably still rely on trusting others a lot more than you realize. If you really, really, didn’t rely on trusting anyone, I don’t even see how it would be possible to exist on earth.
- logro 3y ago
  Those two topic don't have much in common, trusting a state level hacker actor vs. trusting people in general.
- frankjr 3y ago
  You're trusting millions of people just to be able to write this comment.

rwmj 3y ago

I think Joey's right that we should all go back to the "pre-Jia-Tan" xz, and I've raised this with Red Hat too. It's actually not a big deal as xz and liblzma is relatively stable and the version from 2 years ago is fine, although I understand that Debian's dpkg uses some new API(s) from liblzma which makes this a problem albeit a minor one. (Unfortunately the Debian bug report that Joey filed got derailed with a lot of useless comments early on.)

iso8859-1 3y ago
How do you know what 'pre' means given that pseudo-anonymous identities are free and Tan is already suspected of having some (e.g. Hans Jansen and Jigar Kumar: https://research.swtch.com/xz-timeline https://research.swtch.com/xz-timeline)
- rwmj 3y ago
  I mean we go back before all possible sockpuppets. We do have a reasonably good idea of when the attempt started.
  - glitchcrab 3y ago
    The point that the person you replied to is trying to make is how do you know when the repo is clean? How can you ever be sure that someone hasn't introduced a backdoor at some point? It's bigger than just what has been discovered.
    - rwmj 3y ago
      How do you know anything has not been compromised? You go and look at the commits and the code. It's hard work with no easy answers despite what many think.
    - rrr_oh_man 3y ago
      By not letting the perfect be the enemy of the good-enough-for-now

ducktective 3y ago

I'm here wondering why big tech companies that have Too Much to Lose didn't already massively fund a project that freaking sshd depended on it (through systemd). Like how does it hurt Google to assign 100 people to review and investigate commits of some project as basic and fundamental as a compression tool

junon 3y ago
sshd didn't depend on it, to be fair. Not officially at least.
WesolyKubeczek 3y ago
Poke at any large company at all, and you'll find that their in-house critical fundamental infrastructure thing is chronically underfunded, understaffed, bug-ridden, everyone is worried but no budget is ever approved.
- ghaff 3y ago
  Within a given part of an organization, just about everyone thinks they're underresourced and understaffed.
01HNNWZ0MV43FF 3y ago
Very shrimple https://en.wikipedia.org/wiki/Tragedy_of_the_commons https://en.wikipedia.org/wiki/Tragedy_of_the_commons
dlandau 3y ago
That's exactly what they do, though: https://cloud.google.com/assured-open-source-software/docs https://cloud.google.com/assured-open-source-software/docs
- arp242 3y ago
  Google also does code reviews for some commonly used projects (or maybe that's part of the same thing? I don't know). I went through that last year with one of my Go libraries. The idea is good, but the entire process is so bureaucracy-heavy and time-consuming that I found it both frustrating and entertaining in equal parts; like something out of Brazil (the film, not the country). So many emails, so many video meetings, so many people involved, so much talking. And all for looking at a 4,500 line Go library. "Here's the code; just clone and look at it, and let me know if you find something"... It's not like you need my permission to do any of this *shrug*.

rfoo 3y ago

Nit-picking but, eh, png does not use lzma at all. > PNG compression method 0 (the only compression method presently defined for PNG) specifies deflate/inflate compression with a sliding window of at most 32768 bytes. Deflate compression is an LZ77 derivative used in zip, gzip, pkzip, and related programs.

jwilk 3y ago
I don't think joeyh wanted to imply that PNG uses liblzma. PNG is just a convenient place to put opaque binary stuff that'd trigger an xz compression bug.
- rfoo 3y ago
  I still don't understand how would that work. The post said: > Let's say they want to target gcc. Well, gcc contains a lot of documentation, which includes png images. So they spend a while getting accepted as a documentation contributor on that project, and get added to it a png file that is specially constructed, it has additional binary data appended that exploits the buffer overflow. And instructs xz to modify the source code that comes later when decompressing gcc.tar.xz. It says "when decompressing", and I would imagine that such a bug needs specifically constructed lzma stream to trigger. If you want to do it by changing a source file (a png here) you need to make "second-order" bugs: i.e. the compressor needs to output a broken lzma stream which when later decompressed would exploit (not simply cause) a memory corruption bug. This is too brittle [1] and are very likely to be detected. [1] Disclaimer: I'm not an expert in writing backdoors. I consider myself reasonably competent for writing exploits, and I've written deliberately buggy programs (for CTFs) before.
  - deleted 3y ago
    [deleted]
  - j16sdiz 3y ago
    Consider this backdoor: - if the decompressed stream contain a magic keyword, run the rest of the file as x86-64 binary. Now you just need any opaque binary file to host the payload. A PNG works fine, because most decoder don't care extra bytes at the end.

moomin 3y ago

> Hopefully, Lasse Collin will consider these possibilities and address them in his response to the attack. Here's the thing: Lasse Collin was overloaded back in 2021. I've no particular reason to believe that isn't still the case. He needs help. Dealing with this solo is an incredible amount of work. Also, he needs help from a verifiably trustworthy source, verifiable in a way that doesn't require a lot of effort. In practice, that almost certainly means help from a major open source company. I seriously doubt that's going to happen, because the people who really need to learn this lesson won't, because it's probably not in their financial interest to realize that supply chain problems start with them doing things on the cheap. While we keep on running the world's infrastructure like XKCD 2347 every so often everything's going to topple over.

j16sdiz 3y ago
... but `xz` is pretty much feature complete to me. Lasse Collin was doing bug-fix-only release just fine.
- adql 3y ago
  The start of the attack was few fake accounts trying to shame the maintainer for not "developing" it constantly and so give maintainer rights to someone else. And there wasn't really anyone to say "nope, it's fine, fuck off"
  - FdbkHb 3y ago
    > And there wasn't really anyone to say "nope, it's fine, fuck off" That's because for people for whom the project is doing fine and they haven't experienced any bug, why would they go to the mailing list, forum or whatever other form of communication channel the project has? One has to understand and keep in mind that places that can gather feedback will invariably attract more of the negative kind than the positive kind because people who are happy are not motivated to say they're happy. Those people wouldn't even know people were complaining about xz. There's thousands of libraries/independent software projects installed on any computer. No one has the time to check the place of all those software projects and go there just to say "hey, I'm happy, no need to change anything, thanks?", right? People who are discontent with something on the other hand are sure to be vocal about it. But just because they're the most vocal doesn't mean they are the majority of your users.

Maledictus 3y ago

Is XZ embedded affected by any of this?

squarefoot 3y ago
Good question. Embedded systems are harder, sometimes impossible, to upgrade, and there are chances a backdoor in a small board inside an appliance that doesn't offer easy physical access could take years before it is found and removed.

KyleSanderson 3y ago

There seems to be a fundamental misunderstanding with a lot of these writeups. Are they 100% sure history was not rewritten at any point? Going back in time on the repo prior to listed involvement doesn't do anything as the attacker had full control. Starting from the last signed release prior to their involvement is the only way to actually move this forward (history may be fully lost at this point), the rest is posturing.

AtNightWeCode 3y ago
People think git is immutable. It is not.
- fl7305 3y ago
  Can you elaborate? Are you thinking of intentional SHA-1 has collisions? Would that work in practice?
  - AtNightWeCode 3y ago
    The history. Every time something like this attack happens people think they can read the complete git history in the repo.
    - fl7305 3y ago
      If some commits are signed by people you trust, can the chain before that still be compromised?
- Lichtso 3y ago
  Yes and no. A local GIT repo can be changed (including its history) however you please. But once you have shared it with others you can't take that back. If you try to, then others will notice that the hashes mismatch and that their HEAD diffs uncleanly. I know the term is infamous here, but GIT is essentially a blockchain. Each commit has a hash, which is based on the hashes of previous commits, forming a linked list (+ some DAG branching).

meinersbur 3y ago

The greater concern should be how many other sleeper contributors are out there. Anonymous contributions are accepted every day, and we know of cases with malicious intent such by "James Bond" (https://lore.kernel.org/lkml/20200809221453.10235-1-jameslouisebond@gmail.com/ https://lore.kernel.org/lkml/20200809221453.10235-1-jameslou...). I am not specifically worried about other contributions by "Jia Tan", those are being extensively looked at right now. They and other sleepers may just as well have contributed to any project with a different name and therefore "Jia Tan" does not pose more danger than any other contribution whose submitter cannot be held responsible.

rkta 3y ago
What's malicious about that patch? From reading the thread it looks like an attempt to fix a FP from some tooling.
- meinersbur 3y ago
  One of the patches that the University of Minnesota was banned for from contributing to the Linux kernel. They were trying to introduce a use-after-free (Fig. 9 in their paper). https://news.ycombinator.com/item?id=26887670 https://news.ycombinator.com/item?id=26887670
  - meinersbur 3y ago
    I just had to think about how ironic it would be if "Jia Tan" turned out to be a Post-Doc from the University of Minnesota continuing that research on hypocrite commits.

lispm 3y ago

Why not get rid of xz completely? Hoe about using a simpler piece of software, which could be maintained by more people?

deleted 3y ago
[deleted]
adql 3y ago
> Why not get rid of xz completely? ... you mean compression package used by most big distros to make their packages? Do you really need to ask > Hoe about using a simpler piece of software, which could be maintained by more people? It's not a complex piece of software. Lib itself is ~15k lines of code. It does it's job well and it needs little work. It didn't had any outstanding bugs lingering unfixed for years Complexity have nothing to do with the problem, it's just... uninteresting enough that there is no reason for contribute.
- imjonse 3y ago
  Complexity is one of the root causes of the problem. The script was added via messy autoconf scripts inscrutable to most people. That is decades old tech which has alternatives.
- simoncion 3y ago
  > ... you mean compression package used by most big distros to make their packages? Do you really need to ask Because of the existence of gzip, zlib, bzip2, and many others, it's trivial to drop xz. So yeah, why not get rid of xz completely?
- lispm 3y ago
  I've seen for example here a lot of issues raised: https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html

deleted 3y ago

[deleted]

loftsy 3y ago

I took a look at the diff linked in the article with code that "we are all running". The top of the diff certainly looks interesting. They remove the bounds check in dict_put() and add a safe version dict_put_safe(). This kind of change is difficult to make without mistakes because it silently changes the assumptions made when code calling dict_put() was originally written. ALL call sites would need to be audited to ensure they are not overflowing the dictionary size. The diff I am referring to is here: https://git.tukaani.org/?p=xz.git;a=commitdiff;h=de5c5e417645ad8906ef914bc059d08c1462fc29 https://git.tukaani.org/?p=xz.git;a=commitdiff;h=de5c5e41764...

logro 3y ago
This is 100% malicious or novice coder. And we surely know it's not the latter. If you need an unsafe call, you add a dict_put_unsafe(). That again should of course be rejected in a code review.
kmfpl 3y ago
I agree, this looks extremely sketchy. Especially because the code is just writing a fully controlled byte in the buffer and incrementing its index. This would give you a controlled relative write primitive if you can repeatedly call this function in a loop and going OOB.
liendolucas 3y ago
I think at this point is clear that everybody has to assume that XZ is completely rotten and can no longer be trusted. Is it XZ easy to replace with some other compression tool? Or has it been so widely adopted that is going to take huge effort moving out of it?
- kzrdude 3y ago
  Huge effort, because it is the default .deb compressor in Debian for example
  - rthnbgrredf 3y ago
    Arch Linux has replaced it with zstd in 2020 already. It's doable for the next major release of Debian.

deleted 3y ago

[deleted]

rpigab 3y ago

What if over 80% of all open source projects are secretly sleeper agents for various malicious actors, states, terrorists and whatnot, and they pretend to give use precious software updates for free so they can attack later? What if proprietary software is running the same way, except they don't even give you free updates and you can't audit the source code and have to trust them when they push updates? What if your mother gave birth to you just so she can slap you in the face when you're 30? Yeah, we can go very far, but in this moment, xz is under so much scrutiny that in 2-4 weeks, I'd trust it with my life unless the big orgs looking at it issue more reverts (hence, the delay). So if there are issues, they're everywhere else.

codezero 3y ago

Consider “Jia Tan” started working on xz because they already found a critical vulnerability and wanted to maintain it, or more tin foil, they burned xz to get upstreams to use another compression that is also already backdoored. When dealing with state actors there’s really no limit to how complex the situation can be.

alt227 3y ago
This is something I also wondered but havent seen discussed anywhere. This could all be a smokescreen to get distros to switch to the next best compression library which already contains malicious code. Hopefully maintainers of any upstream compression libraries are all looking hard at their code bases right now.

nomilk 3y ago

It seems detecting holes in jia’s code could be extremely difficult. Given the stakes, as a precaution, would it be viable to simply wipe and rewrite (from scratch) the last ~2 years of commits to xz?

imjonse 3y ago

I am wondering if the person who sent the patch to disable systemd reliance on lzma shortly after the release of the backdoored xz knew about the plan. Maybe an agent of a competing entity?

cletus 3y ago

The US spends $30-60 billion a year on agriculture and subisides [1]. This is controversial for all the obvious reasons. Without subsidies we'd waste less food but overproduction of food is an intentional objective of these programs. Why? Because if there's a major drought or crops are lost to ice or snow or flooding, Americans won't starve. It's why we have things like the US government having a reserve of over a billion pounds of cheese [2]. People not starving is a national security interest. It's getting to the point where the software we rely on is also a national security interest. The US government should be paying to maintain and improve this software. Security risks in Linux and core packages threaten to shut down key infrastructure. Paying developers to maintain open source projects that are actually used could be an incredibly effective use of tax dollars. [1]: https://usafacts.org/topics/agriculture/#581290c9-a960-49aa-a4f9-ab52dec5bbdd https://usafacts.org/topics/agriculture/#581290c9-a960-49aa-... [2]: https://www.deseret.com/2022/2/14/22933326/1-4-billion-pounds-of-cheese-stored-in-a-cave-underneath-springfield-missouri-jimmy-carter-reagan/ https://www.deseret.com/2022/2/14/22933326/1-4-billion-pound...

chjj 3y ago
I'm not sure why people keep misidentifying the problem as "lack of funding". Lasse Collin was doing fine as a maintainer up until Jia Tan showed up. He was psy-op'd into believing there was a crowd of angry people eagerly awaiting a new release when there wasn't. No real person was unhappy with the way he'd been maintaining xz.
- Cthulhu_ 3y ago
  Funding aside, single individuals being responsible for software is not a good thing, see bus factor.
  - rebolek 3y ago
    Fortunately there was Jia Tan to help him. /s
  - Gormo 3y ago
    In fact, there was no issue with Lasse Collin maintaining xz as a single individual, and creating a false impression to the contrary was the primary tactic used by the antagonist to gain access to the project.

bilekas 3y ago

> It feels good to not need to worry about dpkg and tar. I only plan to maintain this fork minimally, eg security fixes. This is exactly the problem in the first place, lack of support for maintainers. OP themselves say "I will only minimally maintain this fork". Okay, but it's so easy in hindsight to criticize what has happened. > Hopefully Lasse Collin will consider these possibilities and address them in his response to the attack. I can't even imagine how he's feeling these days.

dmitrygr 3y ago
> I can't even imagine how he's feeling these days. None of this is his problem or fault. I see no reason he should feel anything about this. Keeping everyone safe was never his job. He wrote code and gave it away for free. That should be enough.
- bilekas 3y ago
  > I see no reason he should feel anything about this. Absolutely agree, but from the sounds of the emails at least, he was going through a bad time then, and nobody feels good when they realise they were taken advantage of.
- david_draco 3y ago
  anyone who has ever been pickpocketed or robbed or worse will know reason and feelings are different things

sergioisidoro 3y ago

It's a bit ironic that after a trust attack this person ends the article sayin > I do have a xz-unscathed fork which I've carefully constructed to avoid all "Jia Tan" involved commits. He may be fully legitimate, and perhaps a famous person in OSS (which I was unfamiliar with), but still ironic :)

WesolyKubeczek 3y ago

From reading comments here alone, I can predict the following (unhealthy) effects on software at large: first, huge scaremongering like this writeup. It all hinges on a notion that who knows who is writing walls of who knows what code, and the code and its purpose is absolutely unpenetrable, inscrutable by anyone else at all. It's not true, as evidenced by analyses of this hack alone and by reverse engineering community at large. Of course, it requires doing what Anders has done, meaning, rolling up your sleeves and actually reading the source and trying to understand it, and not just hope someone else will do it. Whatever one person tangles, another can always untangle. Then, there is going to be a witch hunt and people jumping on any innocent change with pitchforks (already happening, satanic panic over ibus is in the nearby discussion). The code review theater will be in full swing. Previously, people were conserving their brain energy (or masking their incompetence) by skimming walls of changes and stamping LGTM on them based on how well-formed they had been, how long they had known the author, and how the builds were not broken. Now it will become extremely hard to get any changes done at all, because the new mental shortcuts will be: too long, didn't approve; explanations too complex; just plain Reviewer Says No; and so on. Any sloppy PR denial will follow by patting oneself on the back, look at me, I just have thwarted a KGB agent. Some people thinking they know a lot without basing such assertions in reality will try to become overnight „wonder experts in security”, barking on every shell script they don't understand, or every piece of generated text, like generated Makefiles. Vulnerabilities akin to CVE 2022-3786 and CVE 2022-3602 — which got introduced by writing a whole new email address parser from scratch (which IIRC also got checked in as a whole wall of code at once, and I read someone blaming exactly this as the culprit) — will lead to questioning by police at least once. Automated codebase scans with bogus reports like Daniel Stenberg wrote about in [0] will be more abound. Everyone will just jump on any "unsafe" function call without actually understanding its context, and will keep pestering authors to "make a fix" because potentially something (gasp!) may happen. Later everyone will get tired of this charade, and everything will come back to "normal", probably with added processes and red tape to make FOSS maintainership even more of a liability than it is today. Nothing will be done at all to make it less of a burden. [0] https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...

deleted 3y ago

[deleted]

lakomen 3y ago

I'm disappointed, to put it very mildly how archlinux handled the matter. They still use version 5.6.1 and assume that switching from github to the one hosted by Lasse fixes the issue. They say "our sshd isn't compromised". But what the author of this article wrote, who knows what else might be affected. There's a forum post on the archlinux forum, which was closed by ewaller, an administrator account, with a weird reason, that this thread was only to inform people, but when people started calling out the malpractice of the archlinux maintainers the thread got locked. To me, this is very suspicious.

geggo98 3y ago

That’s a good start. In the long run probably three things are necessary: 1) wiring critical software in a language that protects better against such exploits. Might be Rust, Go, perhaps also C# and Nim. 2) Making reproducible builds the norm, that start from the original source code repositories (e.g., based on a Git hash) 3) making maintainers more resilient against social attacks. This means more appreciation, less demands, and zero tolerance against abuse. If the maintainer can be pressured, I am at risk. The last one is probably the most difficult.

JetSpiegel 3y ago

Why not add another layer to the tinfoil hat? How do we know "Jia Tan" is not a Facebook op to "nudge" people to switch to Zstandard?