7 ms·
How the XZ Backdoor Works
- 1vuio0pswjnm7 3y agoIf the backdoor relies on IFUNC then it would not work on distributions using musl. Like the one I use. Always amusing when HN commenters try to belittle use of musl and promote use of glibc. I like musl. I like choice.
- lifthrasiir 3y agoIFUNC was probably chosen because most systems run glibc anyway so that is enough for the widespread vulnerability. The exploit would have used other mechanism tailored for musl if it was a majority instead.
- realusername 3y agoxz is bundled in millions of combination of OS/architecture/tooling/config, even the attacker had to make a choice.
- SV_BubbleTime 3y agoLends some credit to the idea that they had a target or series of targets in mind when deploying this seemingly general attack vector.
- probably_wrong 3y agoIf you really want to bait HN you should point out instead that this backdoor exclusively affected distros using systemd.
- dist-epoch 3y agoHey, you should put a trigger-warning on that
- adql 3y agoBackdoor relied more on compromising supply chain than on particular glibc feature. And the fact the notify support in sshd wasn't implemented directly but via libsystemd that is the one that dragged the lzma dependency into SSH
- fl0ki 3y agoA lot of people who don't like musl just prefer not to have the slowest memory allocator on the planet, especially when multiple threads are involved which they usually are in modern code. I've watched release-optimized builds with musl run orders of magnitude slower than unoptimized builds with mimalloc, and this was on only 20 cores, it wasn't exactly pushing the envelope of big iron scaling. Even after waiting years for its mallocng, which is better, it is still the slowest. It's no longer just that it wasn't a priority, it's inherent fallout from musl's practice of reinventing everything without learning nearly enough about why other implementations were built that way in the first place. With memory allocators there were several legendary implementations to learn from. My pick would be mimalloc because it's not just fast, it's also hardened, which seems relevant in a thread about security. At least once you know this, you can substitute the allocator for your program even if you otherwise use musl. That's common practice when producing static binaries from Rust code. The problem remains that far too many people just use musl without actually benchmarking their program under a real workload to see just how much they've sacrificed in return for, well, what exactly, because it also wasn't hardening.
- _joel 3y agoAgreed, it's not an insignificant addition to the time of a pipeline run (in our use case) and adds up if you're running a good number. Spent a bit of time porting some images to alpine and it wasn't really worth the effort. If you're restricted by device resource, which what it was created for I guess, then fine.
- deng 3y agoYes, using a niche system is a good way to not getting hacked, but then, why would you use Linux at all? Looks like you'd be happier with OpenBSD, or maybe Plan9?
- 1vuio0pswjnm7 3y agoLinux is not the only system I use. It is just one option. I compile sshd from source. It's not always OpenSSH, but when it is, it does not include any patches for systemd.
- 1vuio0pswjnm7 3y agoWhat HN commenters fail to grasp about what might drive one's choice to use musl: It's not that musl is so good, it's that glibc is so bad. At least this is what drives the choice for me.
- fl0ki 3y agoI'll take this bait in good faith: What about glibc is so bad that you'd rather use the slower, buggier, less compatible, less throughly reviewed musl in production? If I had to pick something, it'd be that glibc is behind musl on supporting 64-bit timestamps on 32-bit targets; and at least for my targets, that's a stretch. In return, musl's (past?) problems with domain resolution alone seem like more than enough of a downside to avoid it on all targets. https://twitter.com/RichFelker/status/994629795551031296 https://twitter.com/RichFelker/status/994629795551031296 vs https://datatracker.ietf.org/doc/html/rfc7766#section-5 https://datatracker.ietf.org/doc/html/rfc7766#section-5 Disclaimer: I'm targeting these from Rust, not C or C++ directly.
- 1vuio0pswjnm7 3y ago"I'll take this bait in good faith: What about glibc is so bad that you'd rather use the slower, buggier, less compatible, less throughly reviewed musl in production?" One might conclude from reading this question that someone has already made up their mind. Why would someone framing a question in this way have any interest in what I prefer and why. I am not a "developer". I am an "end user" that compiles programs for own use. Is this a "good faith" question. "Bait" indeed. The difficulty with compiling static binaries using glibc is the deal-killer for me. Coming to Linux from NetBSD, I never experienced any difficulty compiling static binaries using NetSBD's libc. Rarely do I encounter software I want to use that does not compile easily with musl. Instead I encounter software authors that have made the effort, if any is required, to be musl-compatible or that use musl exclusively (see spitbol), including Linux distributions that have adopted musl as their libc (see Alpine Linux) or offer a musl option (see VoidLinux). If others prefer glibc, then I have no problem with that. What I do not understand is why HN commenters want to jump on every comment where I mention using musl myself and try to disparage it. Why would they care what libc I use. If anything, these snarky comments only make me more skeptical of glibc.
- somemisopaste 3y ago> (...) the backdoor adds an audit hook. The dynamic linker calls all the registered audit hooks when it is resolving a symbol. How was this possible without also modifying the LD_AUDIT var? Haven't seen that mentioned yet, or perhaps I'm missing something.
- rwmj 3y agoWhen you're running inside the binary you can do mostly whatever you want. Especially in this case where the back door could run before mprotect(2) has been used to write-protect critical structures like the GOT and PLT (not that that is watertight either).
- tgv 3y agoI guess this is the answer (from https://sourceware.org/glibc/wiki/GNU_IFUNC https://sourceware.org/glibc/wiki/GNU_IFUNC): > Symbols of type STT_GNU_IFUNC (GNU-specific extension) are treated differently from normal symbols. Such IFUNC symbols point to the resolver function, and all calls to such functions are delayed until runtime.
- nwellnhof 3y agoIt's probably as easy as modifying "extern struct rtld_global_ro _rtld_global_ro", exported from ld-linux, the dynamic linker/loader. During IFUNC resolution this struct seems to be writable.
- somemisopaste 3y agoSo in other words LD_AUDIT is useless? If it's that easy to overwrite the GOT I fail to see the purpose in audit functionality.
- Topfi 3y agoGreat summary, especially since I missed this yesterday. To be honest, it's far above my knowledge, so I'd just like to thank Andres Freund and everyone else for investing their time, effort, and knowledge into keeping all of us secure.
- bsmartt 3y ago> To be honest, it's far above my knowledge Just means it's an even better opportunity to learn!
- H8crilA 3y agoDo we have any idea at all who has done it? From what I've read there's no credible attribution in public sources.
- kencausey 3y agoNo. I'm sure it is being looked into by multiple parties.
- bjoli 3y agoPretty much this. Some of the actors that would be affected by this are as large as can be. Sshd is used everywhere.
- npteljes 3y agoI read an analysis that creates a profile, but that's as far as public information goes at the moment. https://lunduke.locals.com/post/5467061/xz-backdoor-i-did-a-more-thorough-analysis-and-i-changed-my-mind-again-specifically-i-compar https://lunduke.locals.com/post/5467061/xz-backdoor-i-did-a-...
- INTPenis 3y agoOpenWRT also uploaded backdoored packages.
- 1oooqooq 3y agohttps://forum.openwrt.org/t/project-statement-about-xz-5-6-1-cve-2024-3094/193250/11 https://forum.openwrt.org/t/project-statement-about-xz-5-6-1... you might want to tell them that.
- INTPenis 3y agoI think they know seeing as they renamed the files with the .backdoored suffix. But I think you're referring to the fact that it was dormant code. I still thought it worth mentioning in the context of a website that seemed to list all distros affected. >Debian sid, Fedora Rawhide, the Fedora 40 beta, openSUSE Tumbleweed, and Kali Linux all briefly shipped the compromised package. NixOS unstable also shipped the compromised version, but was not vulnerable because it does not patch OpenSSH to link libsystemd.
- pxc 3y ago> NixOS unstable also shipped the compromised version, but was not vulnerable because it does not patch OpenSSH to link libsystemd In addition to not including those patches to build OpenSSH against systemd, Nixpkgs builds liblzma from the GitHub snapshots, not the dist tarballs, so the source Nixpkgs builds from never contained the malicious m4 scripts that actually inject the backdoor code in the liblzma binaries. Plus those scripts explicitly and exclusively target DEB and RPM builds, so they wouldn't have run on Nix or Arch builds.
- zekica 3y agoNot really, they updated to 5.6.x but used code from the git tag, not from the tarball, and only in their snapshot (not stable) version. The git repository never contained <i>build-to-host.m4</i>, so the injected build step was never executed and not backdoored library was produced.
- dmitrygr 3y agoI do now wonder if systemd itself is part of a similar long game by someone. It is part of everything. One compromise there and it is game over. It has all the signs: - Replaces old perfectly working systems? Check - Large and inscrutable? Check - Touches a lot of things an init system has no business touching? Check - Pushed in rather suddenly, in a coordinated fashion, and against much reasonable opposition? Check
- andrewshadura 3y agoSame old arguments debunked many times already. 1. Old systems it replaced were difficult to use and maintain. 2. It is in fact modular and the code is actually easy to understand. When I ran into a bug, I managed to find and fix it. 3. Systemd itself replaces the init and service management (initscripts etc), that's quite in the scope. Other bits and pieces like timers and resource management, are also in the scope. Other things like network management, are separate things that are only developed under the umbrella of the systemd project. 4. Systemd was not pushed suddenly, it was many years in brewing, and it became gradually adopted when it was actually ready for production use.
- swader999 3y agoReal time sophisticated astro turfing system in place to stifle any criticism - Check. Just kidding, I think the defense given holds up.
- npteljes 3y ago>One compromise there and it is game over. Consider that every agency like the NSA, and private groups like the NSO Group (authors of the Pegasus spyware) already sit on a pile of unpatched vulnerabilities, for various systems. If one vuln in systemd is game over, then the game is already over. And if what we currently experience is the game over state, then it's manageable.
- fl0ki 3y agoIt looks manageable to most of us because we don't know who was targeted and what would have happened if they weren't. We don't even know what whistleblowers were caught before they could publish their information, and we don't know what information that would have been. Imagine if nobody ever leaked what Edward Snowden did. Imagine that something similarly important, anywhere in the world, has not been leaked because of successfully exploited vulnerabilities.
- throwaway4good 3y agoIs the original git repository still available somewhere?
- rrdharan 3y agoYes: https://git.rootprojects.org/root/xz https://git.rootprojects.org/root/xz
- deng 3y agoNote the malicious m4 build scripts were not checked into git, but only put in the released tar balls. You can see the original content here: https://salsa.debian.org/debian/xz-utils/-/tree/debian/5.6.0-0.2 https://salsa.debian.org/debian/xz-utils/-/tree/debian/5.6.0...
- throwaway4good 3y agoThe test file containing the backdoor seems to be here: https://git.rootprojects.org/root/xz/commit/6e636819e8f070330d835fce46289a3ff72a7b89 https://git.rootprojects.org/root/xz/commit/6e636819e8f07033...
- deng 3y agoYes, the binary "test files" were checked in. But the code that actually decompresses and executes the shell code in that file is in this m4 script, which only exists in the tar archive: https://salsa.debian.org/debian/xz-utils/-/blob/debian/5.6.0-0.2/m4/build-to-host.m4 https://salsa.debian.org/debian/xz-utils/-/blob/debian/5.6.0...
- 1oooqooq 3y ago[flagged]
- coleca 3y agoThis may be a dumb question, but is law enforcement investigating this? Is it even technically a crime?
- H8crilA 3y agoOf course it is a crime. This is in fact more than a crime, it's a counter-intelligence problem, even if done by a non-state actor.
- magic_hamster 3y agoThis is 100% a state actor. We can also kind of narrow down who.
- shaky-carrousel 3y agoThe author's name may be a decoy. I'd have done that.
- coffeeaddicted 3y agoAs someone on reddit mentioned yesterday "Jia Cheong Tan" is an anagram of "CIA Agent John". Which may be accidental or a funny pun by the backdoor coder.
- fl7305 3y agoThat would be far from unlikely. But have we seen anything that would require more than a very smart individual with some time on his hands?
- jeltz 3y agoNo, but the patience is quite amazing which makes me think it is someone who is employed to do this either by an intelligence agency or by a major ransomware company. Of course it could just be a very patient person.
- smitelli 3y ago> The file, supposedly a corrupted XZ file, is actually a valid XZ stream with some bytes swapped — for example, 0x20 is swapped with occurrences of 0x09 and vice versa. My under-caffeinated brain can't unsee that as a silly reference to the old tabs vs. spaces squabble. Maybe it's simply a rational obfuscation choice, or maybe it's indeed a masterstroke of satire.
- bonyt 3y agoWait, you're right, 0x20 is space and 0x09 is ht/tab... they've swapped spaces for tabs...
- pdpi 3y agoThe payload is supposedly a corrupted xz file. An overzealous "replace spaces with tabs" formatter could easily cause this sort of corruption, so it seems like a perfectly reasonable test case.
- acdha 3y agoI think it’s a very shrewd use of that history: their goal was to get a potential reviewer to ignore it and pretty everyone qualified to look at that code has seen those specific values enough to likely tune it out as more tabs vs. spaces noise or perhaps think the original test case started with a file corrupted by some whitespace conversion process, which would be a very easy explanation since very similar things have legitimately happened to other projects. Whoever did this shows signs of being very familiar with programming culture and I’d bet that this was another deliberate attempt to hide in the background noise.
- y_xc 3y ago"The exploit was caught promptly, so almost no users were affected. Debian sid, Fedora Rawhide, the Fedora 40 beta, openSUSE Tumbleweed, and Kali Linux all briefly shipped the compromised package." One thing that is noteworthy is that Kali Linux integrated the backdoor briefly. So we should have a discussion about security of distros packaging speed. Is it more secure to integrate all patches as fast as possible or is it more secure to wait a bit until it is being tested elsewehere. The "elsewhere" would catch the bad effects then. Integrating security patches as fast as possible could be very costy.
- hypeatei 3y agoIt probably depends on what's being updated in the new version. If it's a vulnerability fix, then fast is good. If not, then waiting a little bit is ideal. Not sure how that would work in an automatic way without some sort of tagging system marking it as "critical" but that could be abused by bad actors too.
- greggsy 3y agoMost major distros include mechanisms to install packages not security updates, either via the package manager, or through specific repositories.
- ajross 3y ago> Is it more secure to integrate all patches as fast as possible or is it more secure to wait a bit until it is being tested elsewehere. That's a chicken/egg argument. Someone needs to be "elsewhere". Integration testing is absolutely part of security defense in depth architecture. In point of fact this was caught by running Debian testing. Delaying inclusion would have delayed discovery.
- y_xc 3y agoExactly that is my fear, too. It's like zero trust in everything. - Don't trust that any other party do the security testing for you - Don't trust any developer, it could have malicious intents - Don't trust the patches, they could introduce bugs/backdoors/... So we're straight going in the direction of protectionism and isolation. Wouldn't that be a big step back for the whole OSS/FOSS community? One thing I read the xc-writeups last days (I think it was here https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...) was that we at least shouldnt trust any devs/maintainers that have no online identity. Though the argument that you could create a fake ID is possible. So I'm still obsessed from the question, what are the lessons learned for the community from this and how restrictive the participation in OSS will become.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- miraculixx 3y agoNext attack will create a CVE and provide a "solution". Nothing spreads faster.
- 1vuio0pswjnm7 3y agoThe backdoor relies on IFUNC which means it does not work on distributions using musl. Like the one I use. Always amusing when HN commenters try to belittle use of musl and promote use of glibc. I like musl. I like choice.