12 ms·
The xz attack shell script
- mzs 3y agoI haven't seen Thomas Roccia's infographic mentioned here yet: https://twitter.com/fr0gger_/status/1774342248437813525 https://twitter.com/fr0gger_/status/1774342248437813525
- xomodo 3y agoThx. Timeline shows attack begun by adding ignore entry in .gitignore file. That is hard to detect nowadays.
- moopitydoop 3y agoThis just gives a partial high-level look at how the exploit gets planted in liblzma, it doesn't cover how the exploit works or its contents at all.
- hakdbha 3y ago[dead]
- ihsoy 3y agoThat seems to be largely grasped at straws and connecting dots without reason. For example oss-fuzz was building xz by cloning the github repo directly. There was never a chance for oss-fuzz to discover the backdoor, because only the tarball had it, not the repo itself. So that oss-fuzz PR might genuinely just be a genuine thing unrelated to the backdoor.
- sp332 3y agoThe ifunc part looked more legitimate, it was used to switch between optimized implementations of CRC. So that part was in the git repo. https://github.com/google/oss-fuzz/pull/10667 https://github.com/google/oss-fuzz/pull/10667
- _trampeltier 3y agoJia Tan asked distros to update quickly just before it became public. How possible is it, there is another account / person who learned earlyer from people around Andreas Freund, the backdoor would become public. How possible is it, there is still another insider around?
- OneLeggedCat 3y agoRight. Way too much coincidence. Jia Tan found out that it was about to become public and threw a Hail Mary. How did he find out?
- freedomben 3y agoIf the stakes weren't so high, this would be a damn fun game of murder mystery.
- rsc 3y agoI think the RedHat Valgrind report on 2024-03-04 made the Jia Tan team panic, since the one public rwmj stack trace pointed the finger directly at the backdoor. All it would take is someone looking closely at that failure to expose the whole operation. They fixed it on 2024-03-09, but then two weeks later distros still had not updated to the new version, and every day is another day that someone might hit the Valgrind failure and dig. I think that's why the sockpuppets came back on 2024-03-25 begging Debian to update. And then on the Debian thread there was pushback because they weren't the maintainer (except probably they were), so once Debian was updated, Jia Tan had to be the account that asked Ubuntu to update.
- benlivengood 3y agoThe only upside of finding this attack (aside from preventing it from being rolled out more widely) is that it gives a public example of a complex APT supply-chain attack. Rest assured there are more out there, and the patterns used will probably be repeated elsewhere and easier to spot. Obfuscated autoconf changes, multi-stage deployment, binary blobs (hello lib/firmware and friends, various other encoders/decoders, boot logos, etc), repo ownership changes, new-ish prolific committers in understaffed dependency libraries, magic values introduced without explanation.
- cassianoleal 3y ago> a complex APT supply-chain attack What do you mean by APT? If you mean Debian's package manager, that's not what this attack was. This was done upstream and affected non-apt distros just as much. It's true that upstream is part of apt's supply chain but focussing on apt is misleading. edit: why the downvotes? I get from the responses that I was wrong but given how the exploit was initially found in a Debian system and a lot of people very quickly jumped on the “Debian patched a thing and broke security” bandwagon, I don’t think it was much of a leap to wonder if that’s what was meant. Acronyms and initialisms are not the best way to convey specific information.
- mttpgn 3y agoNot apt the package manager-- it's an acronym for Advanced Persistent Threat
- supriyo-biswas 3y ago> What do you mean by APT Advanced, persistent threat.
- haimez 3y agoIt stands for “Advanced Persistent Threat” - https://en.m.wikipedia.org/wiki/Advanced_persistent_threat https://en.m.wikipedia.org/wiki/Advanced_persistent_threat
- microtonal 3y ago
- sanxiyn 3y ago> The first difference is that the script makes sure (very sure!) to exit if not being run on Linux. The repeated check is indeed mysterious. My only hypothesis is that the attacker may have thought that it should look plausible as a test input to a compression library, hence repetition.
- TheBlight 3y agoIs it though? The attacker probably has Linux x86 target(s) in mind and IFUNC support isn't guaranteed to work with other platforms.
- sanxiyn 3y agoChecking for Linux makes sense. Doing the exact same check for Linux five times in a row is mysterious.
- tamimio 3y agoIt would be an interesting plot twist if the whole thing was an AI hallucination.
- dboreham 3y agoPerhaps expected a very fast machine that might blast straight through the first few checks. Like how you needed two STOP statements in a Cray Fortran program in case it blew through the first one at 80MIPS.
- 3y ago
- frankohn 3y ago[flagged]
- afro88 3y agoDid you get an LLM to write this?
- frankohn 3y agoYes, I told ChatGPT4 what I wanted to say and I left him the wording. I don't really like the style he adopted but, in some way, it is better than my approximative, rough, English.
- ARandomerDude 3y agoI like your words more. Rough English is fine and adds a human touch. ChatGPT just makes me wonder if some shill is bot posting on HN. Besides, as a native speaker, ChatGPT reads to me like a 13 year old desperately trying to sound like an adult.
- dboreham 3y agoThe "Turing13 Test".
- afro88 3y agoIn some ways it is better, but mostly it comes across like a bot. Which makes me not trust the words. And in that way it is a net loss. Your english is good enough. Keep writing the comments yourself!
- menomatter 3y agocurious how were you able to tell? as a non-native speaker, I too use LLM to improve my wordings/clarity but modify it to maintain my rough style
- dzdt 3y ago
- deleted 3y ago[deleted]
- xyst 3y agoWhoever hired these people to infiltrate this project spent a shit ton of hours building it in such a way to avoid detection this long. Fortunately, it was so complicated that they couldn't account for all of the factors. This is why open source will always outperform closed source in terms of security. Sure it pointed a massive flaw in the supply chain. Sure it highlights how under appreciated the foundational elements of FOSS, leaving maintainers subject to manipulation. But the same attack within a private company? Shit, probably wouldn't even need advanced obfuscation. With a large enough PR and looming deadlines, could easily sneak something like this with a minimal amount of effort into production systems. By the time company even realizes what happens, you are already flying off to a non-extradition country and selling the exfiltrated data on Tor (or dark web).
- skrtskrt 3y agooff topic, but how many actual non-extradition countries are there these days (for the US)? Even countries we have strained relationships with will extradite as part of a negotiation when it's convenient for them politically. Russia probably wouldn't have even kept Snowden if it wasn't state secrets he revealed. If it was just some random data breach they would have prisoner-swapped him for an oligarch caught money laundering elsewhere.
- moopitydoop 3y agoOur adversaries aren't going to extradite their citizens to the west. And obviously if it's a state-level actor they aren't going to extradite their own actors. As someone old enough to remember the tail end of the early hacker eras (e.g. Mitnick), I don't think anyone SHOULD be extradited over this, in particular if they're not being charged with actually using the exploit. Prosecute them where they live. Should they be prosecuted 193 times over in every state on Earth? What's the nexus? Github? Every server that installed the compromised xz utils? But you are right they will deport (not extradite) foreigners who are inconvenient to them or when it is politically expedient to do so, if the foreigners are a nuisance, or as part of a political negotiation or prisoner exchange. The whole "extradition treaties" meme is a misconception. You will only get extradited if you flee to a country where you are a citizen (even dual citizen), or the ability to assert citizenship/nationality there. A fugitive fleeing to a country without an extradition treaty is subject to deportation. Every country on earth reserves the right to deny entry to or deport foreign fugitives. They might choose not to if someone is found to be a refugee, subject to the death penalty in a non-death-penalty state, etc.
- bandrami 3y agoAs a longtime autotools-hater I would say this justifies my position, but any build system complex enough to be multiplatform is going to be inscrutable enough to let somebody slip something in like this. But it really is a problem that so much software is built with what's essentially become a giant cargo cult-style shell script whose pieces no one person understands all of.
- echoangle 3y agoI think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.
- kimixa 3y agoI think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python. You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
- echoangle 3y agoThe problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.
- kimixa 3y agoLots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal. I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation. That's a lot of extra code to review, before you even get to any kind of language differences.
- cletus 3y agoMy big takeaway is that modern build systems are fundamentally broken. Autotools, even Makefiles, are (or can be) incredibly obtuse, even unreadable. If I read this attack correctly, it relied on a payload in a test file (to obfuscate it). It shouldn't be possible to include test resources in a production build. As an aside, C/C++'s header system with conditional inclusion is also fundamentally broken. Even templates are just text substitution with a thin veneer of typing. I think about Google's build system, which is very much designed to avoid this kind of thing. The internal build tool is Blaze (Bazel is the open-source cousin). Many years ago, you could essentially write scripts in your BUILD files (called genrules) that were hugely problematic. There was no way to guarantee the output so they had to be constantly rebuilt. There was a long project to eliminate this kind of edge case. Blaze (and Bazel) are built around declaring hermetic units to be built with explicit dependencies only. Nothing shipped to production is built locally. It's all built by the build servers (a system called Forge). These outputs are packaged into Midas packages ("MPMs"). You could absolutely reconstruct the source used to build a particular library, binary or package as well as the build toolchain and version used. And any build is completely deterministic and verifiable. C/C++, Make, CMake, autotools, autoconf and all that tooling so common in Linux and its core dependencies absolutely needs to go.
- __MatrixMan__ 3y agoIf I were starting something from scratch I'd do Bazel for internal deps, Nix for external ones. If that's a tall order, so be it: we train people. Investments in determinism usually end up paying off.
- threePointFive 3y agoI'm not familiar with Bazel, but Nix in it's current form wouldn't have solved this attack. First of all, the standard mkDerivation function calls the same configure; make; make install process that made this attack possible. Nixpkgs regularly pulls in external resources (fetchUrl and friends) that are equally vulnerable to a poisoned release tarball. Checkout the comment on the current xz entry in nixpkgs https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/compression/xz/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/comp...
- zadwang 3y agoThe set of unix utilities have been tested a long time. I just wish the kernel and key utilities keeps fixed and not changed. Unless absolutely necessary. Don’t fix it if it ain’t broken. The software empire seems out of control.
- JonChesterfield 3y agoIt is all broken. The quick hack to make things kind of work today, compounded across fifty years and a million developers. There are occasional shining lights of stuff done right but I think it's fair to say they're comprehensively outcompeted.
- dankwizard 3y ago[flagged]
- pizzafeelsright 3y agoThere's a lot of mind reading there.
- Solvency 3y agoIt's kind of tragically amusing how heinously complex and unnecessarily inscrutable modern technology is, and it's only getting worse. I think developers sadistically enjoy it.
- Trufa 3y agoThis is probably the worst take ever. It's pretty amazing how the tools keep up with the increasing complexity of the products we make. And to be honest, in most cases just make it simpler, I think people just don't like to learn new stuff.
- mittermayr 3y agoAs a developer, this amazes me, and it just shows what — to me — feels like a top-tier attack method, is probably only entry-to-mid level complexity for the folks working at that stage. Some of the things I see posted here on HN are well above this level, so I'd assume for the right kind of money (or other incentives), this is only the beginning of what's possible. And, if you think of ALL the packages and ALL the many millions of libraries on GitHub, this vector is SO EFFECTIVE, there will be hundreds of cases like it uncovered in the next few months, I am certain of it. I worry about all the con/pro-sumer hardware makers, from Philips Hue to Alexas, from the SumUps to the camera makers, from Netgear to TP-Link. All their products are packed with open-source libraries. And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors.
- awesomeMilou 3y ago> And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors. This rationale baffles me, it feels that the dependency-hell circlejerk crowd is working on making OSS maintainers look even more bad with this scenario. Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install. If you are affected by this mess, I'm sorry to say, but it's your fault. If you are worried about developers of software you use for free, as in free beer, going rogue, either put in incentives for them to not do that (i.e. pay them) or fork the project and implement your own security measures on top of what's already there. If you're worried that you could encounter exploits from dependencies in commercial software you use, you should negotiate a contract that includes compensation from damages from supply chain attacks. If you're unwilling to do that, sorry mate, you're just unprofessional. Inb4: Yes, I am really trying to say that you should check the supply chain of even your most basic dependencies such as SSH.
- finaard 3y agoUnfortunately that's "industry standard" nowadays. I lost count how often I had that discussion over the last two decades. Just look at stuff like pip, npm or pretty much any "modern" package manager in use by developers - they're all pretty much designed to pull in a shitload of arbitrary unaudited and in some causes unauditable dependencies. And nobody wants to listen. That's why I prefer to work in heavily regulated areas nowadays - that way I can shorten that discussion with "yeah, but regulatory requirements don't let us do that, sorry" The absolute basic should be having a local archive of dependencies which at least received a basic sanity check, and updates or additions to that should review changes being added. CI gets access to that cache, but by itself does not have network access to make sure no random crap gets pulled into the build. You'd be surprised how many popular build systems can't do that at all, or only with a lot of workarounds.
- kelseydh 3y agoThe lesson I take away from this incident is that we probably shouldn't be allowing anonymity for core contributers in critical open source projects. This attack worked and the attacker will likely get away with it free of consequence, because they were anonymous.
- hk__2 3y ago> The lesson I take away from this incident is that we probably shouldn't be allowing anonymity for core contributers in critical open source projects. This attack worked and the attacker will likely get away with it free of consequence, because they were anonymous. This would be impossible to enforce, and might not be a good idea because it enables other ranges of attacks: if you know the identities of the maintainers of critical open source projects, it’s easier to put pressure on them.
- damsalor 3y agoThe attack almost worked because of too few eyes
- Jonnax 3y agoWho designates it as critical? If someone makes a library and other people start using it, are they forced to reveal their identity? Do the maintainers get paid?
- ulrikrasmussen 3y agoNo thanks. That's not going to help, and will be fairly easy to circumvent for nation state actors or similar advanced persistent threats who will not have a problem adding an extra step of identity theft to their attack chain, or simply use an agent who can be protected if the backdoor is ever discovered. On the other hand, the technical hoops required for something like that will likely cause a lot of damage to the whole open source community. The solution here is learn from this attack and change practices to make a similar one more difficult to pull off: 1. Never allow files in release tar-balls which are not present in the repo. 2. As a consequence, all generated code should be checked in. Build scripts should re-generate all derived code and fail if the checked in code deviates from the generated. 3. No inscrutable data should be accessible by the release build process. This means that tests relying on binary data should be built completely separately from the release binaries.
- say_it_as_it_is 3y agoImagine paying for a security scanning service such as Snyk and finding that it never scanned source code for injection attacks. How many millions of dollars went down the drain?
- xurukefi 3y agoSince I'm a bit late to the party and feeling somewhat overwhelmed by the multitude of articles floating around, I wonder: Has there been any detailed analysis of the actual injected object file? Thus far, I haven't come across any, which strikes me as rather peculiar given that it's been a few days.
- lenerdenator 3y agoI agree, I haven't seen anything about decompiling the object file. If I had a project to develop a backdoor to keep persistent access to whatever machine I wanted, it would make sense that I would have a plug-in executable that I would use for multiple backdoors. That's just decent engineering.
- tithe 3y agoYour best bet may be in the chat (from https://www.openwall.com/lists/oss-security/2024/03/30/26 https://www.openwall.com/lists/oss-security/2024/03/30/26 ): Matrix: #xz-backdoor-reversing:nil.im IRC: #xz-backdoor-reversing on irc.oftc.net Discord: https://discord.gg/XqTshWbR5F https://discord.gg/XqTshWbR5F
- klabb3 3y agoAs a naive bystander, the thing that stands out most to me: > Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.” This is a fact of life for parsing libraries like liblzma. The attacker looked like they were just adding a few new test files. Yes, these files are scary, but I can see the reason. But at least can we keep them away from the build? > Usually, the configure script and its support libraries are only added to the tarball distributions, not the source repository. The xz distribution works this way too. Obligatory auto tools wtf aside, why on earth should the tarballs contain the test files at all? I mean, a malicious test could infect a developer machine, but if the tars are for building final artifacts for everyone else, then shouldn’t the policy be to only include what’s necessary? Especially if the test files are unauditable blobs.
- wiml 3y agoBecause, despite containing some amount of generated autoconf code, they are still source tarballs. You want to be able to run the tests after compiling the code on the destination machine.
- finaard 3y agoIt's pretty common to run tests on CI after building to verify your particular setup doesn't break stuff. Last time we were doing that we were preferring git upstream, though, and generated autocrap as needed - I never liked the idea of release tarballs containing stuff not in git.
- klabb3 3y agoThis strengthens the argument I’m making, no? You bring in the source repo when doing development and debugging. In either case - tarball or not - it doesn’t seem that difficult to nuke the test dir before building a release for distribution. Again, only really necessary if you have opaque blobs where fishy things can hide.
- lathiat 3y agoThe distributions often run the same tests after it’s built to make sure it’s working correctly as built in the distribution environment. This can and does find real problems.
- LunicLynx 3y agoImagine this inside GitHub copilot, just because it has seen it enough times.
- sureglymop 3y agoThis is why, in my opinion, if AI will bring exponentially more efficiency, it will also bring an exponential increase in security issues. The most common vulnerabilities are due to how input is handled (or not). XSS vulns, SQL injections, other injections, etc. Now, the AI agent itself also produces untrusted input. But it does so, in a way, as an extension of the user using it who is actually a trusted entity. Because of this, it is solely up to the user to validate everything coming out of the AI but most users will not do that thoroughly. What you mention in your comment is a good example of that. If the AI reproduces malicious code e.g. inspired from this campaign, it is essentially an injection attack where the user misses it and didn't properly validate the untrusted input.
- PHGamer 3y agoif anyone has worked in any development. (closed or open) you know half the time developers are lazy and just approve PRs. Linus Torvalds is like the glimming exception where he will call out shit all day long.
- dijit 3y agoSecond this. And in the event someone is pedantic enough to actually care: that person will be considered a pariah that all development stifles due to. Tensions with the team for nitpicking etc; FD: I have a situation like this now, I am not the one being picky- one of the developers I hired is. I had to move him out of the team because unfortunately his nitpicky behaviour was not well regarded. (he also comes from eastern europe and has a very matter-of-fact way of giving feedback too which does not aid things).
- Shocka1 3y agoThis cracks me up. I had the privilege of working on a team where about 5 or 7 of us out of 10 were extremely picky about code and documentation, all different nationalities. We would sometimes argue about a word in a doc for hours and didn't get a whole lot done. Feelings were hurt a lot, but what we did get accomplished was very high quality. I've also worked on teams where we were all cowboys and would sling code just about as fast as we could type it. Both can be fun, but as you probably already know there is a balance.
- jcarrano 3y agoWhat's the advantage of IFUNCS, over putting a function in the library that selects the implementation, either via function pointers or a switch/if? In particular given that they seem to be quite fragile and exploitable too. I don't have much experience in low-level optimization, but would a modern CPU not be able to predict the path taken by a brach that tests the CPU features.
- jcalvinowens 3y ago> either via function pointers or a switch/if? > but would a modern CPU not be able to predict the path taken by a brach that tests the CPU features. That's true, but the CPU has finite branch predictor state, and now you've wasted some of it. Indirect calls hurt too, especially in you need retpolines. This is a great read: https://www.agner.org/optimize/microarchitecture.pdf https://www.agner.org/optimize/microarchitecture.pdf The Linux kernel has interfaces for doing the same thing, more explicitly than ifunc: https://docs.kernel.org/next/staging/static-keys.html https://docs.kernel.org/next/staging/static-keys.html https://lwn.net/Articles/815908/ https://lwn.net/Articles/815908/
- account42 3y agoThe advantage is that it avoids an extra function pointer indirection when the function you are trying to swap out is already an exported library function. For purely internal functions like in this case the only advantage is that it allows this exploit.
- k3vinw 3y agoHa. This backdoor belongs in the same museum as automake!
- deleted 3y ago[deleted]
- Woodi 3y agoFew notes: - "auto" tools are installer tools and, after all, do good job. Maybe we can download any project that use it, run it once, parse results and put it in /etc ? - becouse it's already there. Then next step: build just-detection tools. Side note: just straight 'make all' will detect it too :) And it show additional next problem that auto's already solved - setting up all that inc/lib paths... - m4 should be replaced and not by monstrosity like cmake - GNU's came after MIT's and BSD's, evolved from them. But corporations adapted and assimilated. We need next evolution step: something that forces corporations to spend money on what they use, fund devs or participate in development. OSlicensecertification.orgs should help with that instead of helping corps to be predators becouse some outdated bullshit talks - yea, auto* tools are way overcomplicated - finally do that TTY 2.0 !
- djmips 3y agoSo people are talking about the review process but what if the maintainer and the reviewers are all complicit?