14 ms·
What we know about the xz Utils backdoor that almost infected the world
- shp0ngle 3y agoI didn't realize it was a Microsoft engineer that works on Azure Postgres that found the issue. Thanks, Microsoft, I like Azure now.
- farmdve 3y agoAnd I think people should now look at all oddities with Valgrind. Since that is how the issue got discovered. And then look at the problematic library and look for similar outliers of fake personas taking over a project. It seems it is common practice for people to ignore these errors.
- korginator 3y agoValgrind will tell you about memory leaks and won't always behave the way it did here when there's a backdoor. In this case it just so happened that valgrind was throwing errors because the stack layout didn't match what the exploit was expecting. Otherwise valgrind would have probably worked without issues.
- rowanG077 3y agoThe idea is not "valgrind will detect all tomfoolery". The idea here is "if valgrind detects something it may be an indication of tomfoolery.'
- farmdve 3y agoExactly. Also remember this >> odd valgrind complaint in automated testing of postgres I would imagine compiling a list of odd complaints may yield something , or nothing at all.
- valevk 3y ago> the stack layout didn't match what the exploit was expecting. What does that mean? Why is the exploit expecting something from the stack layout and why does valgrind complain?
- galangalalgol 3y agoI am also curious, and if something like asan would also have found it? It seems social engineering was used to get MS to stop fuzzing the library for malicious code, so if the malicious party expected the valgrind behavior they might have removed it as well.
- farmdve 3y agoWhich to me is a very carefully orchestrated thing. You don't just spend 2 years of your life doing that. No loner would pre-plan this to such an extent and create sockpuppet accounts for all this.
- XorNot 3y agoThat's because you're a normal, well adjusted person. Consider TempleOS[1] which was created by a programmer having a series of manic episodes which he believed was God's instruction to create 640x480 pixels of perfection. He spent the rest of his life on this. People vastly underestimate the tenacity of individual fixated people: so much so that in the physical world victims usually feel isolated by their peers who just don't believe the degree of effort they stalker will actually go to. [1] https://en.m.wikipedia.org/wiki/TempleOS https://en.m.wikipedia.org/wiki/TempleOS
- evandale 3y agoTempleOS feels a little different because Terry was fairly well-known in the community and didn't try to hide his identity. I'm pretty sure he went to conferences and has met with actual people who could verify his identity. I haven't seen proof that Jia Tan is a real person and to me that's the most malicious part of the attack. I'm pretty confident that whoever is hiding behind the Jia Tan identity is a well adjusted individual (or group) and knows exactly what they're doing. It feels far too coordinated and careful to chalk up to a psychotic episode or manic behavior.
- glandium 3y agoDo you remember the Debian openssl flaw? Okay, it's almost 20 years old now, so you may have forgotten, or you could be too young, I don't know. But it was caused by someone attempting to fix an oddity found by valgrind. https://blogs.fsfe.org/tonnerre/archives/24 https://blogs.fsfe.org/tonnerre/archives/24
- jmclnx 3y agoI remember this and the like is a good write-up, thanks for the link. I have see things like this over the past 30+ years. So a couple of Items to add: * Comment the hell out of hidden logic like this. Explain why nor what :) * Better yet, even though that uninitialized buffer helped with performance. These days it would be better to take the hit and add a random initialization of that buffer. Maybe read /dev/urandom or some other thing. You do not know who will come along after you, so try and make things explicit.
- SAI_Peregrinus 3y agoThere was an upstream OpenSSL bug there: they depended on reading from uninitialized memory to add entropy and thus increase startup speed of their RNG. But reading from uninitialized memory is undefined behavior, it's not guaranteed to add any entropy and should always be treated as a security bug. The Debian maintainers tried to fix the bug, but screwed up the fix. They should have reported the bug upstream, not just made their own fork.
- teddyh 3y agoIIRC, they did report the bug, and it was rejected.
- SAI_Peregrinus 3y agoMy memory is fuzzy. In that case, I'd blame the OpenSSL devs if the report wasn't a patch with the (non-working) "fix". Even if it was, they should have accepted the bug & rejected the patch.
- deleted 3y ago[deleted]
- mistrial9 3y agono - it is a PostgreSQL PGDG engineer hired by MSFT.. I cannot stand this spin
- yosito 3y agoAre we ever going to figure out who Jia Tan is?
- noncoml 3y agoNo
- usr1106 3y agoIt sounds unlikely that he was an individual working on his own. So if the organisation he worked for has another Snowden, yes. (I am not saying it is necessarily the same organization, there are at least 4 obvious candidates and it could be a more surprising one.)
- adzm 3y agoI am certainly curious what their private repos involve.
- ChrisMarshallNY 3y agoIf it was state-sponsored (considering the timeline, that's a real possibility), then "Jia Tan" could actually be a collection of folks, working for the same team.
- secondcoming 3y agoI think 'Jia Tan' is one person who's committing the changes, based on the language used in the commits. Whether those changes came from a group will never be known.
- hackeraccount 3y agoNever say never. There's a paper trail going back a couple of years. People especially professionals can be good with trade craft but everyone makes mistakes.
- reisse 3y agoIt amuses me how the default hacker stereotype changed from an autistic nerd person (no offense) to a state-backed USA/Chinese/Korean/Israel/Russian APT group. I can absolutely imagine people like crimew pwning sshd because it's fun! it's interesting! it's a way to get people to think more about open source community! just why not?
- mapasj 3y agoI’m guessing the original maintainer of xz handed responsibilities to Jia Tan without ever seeing him/her or at least sharing a phone call. Is that common to only communicate only through email/github? I guess some maintainers of open source projects will be more cautious after this story.
- Gigachad 3y agoThat’s basically how it is right now. Millions of companies freeloading off the work of unpaid open source developers. Unsurprisingly they sometimes leave and it causes problems.
- cjk2 3y agoThis. I've had my buggy shit accepted into at least three open source projects so far with little to no verification.
- supriyo-biswas 3y agoIf you’re being berated by multiple people as to your speed of delivery, then it is not unexpected for them to be convinced that they are somehow the problem, and transfer the project to whoever they feel at the time is the best choice without thinking through their decisions. However, knowing a person personally doesn’t necessarily solve the problem. I used to work on an open source project a long time ago (under a pseudonym) that I do not wish to name here for reasons that’ll become clear shortly. The lead programmer had a co-maintainer who the lead seemed to have known quite well. The co-maintainer constantly gaslit me, and later, other maintainers, belittled them, criticized them for the smallest of bugs etc. (and not in a Linus Torvalds way, where the rants are educational if you remove the insults) until they left; and was egged on by the lead maintainer as they agreed with the technical substance of these arguments. Many years later, the co-maintainer attempted a hostile takeover of the project, which did not go as expected, and soon after, multiple private correspondences with other people became public where it became clear that the co-maintainer always wanted to do this, and gaslighting other maintainers was just part of this goal. All of this, despite the fact that the two of them knew each other.
- GoblinSlayer 3y ago>and argued that Lasse Collin, the longtime maintainer of xz Utils, hadn’t been updating the software often or fast enough. This meme was a mistake.
- phkahler 3y agoWhy does SSH use xz? Should it? Is it really that important?
- jerrygenser 3y agoopenssh does not use xz but in some distros xz is used when it interfaces with systemd
- Tuna-Fish 3y agoIt does not. OpenSSH pulled in libsystemd to provide startup notification. Libsystemd pulled in liblzma. No code from liblzma normally ends up in OpenSSH. But because it is built as a dependency for libsystemd, it's build scripts are ran in the same environment as libsystemd, and OpenSSH. The attack payload was hidden as an obfuscated binary blob in the liblzma tests directory, masqueraded as a compression test case. When lzma was compiled from the git sources, generating the build scripts using autotools, nothing untoward was done. But lzma was also provided as a source tarball that was used by distro packagers, that had the autotools already ran. The attacker replaced the autogenerated, unreadable script output with one that checked if liblzma was being compiled in the same environment as OpenSSH and if it was being compiled so that it was going to end up as a .deb or .rpm package, and if both were true, embed the attack payload into OpenSSH. Then the attack payload started with a lot of checks, including testing whether OpenSSH was being started normally by init scripts or manually, and for the presence of usual debugging tools, and only attached the payload to the running process if it seemed like a "natural" bootup with no running debugging tools. When running, the payload hooked into private key verification, and if the correct private key attempted to login, the payload would take the rest of the incoming packet and call system with it, that is, provide remote code execution as root.
- filmor 3y agoThe build scripts of OpenSSH are irrelevant. The malicious code is embedded on building an rpm or deb for liblzma itself and becomes active when the dynamic library is loaded. There is a recent PR for systemd that instead of linking to the compressors during build dlopen-s them when used (https://github.com/systemd/systemd/pull/31550 https://github.com/systemd/systemd/pull/31550) which disables this particular path, but any initial load of a backdoored liblzma makes sshd exploitable (see https://lwn.net/ml/oss-security/20240329155126.kjjfduxw2yrlxgzm@awork3.anarazel.de/ https://lwn.net/ml/oss-security/20240329155126.kjjfduxw2yrlx..., section "Analyzing the injected code"). Lennart Poettering stated on a mailing list that e.g. libselinux als links liblzma and ends up in a lot of services on SELinux-enabled systems.
- EdiX 3y agoMy personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away from them. The other big supply chan attack (event-stream) also took advantage of something similar. 1a. As a consequence of (1) autogenerated artifacts should always be committed. 2. Autogenerated artifacts that everyone pagedowns over during code reviews is a problem. If you have this type of stuff in your repository also have an automatic test that checks that nobody tampered with it (it will also keep you from having stale autogenerated files in your repository). 3. A corollary of (1) and (2) is that autotools is bad and the autotools culture is bad. 4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. 5. In general there's a culture that code reuse is always good, that depending on large libraries for small amounts of functionality is good. This is not true, dependencies are maintenance burden and a security risk, this needs to be weighted against the functionality they bring in. 6. Distro maintainers applying substantial patches to packages is a problem, it creates widely used de facto forks for libraries and applications that do not have real maintainers looking at them. 7. We need to make OSS work from the financial point of view for developers. Liblzma and xz-utils probably have tens of millions of install but a single maintainer with mental health problems. 8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations.
- delfinom 3y ago>4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. They never did. In fact the systemd maintainers are confused on that point and adding documentation on how to implement the simple datagram without libsystemd. >7. We need to make OSS work from the financial point of view for developers. Liblzma and xz-utils probably have tens of millions of install but a single maintainer with mental health problems. Way more than tens of millions. Python, php, ruby and many other languages depend on libxml2, libxml2 uses liblzma. And there's many other dependencies. >8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations. Not any maintainer's job. OSS is provided without warranty. Also indication is "Jia Tan" may have been completely fake as their commit timestamps show even on the same day that their timezone switches from Eastern Europe to Asia. So at the very least, they were playing identity games.
- fractalf 3y agoWhat if this is just a decoy and the real backdoor is elsewhere?
- darthrupert 3y agoI think it's more likely that there are 100 real backdoors and this was one of them.
- Tuna-Fish 3y agoIt would be a terrible idea to release such a decoy, because it is now creating massive amounts of scrutiny on other libraries.
- c0wb0yc0d3r 3y agoThat could be the point... https://en.m.wikipedia.org/wiki/Misdirection_(magic) https://en.m.wikipedia.org/wiki/Misdirection_(magic)
- andrewstuart 3y agoI don’t suppose this will lead the us govt to invest heavily in the open source projects behind SSH?
- bombcar 3y agoDepending on if this Jai Tan was employed by the NSA, perhaps they already are ...
- lelanthran 3y ago> I don’t suppose this will lead the us govt to invest heavily in the open source projects behind SSH? The cynic in me wants to know, what makes you think Jia Tan wasn't one of their "investments"?
- mrkramer 3y agoAt the end of the day, backdoors in the open-source software projects will be caught sooner or later, the problem are backdoors in closed source projects ala NSA backdoors. To this day, I'm not completely sure if my Windows machine is secure or not. I remember Gates once said that governments in the early days of Windows demanded from Microsoft to show them the source code of Windows but even that is not enough since Microsoft can hotpatch whatever they want in their machines.
- shrubble 3y agoMicrosoft was the very first to cooperate with the PRISM program...see the 2nd image here https://www.theguardian.com/world/2013/jun/06/us-tech-giants-nsa-data https://www.theguardian.com/world/2013/jun/06/us-tech-giants...
- seniorThrowaway 3y agoOf course MS wants to stay in the good graces of its biggest customer
- deleted 3y ago[deleted]
- supposemaybe 3y agoSo while everyone thinks this backdoor was caught early, its purpose might have been achieved already. Especially if those targets were developers who used rolling release distros, like Kali and Debian.
- nubinetwork 3y agoThis might be possible. I picked up some SSH traffic earlier in the week, and didn't think much of it at the time. Of course, this could also be a red herring. https://www.nubi-network.com/news.php?id=21 https://www.nubi-network.com/news.php?id=21
- supposemaybe 3y agoIf it was related, it’d already be too late for many before the real payload was uploaded.
- AshamedCaptain 3y agoWhy do they say "almost" infected the world? At least 3 quite popular Linux distributions (arch, gentoo, and opensuse tumbleweed) ended up shipping the backdoor _for weeks_ , and it was most definitely working in at least tumbleweed. For weeks! A backdoored ssh! Hardly "almost".
- nequo 3y agoMy guess is that most Linux boxes that run sshd use an LTS distro, not one with rolling release. But I don’t know how to get any data on this.
- christophilus 3y agoThe exploit didn’t actually run on those distros, though. Only on deb/rpm targets. So, effectively, it was stopped before it hit production.
- deleted 3y ago[deleted]
- cozzyd 3y agoTumbleweed is rpm-based, though maybe not so used in production
- Denvercoder9 3y agoArch, Gentoo and openSUSE Tumbleweed are hardly the world.
- samtheprogram 3y agoArch and Gentoo were also not supported, although the code shipped, because the exploit explicitly checked for RPM- and deb-packaged distros. Suse is RPM based, but don’t remember whether the check was for the utilities or another method — Suse uses zypper for package management, as opposed to yum/dnf on the far more popular RedHat-based distros, so it depends how the exploit checked.
- usrusr 3y agoWhat I haven't seen discussed much is the linking mechanism that allowed the lib to hook into RSA_public_decrypt. Plenty of talk about what could or could not be achieved by even more process separation and the like, but little about that function call redirect. Could it be possible to establish a way to link critical components like the code for incoming ssh with libraries in some tiered trust way? "I trust you when and where I call you, but I won't allow you to introduce yourself to other call sites"? This would surely fall into the category of "there would be ways around it, so why bother?" that triggers a "by obscurity" reflex in many, but I'd consider it reduced attack surface.
- supriyo-biswas 3y agoAn Erlang actor like model where a program’s sub components call each other by message passing with each having its own security context may work. However, there are multiple security contexts at play in an operating system; with regards to the XZ backdoor it’s mostly about capability based security at the module level, but you also have capabilities at the program level, and isolation at the memory level (paging), isolation at the micro architectural level, and so on. Ensuring all of these elements work together while still delivering performance seems to be rather challenging, and it’s definitely not possible for the Unix-likes to make a move to such a model because of the replacement of the concept of processes.
- kzrdude 3y agosystemd merged a change to using dlopen for compression libraries recently https://github.com/systemd/systemd/pull/31550 https://github.com/systemd/systemd/pull/31550 which is a safer linking method in that sense.
- nextaccountic 3y agoWhy is it safer?
- jcranmer 3y agoIt means the libraries are only loaded when they are needed, so if you never use the (e.g.) xz compression feature, the xz library will not be loaded, and a backdoor added in the xz library simply can't trigger. (Another side note is this may change the initialization order of libraries--so the initialization functions of an xz library don't run until xz is first used, and this may fail to let you intercept the ssh routines in time.)
- humansareok1 3y agoThis represents a massive failure of the Open Source model and it's worth thinking carefully about as more and more people advocate for fully open source AI models. People could spend a lot of time on token safety features only to have them backdoored by a sneaky PR and it's not clear what level of damage this could cause in the near future.
- mikedelfino 3y agoIsn't it the opposite, though? The whole thing was caught precisely because it was open source. Sure, you can argue that the vulnerability was introduced by a contributor, but what prevents the same thing from happening in a private repository, perpetrated by an actual employee? Especially if you consider that many vulnerabilities can be introduced with code more likely to pass as legitimate mistakes.
- bombcar 3y agoOr think about the closed-source products that use open source code (basically everything these days) - had this targeted versions of xz building on Windows only it might have lain dormant longer (not sure this attacks would have been useful, but something analogous could be).
- humansareok1 3y agoBut it wasn't caught because it was open source? It was caught basically by chance by an engineer using the distro who admits they only caught it because of a series of coincidences.
- smikesix 3y agothats not an issue with open source, but an issue with trust and structure. in closed source it can also happen and it would be even harder to spot, since you have no clue what is going on and there would be maybe 1-2 ppl in the whole world that could identify that issue. with open source anyone can. the joke is that he found it because the ssh login felt too long. this dude needs a medal. what needs to be controlled are system relevant libraries that open up such possibilites and also not including not code data into builds. any binary blob is bad, anything that cant simply be read is bad. if you put in code, ppl will control it and most of the time see what it does. but if you hide the real code in a binary blob, such files should not be included in builds, but shipped separately
- sandstrom 3y agoThe goal is to use a standardized test framework to ease writing of tests in XZ. Much of the functionality remains untested, so it will be helpful for long term project stability to have more tests -- Jia, 2022-06-17 This was a long time in the making.
- xyst 3y agoWhat can be learned here and improve? - systemd and libsystemd is an absolute mess. What are the alternatives at this point? - what processes can be improved for Linux distributions to catch malicious actors trying to push back doors into the ecosystem? - is there a dependency graph that would show how many programs use a specific utility library? Might aid in finding other possible attempts to backdoor
- looperhacks 3y ago> systemd and libsystemd is an absolute mess how? Because they link against a third-party library? > is there a dependency graph that would show how many programs use a specific utility library Agree, important topic > is there a dependency graph that would show how many programs use a specific utility library Depends on the distro, but your package manager should be able to do that. On a global scale, that's likely impossible, because there's no true register of how programs are linked - in fact, the inclusion of libsystemd was distro-specific
- metalforever 3y agoBecause its a very large monolithic piece of code, which causes issues with trying to audit it / brings it code that might not be necessary / increases attack surface
- LtWorf 3y agoWell said… by someone who has clearly never opened the sources of systemd.
- TacticalCoder 3y ago> What can be learned here and improve? - which firewalling setups would still prevent that exploit from working (like whitelisting IP addresses or blocks allowed to SSH in or, drumroll..., port-knocking). I don't use port-knocking but now I really wonder what's going to be the argument against port-knocking. Is it really security theatre in the fact of the rube-goldberg too-big-to-ever-be-secure systemd, loading totally unrelated code for a freaking ssh server? This method of operating was just proved to be a security nightmare. And people who are using port-knocking (or IP whitelisting) were/are totally fine versus these kinds of exploits. So, yup, today I really want to read what people dismissing port-knocking as security theatre have to say (even though I, myself, don't use port-knocking: but I do use IP whitelisting in firewall rules to decide which IPs can SSH in).
- nabla9 3y agoI bet $101 that we find something similar in the wild in the next 12 months as the maintainers start to look at each other's past commits with suspicion.
- bombcar 3y agoI wonder if we'll find the cases that were done and used, because if I had something like this and it worked, afterwards I'd "find it" with another account and get it fixed ...
- dartos 3y agoI think attacks like this have been on the rise for the past decade. Just look at any critical, yet largely unknown codebase with very few maintainers.
- markus_zhang 3y agoI have said this before and I'm saying it again: Open source maintainers' first responsibility is to take care of him/herself, mentally and financially. You guys should proactively seek payments from whoever uses your work commercially, and if the $$ is not good enough, you are on your own to continue do this voluntarily. If you keep the front door open, eventually thieves will come and steal your stuffs. By the same principal, if you do not rigorously request payments and donations, you should expect people to take advantage of you. From this perspective, Tan, whoever he/she is, actually did you a great service -- those big companies got a rude awakening and are scrambling to double check whether they are impacted, over weekend. It is a pity that this did NOT get into a stable release -- I know it's very rude to say so, but let me be honest here -- it will be a greater help to all open source maintainers if this actually gets into a stable release and fuck over as many people as possible.
- michaelsbradley 3y agoNext logical step in this misguided way of thinking: “do a great service to the world by subverting open source software until maintainers get their due”. No, the ends do not justify the means.
- markus_zhang 3y agoSometimes good ends can ONLY be brought by bad means. Like, realistically, how would you convince the Forbes 500 to increase spending on software security? They won't budge if no one is knocking the doors. In general, it is in human nature that we prefer to mend the situation as late as possible, as long as it is not too late, instead of curing a disease before it develops. People who do the first thing are praised and hailed, while people who do the second are laughed at.
- bombcar 3y agoThe result will be Fortune 500 companies spending millions and billions on "Security Companies" who pocket the money and produce lots of paperwork absolving everyone of any form of blame, whilst the actual maintainers will still get a can of diet code and a misspelt acknowledgment somewhere.
- levi_n 3y agoIm curious as to why they picked the commit cadence they did. Why do this over the course of two years and not, say 8 months or 15 months? After committing the first patch, why did they wait x days/weeks/months to commit the second? Were they timing the commits off of release schedules, following some predetermined schedule, or something else?
- TacticalCoder 3y ago> Malicious updates made to a ubiquitous tool were a few weeks away from going mainstream. Imagine working, as an individual or as a group, for years and then getting caught mere weeks or months before most major distros were to incorporate your backdoor. Someone or several people out there must be pissed off.
- kuchenbecker 3y agoYour implying this is their first Rodeo and don't have others in various stages.
- deleted 3y ago[deleted]
- yc4mnfo 3y ago[dead]
- memp646 3y ago[dead]
- 1vuio0pswjnm7 3y ago"OpenSSH, the most popular sshd implementation, doesn't link the liblzma library, but Debian and many other Linux distributions add a patch to link sshd to systemd, a program that loads a variety of services during the system bootup. Systemd, in turn, links to liblzma, and this allows xz Utils to exert control over sshd." Compare with: "Xz is an open-source compression program, as well as a library that can be used to help you write your own program that deals with compressed data. It is used by a fairly large number of other programs, one of which is OpenSSH." https://news.ycombinator.com/item?id=39881049 https://news.ycombinator.com/item?id=39881049 GNU's binutils links to liblzma. binutils is even more ubiquitous than OpenSSH; in most cases it's probably used in the compilation of OpenSSH, the operating systems on which sshd runs, and so on. The bad guys certainly picked a good project to potentially get deep into open source software.
- jbritton 3y agoDoes anyone know if this has been reported to the FBI and Homeland Security? Or can one just assume they will become aware?
- hackeraccount 3y agoThat gets at a question I've wondered about. Is there a crime here? I mean it doesn't seem like the actor broke into a system. What law was broken?
- browser1 3y ago[dead]
- cyberturtle2 3y ago[dead]