6 ms·
Ask HN: How bad is the xz hack?
I've been avidly reading all I can about it. I understand how it came about, and my sympathy is firmly with the maintainer. I know research is on going as I type this, but is anyone with the expertise I don't have able to explain in simple terms how bad it is, or could have been?
- ekimehtor 3y agoIt is still pretty early but yeah, I suppose you could look at it as “-1 day” vulnerability… or a back door which goes back to my comment about it being still pretty early. Who added this code? And who is auditing it?
- pajko 3y ago20 on a scale of 10. It's a remote code execution backdoor against the whole world... How bad can it be?
- dewey 3y ago“Whole world” don’t you think that’s a bit overly dramatic?
- pajko 3y agoNope, given that it has a plugin architecture and in time it would have been everywhere. https://github.com/facebook/folly/commit/b1391e1c57be71c1e2a330a46d124e96686d57e9 https://github.com/facebook/folly/commit/b1391e1c57be71c1e2a...
- dralley 3y agoYes, the only mitigating factor here is that it was discovered before reaching any major distributions
- zilti 3y agoThe most popular distributions are rolling-release ones, so not sure about that.
- k8sToGo 3y agoOn servers? I am sure that’s not true. And I doubt most people run ssh daemon on desktop. And if they do they aren’t exposed to the internet I hope.
- gv83 3y agoI’ve never seen a server with a rolling distro tbh
- doubled112 3y agoI’ve seen them in production at a smaller shop. They do exist. Kept things exciting.
- hoffs 3y agoMore like 1 to 5%, I doubt any higher percentage bothers to update
- bdhcuidbebe 3y agoIt seems it was discovered in the last minute, close to ending up in ubuntu 24.04 LTS and derivates. Luckily most users was unaffected unless they used a rolling release distro. Finally, we still dont have a full analysis so its not known if systemd-enabled sshd is the only attack vector.
- aaronrobinson 3y agoThis is just a small number of committers and one vuln but by its execution suggests there’s more to be discovered and what’s now on those machines this vuln could execute upon and what’s it done to machines on those networks.
- tambourine_man 3y agoAnd what us regular folks who run Ubuntu LTS on VMs need to do, if anything.
- cpach 3y agoFirst: Apply all official patches via APT. That’s the most important thing. The easiest way to avoid getting pwned is to apply security patches without too long delay. Then consider defense-in-depth. If you have sshd exposed to the Internet, you could put it behind a VPN such as Tailscale (or pure WireGuard). And FWIW: With our current knowledge, any LTS version of Ubuntu should not have been affected by this backdoor.
- politelemon 3y agoAs far as I understand it, you need to check for the version of xz/liblzma. Run this: dpkg -l | grep liblzma # or xz --version If you see 5.6.0 or 5.6.1, you should panic, and probably recreate. But it is unlikely you will see that especially on LTS as they are using an older version. On Ubuntu 22.04 I get version "5.2.5" Ubuntu 24.04 was about to be affected but they removed that version when the news broke.
- wood_spirit 3y agoI too have been trying to find out about the hack. There has been a lot of interesting analysis of the code, which has a feeling of the awe and sophistication of stuxnet yet on the scale of “this could be the work of a single motivated coder … like one of us”. But what I haven’t found yet, and would like leads on, is what ther thinking is about the attack in a wider context… What is the current thinking on: 1. Was the malicious code inserted by the new contributor, or was this likely a compromised account? 2. If the new contributor did it, did they become an xz maintainer with the aim of acting maliciously? 3. If the new contributor was malicious, why take so long to attack and why do it in xz? Or are they also attacking lots of other systems, perhaps under other names? 4. And what can we do about these attacks? How can we build a system that isn’t ultimately brought crashing down by one or two bad actors? 5. Small technical detail I haven’t spotted in writeups: was the attack commands signed in some way so only the attacker can use it and the world can go hunting for a smoking gun cert that matches the attacker?
- randmeerkat 3y ago> If the new contributor did it, did they become an xz maintainer with the aim of acting maliciously? Is no one asking if there was a state actor behind the contributor? My question is what was the ultimate end goal, and who motivated this exploit?
- wood_spirit 3y agoI think it is just as plausible that a state actor casts about for an account of a committer they can trivially compromise who is on one of the thousands of tired projects that can slip this kind of attack in…
- bgmeister 3y ago> why take so long to attack First they need to gain some credibility. And they also need to obfuscate what they were trying to do. The payoff if they were to succeed is potentially enormous, so it makes sense to do things slowly. > and why do it in xz? xz is almost a perfect target- undermaintained, complicated code that isn't well understood by many people, deployed and used widely. > Or are they also attacking lots of other systems, perhaps under other names? The answer probably depends who the attacker(s) are, and what sort of funding or other motifivations they have. We can only guess. > 4. And what can we do about these attacks? This is a big unanswered question. I haven't seen any great suggestions yet. We were very lucky with this particular attack. > How can we build a system that isn’t ultimately brought crashing down by one or two bad actors? The complexity involved in this attack suggests that it was probably a group, IMO. > 5. Small technical detail I haven’t spotted in writeups: was the attack commands signed in some way so only the attacker can use it and the world can go hunting for a smoking gun cert that matches the attacker? Yes, sort of. Filippo's thread has a good description: https://bsky.app/profile/filippo.abyssdomain.expert/post/3kowjkx2njy2b https://bsky.app/profile/filippo.abyssdomain.expert/post/3ko...
- supposemaybe 3y agoI feel total sympathy for the maintainer if they 100% didn’t know what was going on. It’s also unfortunate that the maintainer will be associated with the lack of judgement. But they will get through it. Probably time to pass the torch fully now though.
- TheBigSalad 3y agowasn't trying to pass the torch the problem?
- lyu07282 3y agoI think the really scary part is the particular modus operandi that this demonstrates, that is easily repeatable. A malicious actor builds up a reputation in the open source space and eventually takes ownership of the release process at which point a backdoor is inserted. There is absolutely nothing that would suggest this hasn't happened before or will not happen again.
- zh3 3y ago...or forces a well-respected contributor to add evil code under duress.
- _factor 3y agoImagine a state actor offering retirement money and a plane ride to a non-extradition island nation for a few commits. Assuming the contributor is using a VPN already, no one would know they were adding surreptitious backdoors in their code. The only solution is to scope security relevant functions and add a multi stage formal verification process before freezing the commits. This assumes no collusion.
- kijin 3y agoIn practice, it's a non-issue for anyone using a stable version of a major Linux distro, such as Ubuntu 22.04 LTS or RHEL/Alma/Rocky 9. The liblzma in those distros were cut and frozen long before our suspect began to have serious involvement in its development. (Not sure about Amazon Linux. Arch doesn't seem to be affected.) The more worrying question is, what else was compromised by these kinds of people, and for how long?
- opengears 3y agoI could have been much worse, but was identified last minute. Debian, Ubuntu and Fedora had the code only in their testing channels, and overall only .deb or .rpm based distros with systemd have been affected (see https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b... ). We have to make sure open source maintainers are not underpaid, and stop the madness of global commons exploitation.
- rubymamis 3y agoFrom a different comment of mine: > It baffles me how such an important package that so many Linux servers use every day is unmaintained by the original author due to insufficient funds. Something gotta change in OSS. I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.
- aeim 3y agoand governments
- aborsy 3y agoand users, who could pay a monthly fee to GitHub that will distribute among maintainers. The donation system doesn’t work well. There ought to be streams of input funds.
- iJohnDoe 3y agoThis is a great idea and Github should look into this. Not easy to implement (which maintainers qualify for funds, etc) but should be looked into.
- deleted 3y ago[deleted]
- appplication 3y agoThe problem is once you change the license and start charging, someone forks it and just offers a less funded, but free version of essentially the same, and the ecosystem fractures. See Redis, Terraform for recent examples. Not exactly the same, but in this context it’s funny how much ire they have drawn when really the root of it is “how do we ensure continuing to develop this software is financially supported by users”.
- junon 3y agoThis is probably the worst backdoor in history in the open source sphere, if not ever. Certainly the worst of my career.
- deleted 3y ago[deleted]
- diffeomorphism 3y agoWorst as in "bad" or worst as in "did not have any impact as a backdoor"?
- formerly_proven 3y agoBoth? This is eternal blue, Linux edition, except it was never shipped.
- junon 3y agoBad. And we don't know if the intended target, if any, was ever hit. They might still have gotten what they wanted. In terms of severity and nefarious intent, it's probably the worst that's happened.
- kator 3y agoMy favorite was sendmail's WIZ followed by SHELL bug.
- ransom1538 3y agotl;dr. SSH though VPN only. Sleep tight.
- deleted 3y ago[deleted]
- iforgotpassword 3y agoDoes anyone know if distros using the systemd-notify patch are considering not linking to libsystemd and instead implementing the notify-protocol themselves? As this would only be like 10 lines of code, instead of linking a huge library that will in turn pull in another bunch of libs, it seems like it should reduce attack surface.
- dralley 3y agoYes, either that or splitting it out into a "libsdnotify" But since it's only a few lines of code most seem to think that unnecessary. It should be noted that systemd actually fixed this a few days before the backdoor was pushed https://github.com/systemd/systemd/pull/31550#issuecomment-1972737923 https://github.com/systemd/systemd/pull/31550#issuecomment-1...
- iforgotpassword 3y agoGreat, I was trying to figure this out for Debian but I get PTSD every time I end up on any of the sites related to a package because they all look different and I can never figure out where I'd even find what I'm looking for (i.e. issue tracker, mailing list). :-) I'm not so sure what to think about the dlopen patch though. It obviously fixes this attack vector, but in general it makes it harder to discover dependencies in a simple way. I wish PE-style late binding would exist for ELF.
- _ink_ 3y agoWhat I find curious is, everybody is acting like, yeah, we found that one attempt! But there gotta be more. And this is only open source. At Google, Meta, Microsoft there gotta be the same forces at play. So yeah, congrats to everyone involved discovering this, excellent work, really! But how sure can we be, that there was no other successful attempt?
- Lacerda69 3y agoI think we should assume there as at least one or more other successful attempts.
- cpach 3y agoWe can never be sure about that. One’s threat model has to assume that Mallory can find a way in. Backdoor’s are not super common, but there will always be 0-days. Security operations need to be setup with this in mind. Design your infra for defence-in-depth. Logging and access control is very important. Log everything that happens in your systems. Don’t use long-lived credentials. Don’t give people more access than they need. Audit users’ privileges, remove privileges that are no longer required. Try to make your logs tamper-proof. (Perhaps keep copies of the logs offline?)
- _ink_ 3y agoThanks, that's all good advice.
- 1letterunixname 3y agoYep. Assume one or more layers are pwned and rootkitted, especially if it runs Linux because the attack surface is huge in most distros. Never fall for the braggadocio of arrogant, lazy MAANG employees who blindly declare the infra to be "secure" without evidence. Where possible, run a transparent proxy/scrubbing/NIDS/NIPS network layer running OpenBSD or similar. Log setup using WORM media, log structured fs, and/or blockchain would be ideal.
- jpsouth 3y agoSomeone else posted this in another thread and I found it a really interesting read. It doesn’t really answer your question directly, but it is very related. https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
- TNWin 3y agoHow does this affect Mac users who had the vulnerability?
- lifeinthevoid 3y agoBrew was installing the affected version 5.6.1 of `xz`. They have in meantime rolled back the version to 5.4.6 There's not enough information yet to know for sure what the effect is / was on systems running the backdoored version.
- fossdd 3y agoThe backdoor had a check to only affect .dem and .rpm based distros. macOS /should/ be fine.
- elric 3y agoNot as bad as the fallout is going to be. Already some people are calling for mandatory identification when contributing to open source projects. This will, of course, have a chilling effect on FOSS contributions, while it seems unlikely (to me) that this would improve security in any meaningful manner.
- chgs 3y agoThe next step will be no contributions from people from countries like China, Russia, North Korea Won’t stop state actors of course, just normal people (well ok there are only state actors from NK. And exiles)
- funciton 3y agoWho's going to be on the hook for ID verification? Ah, right... Just a thought, making open source maintainers do more unpaid work is probably not the answer to the problem of overworked open source maintainers.
- ThinkBeat 3y agoThe real question should be how many more similar things there are in the entire Linux codebase I would bet good money there are more that haven't been detected. Every intelligence service in the world wants one or more. Many can hire truly bright programmers who will contribute to Linux over years, doing nothing nefarious at all, to build a trustworthy profile. and even contribute meaningful and positive features. Then, maybe 2 of 3 trusted profiles make tiny changes as part of a much larger valid contribution. each one a tiny adjustment, each one not particularly interesting from a security point of view, unless you look at all 3 at the same time, and of course they were not added at the same time. I am quite certain they have people who are working at Apple and Microsoft as well with the same mission.
- jacknews 3y agoIt sounds like an outrageous conspiracy theory, but I'm also quite sure you are right, this is how these intelligence services operate.
- cpach 3y agoIndeed. For another wild example, see https://en.wikipedia.org/wiki/Crypto_AG https://en.wikipedia.org/wiki/Crypto_AG
- appplication 3y agoIt’s outrageous because it’s basically unprovable, though we have at least one data point the MO exists to some degree. It would be naive to think it’s not attempted at least. How does one go about becoming a corporate plant? If the CIA offered me $1M to inject a back door into some shenanigans corporate product and told me it was being used to target Bashar al-Assad or cripple Russia’s war effort, would I do it? I don’t know honestly. Principles say it’s not right, but I wouldn’t blame someone (or myself) for saying yes. Of course, being a data guy I’m probably not at risk of such a decision, since no one is injecting backdoors into corporate analytics. Dare to dream of a shadowy government payout I suppose.
- 3y ago
- 8191 3y agoIt was somehow astonishing to me that the code which is actually shipped (tarball) is not the very same code as the one in the repo. Somehow renders code reviews useless. Seems to be a common practice anyway.
- afavour 3y agoIt was caught just in time so the “badness” in terms of “what do I need to do to mitigate this right now” is thankfully pretty low. But IMO the real “bad” here is that it calls into question the entire model of volunteer open source contribution, which underpins a crazy amount of the tech world. No easy answers there.
- curious_curios 3y agoMakes me wonder if there’s going to be a push for KYC (know your committer) in the open source world.
- account-5 3y agoSo, if say, I'm using MX Linux (semi-rolling release) based on debain stable, using sysVinit instead of systemd, I should be ok related to this issue?
- paulmd 3y agoperfect 10 on severity, probably close 10 on impact. Caught before it was widely exploited (it got noticed because it just rolled out, in an attempt to get it into the release window for 24.04 etc) but it’s essentially RCE on all Debian, Ubuntu, and fedora openssh servers.
- geenat 3y agoSimilar exploit vector in Swoole (binary blobs ran under the guise of "testing") and exactly why it was forked for Open Swoole: https://old.reddit.com/r/PHP/comments/1b9acx3/article_intro_to_swoole_and_async_php/ktxfboi/ https://old.reddit.com/r/PHP/comments/1b9acx3/article_intro_... Thankfully this only effects Swoole users, not all of PHP.
- cyberturtle2 3y ago[dead]