9 ms·

Inside the failed attempt to backdoor SSH globally that got caught by chance

221 points by transpute 3y ago

throwaway984393 3y ago
[dead]
noobermin 3y ago
What an uninspiring take, there is literally nothing we can do, the status quo is fine, the abuse of unpaid devs of open source is a harder problem to solve than commercial space flight to Mars. It's fine to state this is difficult to solve but why be so defeatist about it?
- onion2k 3y ago
  It's much easy to do anything if you're honest about the reality you're working in.
vladms 3y ago
It might be worth it to analyze a bit more. Like, what would be different if the software would be closed source and developers payed by companies? I think it would be at least as hard to notice such an exploit and sometimes it might be easier (if the company is located in your jurisdiction). Maybe the current mindset of assembling programs could be improved. There is a trend in some architecture to separate everything in their own container and while I don't think it can be directly applied everywhere that model gives more separation for cases like this. Engineering is an art of trade-offs and maybe now we can afford making different trade-offs than 30 years ago (when some things where decided)
- dakiol 3y ago
  Does any private Microsoft/Google/Apple/whatever program have any backdoors? We don’t know and we will never know. At least with open source we are able to detect them.
  - microtonal 3y ago
    I don't think this is necessarily true. People do a lot of reverse engineering of proprietary OSes and a lot of vulnerabilities are found that way (besides fuzzing). And the tooling for reverse engineering is only getting better. Also, let's not forget that this particular backdoor was initially found through behavioral analysis, not by reading the source code. I think Linus' law "given enough eyeballs, all bugs are shallow" has been refuted. Discovering bugs does not scale linearly with eyeballs, you need the right eyeballs. And the right eyeballs are often costly. If your implicit premise that having the source code available makes it easier to analyze code than closed source, you can also flip the argument around: it is easier for bad actors to find exploitable vulnerabilities because the source code is available. (Note: I am a staunch supported of FLOSS, but for different reasons, such as empowerment and freedom.)
  - shzhdbi09gv8ioi 3y ago
    Yes, and they are eventually discovered by reverse engineering. Example: https://en.wikipedia.org/wiki/NSAKEY https://en.wikipedia.org/wiki/NSAKEY
- beagle3 3y ago
  … and everything old is new again. DJB’s qmail, written in 1995, was made of 5 distinct processes, owned by different users, with no trust among them. Coincidentally, it was the MTA with the best security record for more than a decade (and also the most efficient one). It would have likely had a similar record even if it was only a monolithic process - because DJB - but it was built as 5 processes so even if one falls, the others do not.
  - patchtopic 3y ago
    it was also essentially unusable without a crapload of third party patches that DJB would not include into the master release, but yes it was quite secure :-)
    - Anarch157a 3y ago
      And it was highly vulnerable to denial of service attacks. It didn't check if the mailbox was valid during the envelope phase, so it would queue basicaly everything, then check the mailbox and send a bounce if necessary. Sending thousands of messages to random boxes (dictionary spam attack) would queue thousands of bounce messages that would be rejected by the (faked) sender domain, bringing the Qmail server to it's knees. As me how I know this... Thing is, in most companies, is cheaper and more efficient to deal with a sporadic vulnerability than having your e-mail system DOSed every other week. This is the kind of compromises that normal people and companies have to do all the time, but radicals and cryptopunks like DJB can't seem to understand. Sure, he's a brilliant mathematician and cryptographer, but his grasp of reality outside academia seems very flimsy, IMO.
      - beagle3 3y ago
        My qmail setup in 2000, on a humble beige box, was occasionally under a “thousands of bad addresses” attack, but I only found out about it a few days later while reviewing the logs. There surely was a threshold where it would be down on its knees - but “thousands” and even “tens of thousands” wasn’t it. The exchange server it replaced, though, would crash and burn very often, for a variety of reasons.
- discreteevent 3y ago
  Google's Fuchsia OS looks promising but it doesn't look like (or something like it) will go anywhere until the world accepts that you probably have to build security into the fabric of everything. You can't just patch it on.
  - microtonal 3y ago
    I think Google has given up on Fuchsia, outside some specific domains, right? I think currently even Android, iOS, and ChromeOS have far better security than most desktop and server OSes. I think of the widely-used general purpose OSes, only macOS comes fairly close because it adopted a lot of things from iOS (fully verified boot, sealed system partition, app sandboxing for App Store apps, protected document/download/mail/... directories, etc.).
    - udev4096 3y ago
      QubesOS is the closest we have, in terms of better security model, in the desktop area
- ta1243 3y ago
  Companies are infiltrated all the time [0] However the incentives even if a company detected the infiltration is to keep quiet about it. Lets say that a closed source "accessd" was backdoored, a database admin notices the new version (accessd 2024.6 SAAS version model 2.0+ with GPT!) is slower than the previous version, they put it down to enshittificaiton. Or they contact the company who has no incentive to spend money to look into it. There's no way the database admin (or whoever) can look into the git commit chain and see where the problem happened. [0] https://rigor-mortis.nmrc.org/@simplenomad/112184869681420177 https://rigor-mortis.nmrc.org/@simplenomad/11218486968142017...
- pxeger1 3y ago
  There isn’t much closed source software which is depended on as heavily as things like xz. The only one I can think of is Windows, which I think it’s safe to assume is definitely backdoored.
denton-scratch 3y ago
> libsystemd is linked to all systemd servies, which opens a rich attack surface of third party services to backdoor. Many people have warned that systemd does way too much for an init system, and presents way too much attack surface. Cluck, cluck (chickens coming home to roost).
- himinlomax 3y ago
  The backdoored library was linked to libsystemd, not systemd itself.
  - blkhawk 3y ago
    "the car bomb was build into the spare wheel not the car itself...."
    - nottorp 3y ago
      "... and although the spare wheel is included by default, the car is modular and you can always remove it yourself" [By dismantling the car completely and reassembling it, since you'd have to rebuild from source...]
      - chgs 3y ago
        Most cars no longer come with a spare wheel, certainly by default. Not sure if that’s because cars don’t get punctures any more due to the great state of the roads here in the U.K, or because people just can’t even change a wheel.
        nottorp 3y ago
        Not in most systemd distros though. Those include all kinds of spare wheels that automatically take over your actual wheels.
        JonChesterfield 3y ago
        It's to help meet emissions targets according to the garage I bought from. The spare wheel weighs more than not having one.
ChrisMarshallNY 3y ago
> the world owes Andres unlimited free beer. He just saved everybody’s arse in his spare time. I am not joking. Agreed. > and face significant amounts of online abuse from users of the software. Much as I’d like this to change, I suspect it will not. I’ve been doing open-source work since the mid-90s, and have suffered abuse that would curl your toes, but I still do it, for reasons that would be obscure, for many folks, hereabout. I think the onus is on the users of the software. It’s currently way too easy to make “LEGO block” software, by splicing together components, written by others, that we do not understand (which, in itself, is not necessarily bad. We’ve been working this way for centuries). I remember reading a report that listed YAML as one of the top programming languages. If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. Since it’s currently possible to make huge buckets of money, by writing crap, we’re not incentivizing high-Quality engineering. I don’t think there’s any easy answer, as a lot of the most effective solutions are cultural, as opposed to technical. Yesterday, someone posted a part of an old speech, by Hyman Rickover[0], that speaks to this. [0] https://news.ycombinator.com/item?id=39889072 https://news.ycombinator.com/item?id=39889072
- microtonal 3y ago
  If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. This, each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded. Of course, it will always be hard to detect very sophisticated attacks from actors that can play the long game. But if maintenance is not just a hobby, PRs can receive more attention, and if it's not an added workload besides a day job, there is less churn/burn-out which can be used for malicious actors to take over a project.
  - bradley13 3y ago
    "...each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded." Agreed, 100%. And yet: In a comment on Slashdot one guy said that his company had "thousands" of machines running Linux, and he was "proud" to have never paid a penny for it. I called him a parasite: open-source brings an immense value to his company, and he should support it. My comment got a lot of hate, which I just don't understand. Sure, OSS licenses say you can do whatever you want. However, there is surely some ethical obligation to support something that your entire business depends on?
anthk 3y ago
>OpenSSH runs... Not every OpenSSH build it's linked against xz. OpenBSD's one isn't.
- udev4096 3y ago
  Correct. OpenSSH doesn't care about linux or systemd. Makes me want to switch to BSD now
  - usr1106 3y ago
    Arch Linux uses systemd, but does not patch sshd, so its not vulnerable to the known backdoor either. However, to my knowledge nobody has fully analyzed the malware binary yet. So we don't know whether it only contains a single mechanism to attack sshd or whether it also has other harmful components.
  - jmclnx 3y ago
    Well there is always Slackware, no systemd and unpatched sshd. win-win :) And correct, this was not an issue for Slackware Current (which has xz 5.6.1). Slackware 15 has xz 5.2.5
xorcist 3y ago
The point about owing Andres unlimited beer is a good one. He should get something good out of this!
technion 3y ago
I can confirm the statement about the security industry. I just opened LinkedIn and my feed, along with two spam dms, are companies trying to imply their product had always detected this.
feverzsj 3y ago
Lots of arch users sensed the unusual slowness of ssh, but less would have gone that far because of naive trust of the community.
- JellyYelly 3y ago
  sshd in arch isn't built with the systemd patch, and thus experienced no slowdown. You can read about this on the arch linux blog: https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore...
- shzhdbi09gv8ioi 3y ago
  arch was unaffected as far as we know.
  - deleted 3y ago
    [deleted]
  - udev4096 3y ago
    Just the one attack vector that we know of. There could be many, which are yet to be discovered
    - delfinom 3y ago
      Eh? People have dumped strings and more from the attack payload. It's exclusively targeting openssh. However they left the door open to future payloads by simply dropping new blobs into the test cases folder.
      - BuildTheRobots 3y ago
        The XZ attack is (seemingly) exclusively targeting openssh. Considering it's a library that isn't even a direct dependency of ssh and someone was willing to put in over two years of effort, it doesn't seem crazy to start wondering what other (small) projects have been suffering maintainer fatigue or are even just big enough where a "lgtm" response is almost guaranteed by a sane looking PR. "Smuggle in unexpected code/blobs as test files", "bastardise make scripts to inject your code into the build" and then "have your code preferentially override [decryption] routines" aren't just (as of last week) unexpected, but would also seemingly be usable against any other project out there. I'd suggest it's more like finding and killing a cockroach in your bathroom. Thinking that the roach problem is now gone for good and was only confined to the one room seems... optimistic at best. I also think it's fun to wildly speculate on what else could be affected - or to play the game another way, "if I had control over sshd, what else would I want?" From an outside point of view, I'd quite like web servers. They're often externally accessible when ssh isn't, are doing cert validation/compression/decryption and I'd wager on most boxes, could easily give you an ssl wrapped tunnel to (the vulnerable) sshd running on the servers loopback address without selinux blocking you. Actually, having written that, has anyone looked at the SSH _client_ process rather than server? Surely it needs to be able to parse certs in the same way as the server does, and getting clients to leak their private keys somewhere seems like it would be extremely useful as well.
starspangled 3y ago
It's a damning indictment of the computer security industry, IMO. They push wildly complicated and performance-costly bandaids for vanishingly unlikely exploits for dubious benefit, and pay very little attention to holes like this you can drive a truck through. https://xkcd.com/538/ https://xkcd.com/538/ is very fitting. The fact a database hacker found it, and it missed all these supposed scanners and auditors and security experts is the icing on the cake. But even if it was one of them that found it, it's already in the wild. The technical programming tools and techniques for security are important don't get me wrong, the security industry just seems to have very little concept of cost-benefit, the big picture of how attackers and their targets operate and what motivates them. They seem to mostly exist to justify their own existence. I've already seen "security experts" wheel out the big scary (and meaningless) term "nation state" over this, which makes me laugh. Sure it could have been the CCP. It could also have just been some bored kid in his parents basement, that just wouldn't look good to admit though.
- _8j50 3y ago
  I don't agree with you fully. Look at solarwinds for example,or many other supply chain attacks, they were discovered much later after succesful abuse. The public availability of the software helped catch the attack much faster than commercial software. Even if there was intensive scrutiny of changes to the project, a person familiar with the process can still come up with hard to detect backdoors. Future backdoors may be stealthier but what this case demonstrated is that even a database hacker who doesn't do security audits could catch it simply because it's opensource. The expectation should be, such backdoors would be detected many months or years after the fact if they were your typical closed source popular application. This is a case for open source software usage and funding. The security industry can't do much in terms of prevention against a malicious insider that knows the codebase more than any outsider. And opensource or not, people can get paid or implanted to sabotage software.
  - vouwfietsman 3y ago
    I imagine though that in this case the attacker has a much simpler time staying anonymous, vs commercial software using paid and vetted employees. Or am I missing something about how this was contributed?
rspoerri 3y ago
I wonder how commercial software would compare in such a situation. Assuming you have a bad actor among the employees or a breach in the security. Not a lot of eyes watching that sourcecode i'd assume.
- tonnydourado 3y ago
  Yeah, less eyes on it means it would be less likely to be caught, but there's also much more upfront validation on who gets to change it. Although, I can imagine a sub-sub-contractor identity being somewhat easily forged, it's still harder than just creating a GitHub account.
  - wakeupcall 3y ago
    > but there's also much more upfront validation on who gets to change it. Very doubtful in my eyes. Very few companies have the strict validation which would be required to catch this. Good validation that goes into central components is often skimped on arbitrary deadlines, which companies are full of. On something tangential like this I suspect nobody would really notice. I can also hear the "I noticed now there's an extra delay which isn't supposed to be there, can I investigate?" "Sorry, but this is not critical for this deadline" agile mentality.
    - tonnydourado 3y ago
      Not sure we're talking about the same thing. By validation I mean of identity. On proprietary software , before an attacker get inside access to the code, they would have to interview, get hired, submit (presumably, fake) document id, provide (again, presumably, fake) bank details, etc. This attacker just had a GitHub account and email, as far as understood it. But, as I said, maybe tricking a sub contracting company into hiring you is not as hard. I remember working with contractors whose faces I've never seem on video, let alone in person.
      - wakeupcall 3y ago
        I didn't think of "identity" in this sense, but I don't see this as a show-stopper either. On my current jig developer churn is not high, yet I've only recently met developers hired 6+ months ago. I know first-hand only a handful of the committers I see. Barely know the most common commiters. I generally do watch commits of the trees/projects I'm interested into, but I'm a minority, and such behavior wouldn't catch something similar to the xz situation unless I'm absolutely lucky. This also ignores the fact that you can just as well corrupt a current employee.
otabdeveloper4 3y ago
> Secondly, a core issue here is systemd in Linux. libsystemd is linked to all systemd services, which opens a rich attack surface of third party services to backdoor. I guarantee you more services link to glibc, and that glibc has a much, much larger attack surface than anything systemd.
- IshKebab 3y ago
  Also it's not like this attack would have been impossible if systemd hadn't done that. As soon as anything loaded xz it could pretty much do what it wanted. Slightly more of a pain but not difficult.
topbanana 3y ago
Makes me wonder how many backdoors like this are in the wild, undetected
- farmdve 3y ago
  I would imagine quite a few. It would be prudent to start paying attention to the tools that were created to catch some of the issues(like how people ignore Valgrind errors) as well as commits made to OSS that disables some feature, because it breaks a 3rd party library. >> odd valgrind complaint in automated testing of postgres This implies these kinds of complaints are routinely ignored.
tonnydourado 3y ago
Gonna have a serious talk with my mother about not trying to hack into opensource software that powers most of the world's software. I know retirement is boring, but that's no excuse.
skywhopper 3y ago
Some good observations here, including about the apparent acceleration of effort and ensuing sloppiness due to impending changes to systemd that would have prevented the particular attack vector. Unfortunately, the sarcastic tone starts to become a barrier to separating signal from noise about halfway through. Okay, you’re super clever, the NSA is a threat, too, we get it. Security vendors are largely hucksters, fine. What were we talking about again?
Maxion 3y ago
So, uh, who did do it?
- jwilk 3y ago
  https://news.ycombinator.com/item?id=39877604 https://news.ycombinator.com/item?id=39877604
toomuchtodo 3y ago
How does one buy Andres a beer?
udev4096 3y ago
All in all, am I not vulnerable if I'm running an init system other than systemd?
- ta1243 3y ago
  No, you are likely still vulnerable if you have this versio of this library and a typical sshd instance, I believe it's about the linking rather than the init system.
  - goku12 3y ago
    Remember that the vulnerability was introduced by distributions patching sshd to talk to systemd. It doesn't make sense for a distribution to patch sshd like this unless they use systemd. Thus, choosing another init system may save you indirectly, even though you're right that this is really about the linking. Ultimately, the attacker deemed systemd to be common enough to consider it as the way to hijack sshd, but indirect enough to avoid discovery by audit. It says something about software monoculture.
throwaway290232 3y ago
[dead]
Fnoord 3y ago
This backdoor would've been caught eventually because the added latency is substantial. It wouldn't have occurred on Arch Linux, BSD, macOS, any Solaris derivative OS (Illumos etc). We should be grateful it got caught so quickly. I sent Andres a honest thank you email. It isn't financial (and I am just one individual) but it felt the least I could do. If there is a way to donate, I would. This person could've earned more via HackerOne or black market and instead went with an arguably better path. I don't think we can compete with the latter though, unless we start treating this for what it is: a weapon.
- hashstring 3y ago
  I am grateful too, but I don’t think that this person could have earned more via blackmarkets. This backdoor in ssh was NOBUS. You cannot exploit it unless you have the private key.
rbultje 3y ago
For me, the critical take-home message is: "the issues around tens of thousands of unpaid developers writing the core of the world’s most critical infrastructure nowadays." We need to take opensource software sustainability much more seriously.
xyzelement 3y ago
I just opened LinkedIn and saw a post from Satya Nadela congratulating Andres. Clicked over Andre’s profile and saw he works at Microsoft. I am a bit surprised that no article or HN post about this prevented hack mentions this fun fact. (Yes we should still buy this guy a coffee but he’s also a principle engineer at MS)
- lenerdenator 3y ago
  It's one of the stranger turns in computing history that Microsoft has turned into one of the most reliable security presences for skullduggery like this.
  - red-iron-pine 3y ago
    given how much of the IT landscape they own, they damn well better be security experts. or how much damage they`ve done in the past to said landscape due to poor security.
mihalycsaba 3y ago
Quick, someone link that xkcd where a crucial open source technology is being maintained by basically a single person, while billion dollar companies rely on it.
deleted 3y ago
[deleted]
MaxBarraclough 3y ago
> We should stop using open source and only buy American vendor products! Yeah, good luck with that. This isn't a good take. This presents proprietary software as offering superior assurances against backdoors, if only it were practical to completely avoid Free and Open Source software. The truth is approximately the opposite: code being Free and Open Source should be considered a necessary condition, but not a sufficient one, for resisting backdoors. > We should start funding every open source project! Yeah, good luck with that.. I’ll start saving for my trip to Mars. This kind of defeatism is unhelpful.
- gpvos 3y ago
  > This presents proprietary software as offering superior assurances against backdoors In the image below that remark the author makes clear that proprietary software is often subverted.
lenerdenator 3y ago
> Let’s just keep doing the good SBOM work at CISA, and stop doing stunts around Huawei and such — Huawei is a speck of dust compared to the issues around tens of thousands of unpaid developers writing the core of the world’s most critical infrastructure nowadays. I have to disagree with this. There seems to be this weird mindset in tech that because there's problem X (the Five Eyes countries hacking each others' citizens at each others' request, Meta collecting data on users, a massive attack using xz that almost got into the wild, etc.) that China isn't a problem. It's this strange "our house isn't in order because of our own doing, so it doesn't matter if some dude off the street starts squatting in it" idea. If you have a country that has a tech legacy mainly related to espionage and attacks on other countries' systems - and make no mistake, that's China's main legacy - don't buy their stuff, no matter how many times it's said that it's fine. At some point it won't be fine. You can fix that and better secure FOSS projects; it's not one or the other.
irrational 3y ago
> It’s also important we keep things in perspective, e.g. this one was spotted and averted. My worry is, how many have not been spotted and have not been averted? From my reading of this one, it was blind luck that it was spotted.
deleted 3y ago
[deleted]
jauntywundrkind 3y ago
Looking at how Jia75T got access to projects, it was vacuous sock puppets creating fake pressure. https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo... It feels like we should have better socialized defense against this kind of thing. When I see a name pop up online, I want an immediate readout on how active this person is over time and where. We need better tools to get high level views of who we are dealing with.

221 points by transpute 3y ago

[dead]

What an uninspiring take, there is literally nothing we can do, the status quo is fine, the abuse of unpaid devs of open source is a harder problem to solve than commercial space flight to Mars. It's fine to state this is difficult to solve but why be so defeatist about it?

onion2k 3y ago
It's much easy to do anything if you're honest about the reality you're working in.

vladms 3y ago

It might be worth it to analyze a bit more. Like, what would be different if the software would be closed source and developers payed by companies? I think it would be at least as hard to notice such an exploit and sometimes it might be easier (if the company is located in your jurisdiction). Maybe the current mindset of assembling programs could be improved. There is a trend in some architecture to separate everything in their own container and while I don't think it can be directly applied everywhere that model gives more separation for cases like this. Engineering is an art of trade-offs and maybe now we can afford making different trade-offs than 30 years ago (when some things where decided)

dakiol 3y ago
Does any private Microsoft/Google/Apple/whatever program have any backdoors? We don’t know and we will never know. At least with open source we are able to detect them.
- microtonal 3y ago
  I don't think this is necessarily true. People do a lot of reverse engineering of proprietary OSes and a lot of vulnerabilities are found that way (besides fuzzing). And the tooling for reverse engineering is only getting better. Also, let's not forget that this particular backdoor was initially found through behavioral analysis, not by reading the source code. I think Linus' law "given enough eyeballs, all bugs are shallow" has been refuted. Discovering bugs does not scale linearly with eyeballs, you need the right eyeballs. And the right eyeballs are often costly. If your implicit premise that having the source code available makes it easier to analyze code than closed source, you can also flip the argument around: it is easier for bad actors to find exploitable vulnerabilities because the source code is available. (Note: I am a staunch supported of FLOSS, but for different reasons, such as empowerment and freedom.)
- shzhdbi09gv8ioi 3y ago
  Yes, and they are eventually discovered by reverse engineering. Example: https://en.wikipedia.org/wiki/NSAKEY https://en.wikipedia.org/wiki/NSAKEY
beagle3 3y ago
… and everything old is new again. DJB’s qmail, written in 1995, was made of 5 distinct processes, owned by different users, with no trust among them. Coincidentally, it was the MTA with the best security record for more than a decade (and also the most efficient one). It would have likely had a similar record even if it was only a monolithic process - because DJB - but it was built as 5 processes so even if one falls, the others do not.
- patchtopic 3y ago
  it was also essentially unusable without a crapload of third party patches that DJB would not include into the master release, but yes it was quite secure :-)
  - Anarch157a 3y ago
    And it was highly vulnerable to denial of service attacks. It didn't check if the mailbox was valid during the envelope phase, so it would queue basicaly everything, then check the mailbox and send a bounce if necessary. Sending thousands of messages to random boxes (dictionary spam attack) would queue thousands of bounce messages that would be rejected by the (faked) sender domain, bringing the Qmail server to it's knees. As me how I know this... Thing is, in most companies, is cheaper and more efficient to deal with a sporadic vulnerability than having your e-mail system DOSed every other week. This is the kind of compromises that normal people and companies have to do all the time, but radicals and cryptopunks like DJB can't seem to understand. Sure, he's a brilliant mathematician and cryptographer, but his grasp of reality outside academia seems very flimsy, IMO.
    - beagle3 3y ago
      My qmail setup in 2000, on a humble beige box, was occasionally under a “thousands of bad addresses” attack, but I only found out about it a few days later while reviewing the logs. There surely was a threshold where it would be down on its knees - but “thousands” and even “tens of thousands” wasn’t it. The exchange server it replaced, though, would crash and burn very often, for a variety of reasons.
discreteevent 3y ago
Google's Fuchsia OS looks promising but it doesn't look like (or something like it) will go anywhere until the world accepts that you probably have to build security into the fabric of everything. You can't just patch it on.
- microtonal 3y ago
  I think Google has given up on Fuchsia, outside some specific domains, right? I think currently even Android, iOS, and ChromeOS have far better security than most desktop and server OSes. I think of the widely-used general purpose OSes, only macOS comes fairly close because it adopted a lot of things from iOS (fully verified boot, sealed system partition, app sandboxing for App Store apps, protected document/download/mail/... directories, etc.).
  - udev4096 3y ago
    QubesOS is the closest we have, in terms of better security model, in the desktop area
ta1243 3y ago
Companies are infiltrated all the time [0] However the incentives even if a company detected the infiltration is to keep quiet about it. Lets say that a closed source "accessd" was backdoored, a database admin notices the new version (accessd 2024.6 SAAS version model 2.0+ with GPT!) is slower than the previous version, they put it down to enshittificaiton. Or they contact the company who has no incentive to spend money to look into it. There's no way the database admin (or whoever) can look into the git commit chain and see where the problem happened. [0] https://rigor-mortis.nmrc.org/@simplenomad/112184869681420177 https://rigor-mortis.nmrc.org/@simplenomad/11218486968142017...
pxeger1 3y ago
There isn’t much closed source software which is depended on as heavily as things like xz. The only one I can think of is Windows, which I think it’s safe to assume is definitely backdoored.

denton-scratch 3y ago

> libsystemd is linked to all systemd servies, which opens a rich attack surface of third party services to backdoor. Many people have warned that systemd does way too much for an init system, and presents way too much attack surface. Cluck, cluck (chickens coming home to roost).

himinlomax 3y ago
The backdoored library was linked to libsystemd, not systemd itself.
- blkhawk 3y ago
  "the car bomb was build into the spare wheel not the car itself...."
  - nottorp 3y ago
    "... and although the spare wheel is included by default, the car is modular and you can always remove it yourself" [By dismantling the car completely and reassembling it, since you'd have to rebuild from source...]
    - chgs 3y ago
      Most cars no longer come with a spare wheel, certainly by default. Not sure if that’s because cars don’t get punctures any more due to the great state of the roads here in the U.K, or because people just can’t even change a wheel.
      - nottorp 3y ago
        Not in most systemd distros though. Those include all kinds of spare wheels that automatically take over your actual wheels.
      - JonChesterfield 3y ago
        It's to help meet emissions targets according to the garage I bought from. The spare wheel weighs more than not having one.

ChrisMarshallNY 3y ago

> the world owes Andres unlimited free beer. He just saved everybody’s arse in his spare time. I am not joking. Agreed. > and face significant amounts of online abuse from users of the software. Much as I’d like this to change, I suspect it will not. I’ve been doing open-source work since the mid-90s, and have suffered abuse that would curl your toes, but I still do it, for reasons that would be obscure, for many folks, hereabout. I think the onus is on the users of the software. It’s currently way too easy to make “LEGO block” software, by splicing together components, written by others, that we do not understand (which, in itself, is not necessarily bad. We’ve been working this way for centuries). I remember reading a report that listed YAML as one of the top programming languages. If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. Since it’s currently possible to make huge buckets of money, by writing crap, we’re not incentivizing high-Quality engineering. I don’t think there’s any easy answer, as a lot of the most effective solutions are cultural, as opposed to technical. Yesterday, someone posted a part of an old speech, by Hyman Rickover[0], that speaks to this. [0] https://news.ycombinator.com/item?id=39889072 https://news.ycombinator.com/item?id=39889072

microtonal 3y ago
If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. This, each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded. Of course, it will always be hard to detect very sophisticated attacks from actors that can play the long game. But if maintenance is not just a hobby, PRs can receive more attention, and if it's not an added workload besides a day job, there is less churn/burn-out which can be used for malicious actors to take over a project.
- bradley13 3y ago
  "...each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded." Agreed, 100%. And yet: In a comment on Slashdot one guy said that his company had "thousands" of machines running Linux, and he was "proud" to have never paid a penny for it. I called him a parasite: open-source brings an immense value to his company, and he should support it. My comment got a lot of hate, which I just don't understand. Sure, OSS licenses say you can do whatever you want. However, there is surely some ethical obligation to support something that your entire business depends on?

anthk 3y ago

>OpenSSH runs... Not every OpenSSH build it's linked against xz. OpenBSD's one isn't.

udev4096 3y ago
Correct. OpenSSH doesn't care about linux or systemd. Makes me want to switch to BSD now
- usr1106 3y ago
  Arch Linux uses systemd, but does not patch sshd, so its not vulnerable to the known backdoor either. However, to my knowledge nobody has fully analyzed the malware binary yet. So we don't know whether it only contains a single mechanism to attack sshd or whether it also has other harmful components.
- jmclnx 3y ago
  Well there is always Slackware, no systemd and unpatched sshd. win-win :) And correct, this was not an issue for Slackware Current (which has xz 5.6.1). Slackware 15 has xz 5.2.5

xorcist 3y ago

The point about owing Andres unlimited beer is a good one. He should get something good out of this!

technion 3y ago

I can confirm the statement about the security industry. I just opened LinkedIn and my feed, along with two spam dms, are companies trying to imply their product had always detected this.

feverzsj 3y ago

Lots of arch users sensed the unusual slowness of ssh, but less would have gone that far because of naive trust of the community.

JellyYelly 3y ago
sshd in arch isn't built with the systemd patch, and thus experienced no slowdown. You can read about this on the arch linux blog: https://archlinux.org/news/the-xz-package-has-been-backdoored/ https://archlinux.org/news/the-xz-package-has-been-backdoore...
shzhdbi09gv8ioi 3y ago
arch was unaffected as far as we know.
- deleted 3y ago
  [deleted]
- udev4096 3y ago
  Just the one attack vector that we know of. There could be many, which are yet to be discovered
  - delfinom 3y ago
    Eh? People have dumped strings and more from the attack payload. It's exclusively targeting openssh. However they left the door open to future payloads by simply dropping new blobs into the test cases folder.
    - BuildTheRobots 3y ago
      The XZ attack is (seemingly) exclusively targeting openssh. Considering it's a library that isn't even a direct dependency of ssh and someone was willing to put in over two years of effort, it doesn't seem crazy to start wondering what other (small) projects have been suffering maintainer fatigue or are even just big enough where a "lgtm" response is almost guaranteed by a sane looking PR. "Smuggle in unexpected code/blobs as test files", "bastardise make scripts to inject your code into the build" and then "have your code preferentially override [decryption] routines" aren't just (as of last week) unexpected, but would also seemingly be usable against any other project out there. I'd suggest it's more like finding and killing a cockroach in your bathroom. Thinking that the roach problem is now gone for good and was only confined to the one room seems... optimistic at best. I also think it's fun to wildly speculate on what else could be affected - or to play the game another way, "if I had control over sshd, what else would I want?" From an outside point of view, I'd quite like web servers. They're often externally accessible when ssh isn't, are doing cert validation/compression/decryption and I'd wager on most boxes, could easily give you an ssl wrapped tunnel to (the vulnerable) sshd running on the servers loopback address without selinux blocking you. Actually, having written that, has anyone looked at the SSH _client_ process rather than server? Surely it needs to be able to parse certs in the same way as the server does, and getting clients to leak their private keys somewhere seems like it would be extremely useful as well.

starspangled 3y ago

It's a damning indictment of the computer security industry, IMO. They push wildly complicated and performance-costly bandaids for vanishingly unlikely exploits for dubious benefit, and pay very little attention to holes like this you can drive a truck through. https://xkcd.com/538/ https://xkcd.com/538/ is very fitting. The fact a database hacker found it, and it missed all these supposed scanners and auditors and security experts is the icing on the cake. But even if it was one of them that found it, it's already in the wild. The technical programming tools and techniques for security are important don't get me wrong, the security industry just seems to have very little concept of cost-benefit, the big picture of how attackers and their targets operate and what motivates them. They seem to mostly exist to justify their own existence. I've already seen "security experts" wheel out the big scary (and meaningless) term "nation state" over this, which makes me laugh. Sure it could have been the CCP. It could also have just been some bored kid in his parents basement, that just wouldn't look good to admit though.

_8j50 3y ago
I don't agree with you fully. Look at solarwinds for example,or many other supply chain attacks, they were discovered much later after succesful abuse. The public availability of the software helped catch the attack much faster than commercial software. Even if there was intensive scrutiny of changes to the project, a person familiar with the process can still come up with hard to detect backdoors. Future backdoors may be stealthier but what this case demonstrated is that even a database hacker who doesn't do security audits could catch it simply because it's opensource. The expectation should be, such backdoors would be detected many months or years after the fact if they were your typical closed source popular application. This is a case for open source software usage and funding. The security industry can't do much in terms of prevention against a malicious insider that knows the codebase more than any outsider. And opensource or not, people can get paid or implanted to sabotage software.
- vouwfietsman 3y ago
  I imagine though that in this case the attacker has a much simpler time staying anonymous, vs commercial software using paid and vetted employees. Or am I missing something about how this was contributed?

rspoerri 3y ago

I wonder how commercial software would compare in such a situation. Assuming you have a bad actor among the employees or a breach in the security. Not a lot of eyes watching that sourcecode i'd assume.

tonnydourado 3y ago
Yeah, less eyes on it means it would be less likely to be caught, but there's also much more upfront validation on who gets to change it. Although, I can imagine a sub-sub-contractor identity being somewhat easily forged, it's still harder than just creating a GitHub account.
- wakeupcall 3y ago
  > but there's also much more upfront validation on who gets to change it. Very doubtful in my eyes. Very few companies have the strict validation which would be required to catch this. Good validation that goes into central components is often skimped on arbitrary deadlines, which companies are full of. On something tangential like this I suspect nobody would really notice. I can also hear the "I noticed now there's an extra delay which isn't supposed to be there, can I investigate?" "Sorry, but this is not critical for this deadline" agile mentality.
  - tonnydourado 3y ago
    Not sure we're talking about the same thing. By validation I mean of identity. On proprietary software , before an attacker get inside access to the code, they would have to interview, get hired, submit (presumably, fake) document id, provide (again, presumably, fake) bank details, etc. This attacker just had a GitHub account and email, as far as understood it. But, as I said, maybe tricking a sub contracting company into hiring you is not as hard. I remember working with contractors whose faces I've never seem on video, let alone in person.
    - wakeupcall 3y ago
      I didn't think of "identity" in this sense, but I don't see this as a show-stopper either. On my current jig developer churn is not high, yet I've only recently met developers hired 6+ months ago. I know first-hand only a handful of the committers I see. Barely know the most common commiters. I generally do watch commits of the trees/projects I'm interested into, but I'm a minority, and such behavior wouldn't catch something similar to the xz situation unless I'm absolutely lucky. This also ignores the fact that you can just as well corrupt a current employee.

otabdeveloper4 3y ago

> Secondly, a core issue here is systemd in Linux. libsystemd is linked to all systemd services, which opens a rich attack surface of third party services to backdoor. I guarantee you more services link to glibc, and that glibc has a much, much larger attack surface than anything systemd.

IshKebab 3y ago
Also it's not like this attack would have been impossible if systemd hadn't done that. As soon as anything loaded xz it could pretty much do what it wanted. Slightly more of a pain but not difficult.

topbanana 3y ago

Makes me wonder how many backdoors like this are in the wild, undetected

farmdve 3y ago
I would imagine quite a few. It would be prudent to start paying attention to the tools that were created to catch some of the issues(like how people ignore Valgrind errors) as well as commits made to OSS that disables some feature, because it breaks a 3rd party library. >> odd valgrind complaint in automated testing of postgres This implies these kinds of complaints are routinely ignored.

tonnydourado 3y ago

Gonna have a serious talk with my mother about not trying to hack into opensource software that powers most of the world's software. I know retirement is boring, but that's no excuse.

skywhopper 3y ago

Some good observations here, including about the apparent acceleration of effort and ensuing sloppiness due to impending changes to systemd that would have prevented the particular attack vector. Unfortunately, the sarcastic tone starts to become a barrier to separating signal from noise about halfway through. Okay, you’re super clever, the NSA is a threat, too, we get it. Security vendors are largely hucksters, fine. What were we talking about again?

Maxion 3y ago

So, uh, who did do it?

jwilk 3y ago
https://news.ycombinator.com/item?id=39877604 https://news.ycombinator.com/item?id=39877604

toomuchtodo 3y ago

How does one buy Andres a beer?

udev4096 3y ago

All in all, am I not vulnerable if I'm running an init system other than systemd?

ta1243 3y ago
No, you are likely still vulnerable if you have this versio of this library and a typical sshd instance, I believe it's about the linking rather than the init system.
- goku12 3y ago
  Remember that the vulnerability was introduced by distributions patching sshd to talk to systemd. It doesn't make sense for a distribution to patch sshd like this unless they use systemd. Thus, choosing another init system may save you indirectly, even though you're right that this is really about the linking. Ultimately, the attacker deemed systemd to be common enough to consider it as the way to hijack sshd, but indirect enough to avoid discovery by audit. It says something about software monoculture.

throwaway290232 3y ago

[dead]

Fnoord 3y ago

This backdoor would've been caught eventually because the added latency is substantial. It wouldn't have occurred on Arch Linux, BSD, macOS, any Solaris derivative OS (Illumos etc). We should be grateful it got caught so quickly. I sent Andres a honest thank you email. It isn't financial (and I am just one individual) but it felt the least I could do. If there is a way to donate, I would. This person could've earned more via HackerOne or black market and instead went with an arguably better path. I don't think we can compete with the latter though, unless we start treating this for what it is: a weapon.

hashstring 3y ago
I am grateful too, but I don’t think that this person could have earned more via blackmarkets. This backdoor in ssh was NOBUS. You cannot exploit it unless you have the private key.

rbultje 3y ago

For me, the critical take-home message is: "the issues around tens of thousands of unpaid developers writing the core of the world’s most critical infrastructure nowadays." We need to take opensource software sustainability much more seriously.

xyzelement 3y ago

I just opened LinkedIn and saw a post from Satya Nadela congratulating Andres. Clicked over Andre’s profile and saw he works at Microsoft. I am a bit surprised that no article or HN post about this prevented hack mentions this fun fact. (Yes we should still buy this guy a coffee but he’s also a principle engineer at MS)

lenerdenator 3y ago
It's one of the stranger turns in computing history that Microsoft has turned into one of the most reliable security presences for skullduggery like this.
- red-iron-pine 3y ago
  given how much of the IT landscape they own, they damn well better be security experts. or how much damage they`ve done in the past to said landscape due to poor security.

mihalycsaba 3y ago

Quick, someone link that xkcd where a crucial open source technology is being maintained by basically a single person, while billion dollar companies rely on it.

deleted 3y ago

[deleted]

MaxBarraclough 3y ago

> We should stop using open source and only buy American vendor products! Yeah, good luck with that. This isn't a good take. This presents proprietary software as offering superior assurances against backdoors, if only it were practical to completely avoid Free and Open Source software. The truth is approximately the opposite: code being Free and Open Source should be considered a necessary condition, but not a sufficient one, for resisting backdoors. > We should start funding every open source project! Yeah, good luck with that.. I’ll start saving for my trip to Mars. This kind of defeatism is unhelpful.

gpvos 3y ago
> This presents proprietary software as offering superior assurances against backdoors In the image below that remark the author makes clear that proprietary software is often subverted.

lenerdenator 3y ago

> Let’s just keep doing the good SBOM work at CISA, and stop doing stunts around Huawei and such — Huawei is a speck of dust compared to the issues around tens of thousands of unpaid developers writing the core of the world’s most critical infrastructure nowadays. I have to disagree with this. There seems to be this weird mindset in tech that because there's problem X (the Five Eyes countries hacking each others' citizens at each others' request, Meta collecting data on users, a massive attack using xz that almost got into the wild, etc.) that China isn't a problem. It's this strange "our house isn't in order because of our own doing, so it doesn't matter if some dude off the street starts squatting in it" idea. If you have a country that has a tech legacy mainly related to espionage and attacks on other countries' systems - and make no mistake, that's China's main legacy - don't buy their stuff, no matter how many times it's said that it's fine. At some point it won't be fine. You can fix that and better secure FOSS projects; it's not one or the other.

irrational 3y ago

> It’s also important we keep things in perspective, e.g. this one was spotted and averted. My worry is, how many have not been spotted and have not been averted? From my reading of this one, it was blind luck that it was spotted.

deleted 3y ago

[deleted]

jauntywundrkind 3y ago

Looking at how Jia75T got access to projects, it was vacuous sock puppets creating fake pressure. https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo... It feels like we should have better socialized defense against this kind of thing. When I see a name pop up online, I want an immediate readout on how active this person is over time and where. We need better tools to get high level views of who we are dealing with.