7 ms·
NixOS and reproducible builds could have detected the xz backdoor
- IshKebab 2y agoYeah it certainly would have made hiding the backdoor more difficult. But far from impossible. You can always hide backdoors in source code if you want, it just takes more effort to make a plausible bug, and probably has a higher chance of detection.
- ltbarcly3 2y agoYes, if you use a trusted framework then you are safe from things until that framework is attacked. The xz backdoor might have been detected, but the xz backdoor wasn't crafted with the goal of working against the Nix ecosystem. When a nix core developer ends up being a spy or whatever then there will end up being an attack against the nix ecosystem. Don't reply to this with some claim that Nix is inherently secure unless you want me to track you down and make you admit you were wrong when Nix ends up getting successfully exploited in a year or two.
- vlovich123 2y agoThe standard never has been and never will be absolute security. That’s an impossible threshold nothing would ever meet even though it’s objectively true that software today is generally more secure than software 30 years ago. The strongman claim being made is “Nix is harder and more expensive to exploit than traditional build systems”. So sure, if you find a cheap way to exploit Nix, track me down. But until then, it remains at least plausible & in practice very likely that Nix is harder to exploit than alternate systems on a technical level.
- xvector 2y ago> But until then, it remains at least plausible & in practice very likely that Nix is harder to exploit than alternate systems on a technical level. Do people even read package derivations? Feels like it'd be easy to check in a derivation with an exploit.
- sirspamalot112 2y ago[dead]
- sirspamalot111 2y ago[flagged]
- cocoto 2y agoThe backdoor was not targeting Nix but it had to not raise any suspicion during a build in Nix to not be exposed.
- dzaima 2y agoThe backdoor build script specifically checked for things indicating that it's being built for debian, and if not, not inserting the backdoor; so it only ever was non-reproducible in situations where reproducibility wasn't expected. Not hard to make sure a backdoor with control over the build environment doesn't raise suspicions in non-targeted places.
- immibis 2y agoalso wasn't the backdoor reproducible to begin with? If it had targeted every system, you'd just get reproducible backdoored binaries.
- dzaima 2y agoThe infected version was only the tarball, which was part of the obfuscation (i.e. people may look at git commits, but who individually checks autogenerated code in tarballs of every release) Building from the git commit the release claimed to be from would result in a different binary than building from the tarball if the environment check passed.
- lolinder 2y agoNote that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS): > I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users. As always theory and reality are different, and the thing that made xz possible was never a technical vulnerability with a technical solution—xz was possible because of a meatspace exploit. We as a community are very very bad at recognizing that you can't always just patch meatspace with better software.
- jjmarr 2y agoI still like the blogpost, because NixOS bills itself as a technical solution to prevent build artifacts that are decoupled from the source code (i.e. not reproducible), and the xz backdoor was hidden in build artifacts.
- lolinder 2y agoYeah, it's a good blog post in part because it gets into the details of how it was possible that this vulnerability made it into NixOS, which purports to solve the problem. Also, I'm not a NixOS critic either: I'm writing this from NixOS! I just don't think there's such a thing as a security cure-all as long as humans are in the loop anywhere.
- crote 2y agoSure, but you could achieve the same thing by requiring that the build artifacts are generated by a Github Actions runner.
- tomjakubowski 2y agoThat would only make the xz attack harder, not make it impossible. Just add some step to the action run that fetches blobs from an internet resource you control, then swap out the blobs for malicious ones.
- deleted 2y ago
- lotharcable 2y agoSo the argument hinges on the fact that the XZ maintainer hid malicious code in the tarballs that were not checked into Git. The author demonstrates that Nix can be configured to generate the tarballs from git that go into building the binaries. What I don't see, however, is how is this a feature that requires Nix or NixOS? Any build system out there (including the stuff that goes into RPMs and Debs) can be configured to generate tarballs as a intermediate step. In fact making reproducible builds is a major thing that Debian has been working on for some time now. https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
- Smaug123 2y agoThe article does in fact cite the reproducible-builds project, in the section on "Leveraging bitwise reproducibility". From your comment I am not convinced you understood the point of the article, which is: * the NixOS build process was unable to perform a full-source build of xz because xz is required too early in the bootstrap; * a proposed adjustment to nixpkgs to automatically detect compromises of nixpkgs dependencies which are required early in the bootstrap. Other ecosystems can of course also attempt full-source builds and discover the discrepancy; the entire point of the article is that nixpkgs currently cannot.
- lotharcable 2y agoI see.
- sirspamalot103 2y ago[dead]
- sirspamalot106 2y ago[flagged]
- sirspamalot112 2y ago[dead]
- rowanG077 2y agoit's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.
- microtonal 2y agoOr they have N other such projects in flight. The xz backdoor wasn’t that much work, just playing the long game. The person doing the xz project could easily do several other projects at the same time. A lot of issues do get undetected. E.g. the Debian OpenSSL security accident was only detected when a gazillion servers had predictable SSH keys.
- rowanG077 2y agoMost likely yes. That doesn't really impact the funny factor for me.
- sirspamalot113 2y ago[flagged]
- immibis 2y agoAFAIK Debian OpenSSL was detected when two people had the same GitHub key and since GitHub identifies people by their key it conflicted.
- Dwedit 2y agoYeah, without the latency regression, it probably would have gone undetected much longer. Using a secondary thread and spreading the CPU load over a few seconds would have made it not even register as a spike in CPU usage.
- immibis 2y agoOr do cheap ECDSA instead of expensive RSA. Even if the backdoor is hidden inside RSA decryption and the rest of the system thinks the thing being decrypted should be encrypted with RSA, you don't have to use it for the back door.
- datadeft 2y agocould have / should have => being smart retrospectively
- mcint 2y agoExcellent descriptive analysis. Wrong, misleading title, perhaps "technically correct," but at best with a "backdoored" meaning. It points out the need and use for build-manager tools that go a step beyond union file system layers, but track then enforce that e.g. tests cannot pollute build artifacts. Take a causal trace graph of files affecting files, in the build process, make that trace graph explicit, and then build a way to enforce that graph, or report on deviations from previous trace graphs.
- sirspamalot110 2y ago[dead]
- drpossum 2y ago[flagged]
- brabel 2y agoIn defense of the author: nobody reads your article if the name is boring (that is my experience at least), which it would've been if they titled it more accurately. That gives incentive to authors to use click-bait titles.
- account42 2y agoIn defense of the bank robber: no clerk simply gives you money if you aren't threatening them (that is my experience at least), which it would have been if they acted like a respectable citizen. This gives people the incentive to become bank robbers.
- brabel 2y agoFirst of all: an exagerated title is in no way compared to threatening someone's life. Secondly: your comparison does not even make sense: "which it would have been" what?? Try harder next time.
- nialv7 2y agoI feel the author is a bit tunnel visioned by what happens to happen this time. The Jiatan incident has a sample size of one, it'd be a bit short sighted to think that's the only way it could happen. You can imagine various scenarios where the defenses suggested here will not have worked. Also I (as a nix user myself) think it's unlikely NixOS would have caught it. As evidenced by the fact that it didn't. (Yeah I realize I just said next time it might happen differently but it'd be foolish to put faith in nix without evidence).
- sirspamalot104 2y ago[flagged]
- sirspamalot106 2y ago[flagged]
- sirspamalot110 2y ago[dead]
- sirspamalot111 2y ago[flagged]
- sirspamalot113 2y ago[flagged]
- MortyWaves 2y agoI don't like how this site causes my headphones to crackle...
- sirspamalot113 2y ago[flagged]
- donnachangstein 2y agoNixOS is really irrelevant here because the xz backdoor specifically targeted RedHat and Debian. It's equally relevant to say the xz backdoor didn't affect Windows (ironically the backdoor was ultimately found by a Microsoft employee, an oft-overlooked detail).
- sirspamalot104 2y ago[flagged]
- sirspamalot103 2y ago[dead]
- sirspamalot106 2y ago[dead]
- sirspamalot110 2y ago[dead]
- sirspamalot111 2y ago[flagged]
- attila-lendvai 2y agoa slightly improved version of NixOS (or Guix) would have automatically caught this backdoor once it reached their repos. they are relevant.
- brabel 2y agoA slightly improved version of any OS would've automatically caught it.
- attila-lendvai 2y agomechanism, or it didn't happen. and if any OS, then for windoze please!
- massysett 2y agoArticle says that distributions should get source code directly from the VCS (for instance Github) rather than the traditional installation tarball. I don’t see what this solves though. Couldn’t a malicious maintainer simply add binary blobs directly to the source code repository? The author suggests Github is trusted, as though Github validates code in some way. Which of course it does not.
- sirspamalot104 2y ago[flagged]
- dwheeler 2y agoWhat this solves is the problem that often "what is reviewed" is different from "the source code used to build the software". Verified reproducible builds could have countered the xz utils break, SolarWinds Orion subversion, and many others. It's worth doing.
- sirspamalot106 2y ago[flagged]
- sirspamalot110 2y ago[dead]
- massysett 2y agoBut would that have solved anything here? The main maintainer was overwhelmed. The back door was obfuscated inside a binary blob there for so-called testing purposes. I doubt anyone was reviewing the binary blobs or the autoconf code used to load it in, and for that matter it’s not clear anything was getting reviewed. Fetching and building straight from GitHub doesn’t solve that if the malicious actor simply puts the binary blob into the repo.
- dzaima 2y agoMight not be a big chance depending on the project in question, but it's still tons more likely for someone randomly clicking through commits to find a backdoor committed to a git repo than within autogenerated text in a tarball. I click around random commits of random projects I'm interested in every now and then at least. At the very least it changes the attack from "yeah noone's discovering this code change" to "let's hope no random weirdo happens to click into this commit". A binary blob by itself is harmless, you need something to copy it over from the build env to the final binary. So it's "safe" to ignore binary blobs that you're sure that the build system (which should all be human-written human-readable, and a small portion of the total code in sane projects) never touches. That said, of course, there's still many options for problematicness - some projects have commit autogenerated code; bootstrapping can bring in a much larger surface area of things that might copy the binary blob; and more.
- a-dub 2y agollm commit scanning might be an interesting approach to the oss supply chain security problem.
- sirspamalot106 2y ago[dead]
- __MatrixMan__ 2y agoMaybe, but good luck getting an LLM (one which does not include analysis of this particular attack in its training data) to spot this attack with a prompt that doesn't also create thousands of false positives when focused on the millions of non-malicious commits out there. I think we're decades away from them being that good.
- yellowapple 2y agoCommit scanning probably wouldn't have caught this, since the backdoor happened outside of any commit. Comparing the tarball's contents against the VCS repository would've likely made this easier to catch, but at that point you might as well just use the VCS repository directly.
- a-dub 2y agoi wonder if an llm could have spotted the malicious patches to autotools in the dist tarball... a public, deterministic and assured build facility for oss would be cool. also maybe a deprecation of autotools.
- sirspamalot101 2y ago[flagged]
- sirspamalot102 2y ago[flagged]
- sirspamalot103 2y ago[dead]
- sirspamalot104 2y ago[flagged]
- sirspamalot105 2y ago[flagged]
- sirspamalot106 2y ago[dead]
- sirspamalot111 2y ago[flagged]
- sirspamalot107 2y ago[flagged]
- sirspamalot108 2y ago[flagged]
- sirspamalot110 2y ago[dead]
- __MatrixMan__ 2y agoIf we want to focus on a thing that NixOS could have prevented, we should focus on the CrowdStrike incident. Being able to boot to yesterday's config because today's config isn't working would've mitigated most of the problems.
- sirspamalot113 2y ago[flagged]
- XorNot 2y agoExcept that's a Windows thing where you don't have boot flexibility. Ubuntu on ZFS can do this as well.
- __MatrixMan__ 2y agoMy point is that the lack of boot flexibility caused a lot of problems. If we want to be able to rely on people to get the job done even on days when something is wonky in the bits, then we should give them boot flexibility. NixOS just happens to do it especially well. As for ZFS... Dealing in filesystem snapshots is comparatively a bit awkward. If you want to recreate that config elsewhere you have to move the whole snapshot rather than just the recipe for building it, and even then it'll break if the system architecture is different on the target machine. If you've got two of them (perhaps labeled "good" and "bad"), you're not going to get anything friendly when you try to diff them, nor is there an obvious way to use things like `git bisect` to reason about where the problem occurred. None of these things are show stoppers, but working with code that defines some state is just so much easier than anything you get out of a filesystem which happens to remember that state, but can't tell you why it should be the way it is.
- chungy 2y ago> As for ZFS... Dealing in filesystem snapshots is comparatively a bit awkward. If you want to recreate that config elsewhere you have to move the whole snapshot rather than just the recipe for building it That wasn't the highlight of the point. It was that you can restore to a known good version of the operating system, effortlessly, regardless of what the operating system is. It could be Ubuntu, Nix, or FreeBSD. Broken OS, select an older snapshot in the boot loader, and you're golden again.
- sirspamalot112 2y ago[dead]
- sirspamalot113 2y ago[flagged]
- dataflow 2y agoWhy is nobody questioning this: > To build xz from sources, we need autoconf to generate the configure script. But autoconf has a dependency on xz! Both directions of this seem crazy to me. 1. Why the heck should a build configuration tool like autoconf be unable to function without a compression tool like xz? That makes no sense on its face. 2. For that matter, why the heck should xz, a tool that is supposedly so fundamental, have a hard dependency on a boilerplate generator like autoconf? At the end of the day all autoconf is doing is telling you how to invoke your compiler. You ought to have a way to do that without the tool, even if it produces a suboptimal binary. If you care about security, instead of taking a giant tarball you don't understand and the running another tool in it, shouldn't you just generate that command line somehow (even in an untrusted fashion), review it, and then use that human-verified script to bootstrap? And if you need a (de)compressor that low on the dependency tree so that literally the entire world might one day rest on it, surely you can isolate the actual computation for bootstrapping purposes and just expose it with just the open/read/write/close syscalls as dependencies? Why do you need all the bells and whistles?
- skywhopper 2y ago[flagged]
- yellowapple 2y agoIf you know of a reason for autoconf and xz to be hard dependencies of one another, you can just say it.
- donnachangstein 2y ago> Why the heck should a build configuration tool like autoconf be unable to function without a compression tool like xz? That makes no sense on its face. At face value, both autoconf and its cousin pkg-config are overly complex dogshit software - both with circular dependencies - that should have been retired long ago in favor of something else. I scream with joy when I use software that uses its own bootstrapper or cmake. Before you think "but I've never had this problem, you must be bonkers" - try building software on a fresh Solaris box with no GNU anything installed and you need to install one of these monstrosities with their circular dependencies. Your hair will fall out before you're done.
- banana_dick_1 2y ago[flagged]
- banana_dick_10 2y ago[dead]
- banana_dick_14 2y ago[dead]
- wwarner 2y agoLearned a lot reading this article!
- Jedd 2y agoIs the massive number of spam messages on this thread an attempt to suppress the article / discussion around it? I've not seen this many from multiple but evidently related green accounts before. Given the implications about nation state actors in play, it's tempting to jump to conclusions here.
- defrost 2y agoIt's multiple threads, not just this one, and it's been on and off the past few days. You can add these kind of lines: news.ycombinator.com##tr.athing.comtr:has(a.hnuser:has-text('banana_dick')) news.ycombinator.com##tr.athing.comtr:has(a.hnuser:has-text('sirspamalot')) to any uBlock or AdBlockPlus type extensions with manual compatible custom filters that you might have added to your browser. > It's quite amazing that these new "green" accounts can post as many messages as they want while I'm being restricted That's not what is happening .. look at the account names and the number of comments made by each.
- TacticalCoder 2y ago[dead]
- yellowapple 2y agoStill leaves me curious about the "why". Someone mad at HN about something? Just having a laugh?
- kibwen 2y agoNo, this has been happening for a few days now. Browse threads from the past week with showdead enabled and you'll see them.
- nebulous1 2y ago> showdead enabled Thanks, I didn't know this was something we could turn off. Although it does feel like they could just default to collapsed too.
- TacticalCoder 2y ago[dead]
- delusional 2y agoWhat's with all the baseless claims about NixOS lately? Is Sam Altman invested in it or something?
- gbraad 2y ago"Could have" means unproven here, and actually... They shipped it
- attila-lendvai 2y ago? the article demonstrates how to imorove NixOS (or Guix) to automatically catch any such discrepancy in the future.
- ggggrah 2y agoI think it's a vuln to think of OSS in terms of 'a community'. It's an abstract thought construct that does not represent reality (though it helps to make sense of it in a rather specific manner) xz happened because of the absence of community. It could happen inside this abstract thought of a community as well but here it did not. xz targeted deb and rpm. The vast majority of what is facing the world. Nix did not stop it. I believe this article feeds the possible vuln rather than prevent it.
- udev4096 2y agohttps://news.ycombinator.com/item?id=41486565 https://news.ycombinator.com/item?id=41486565 -> possible targets due to high impact and low maintainers
- thdhhghgbhy 2y agoIf Jia Tan's PR was approved, malicious artifacts could go to github releases just as easily as in a tarball. Struggling to understand the point made about github releases being a security mitigation.
- dsp_person 2y agoWhen I did my first github "release" I was shocked that it involved picking files to upload...
- mikrotikker 2y agoAnd how many of these files have I downloaded over the years? :sob:
- lima 2y agoThe real answer here is downloading a tarball generated from a tag - those are generated by the GitHub backend.
- drpossum 2y ago[flagged]
- microtonal 2y agoThere are plenty of Nix users who are skeptical of such claims (e.g. me included). I generally believe that NixOS is less secure than some other systems. No secure boot by default, no SELinux, too few maintainers for a huge package set, and relatively easy to gain commit access. But it does have a lot of other benefits.
- pjmlp 2y agoThis kind of argument is like how Gentoo used to be better than anything else, because everyone knows how little piece is built, and here we are two decades later, who still is wasting nights compiling everything.
- johnisgood 2y ago> the release tarball being different than the source is > the maintainer provided tarball was honestly generated from the original source code. How, then? What about differing versions, etc. or has it been mentioned and I just missed it? Just make sure the generated tarball can be generated from the source code itself, do not exclude anything, git add & commit everything. Can't we do that? We would still have to look at commit history in this case, I believe, and again, he said it himself, it was harmless to the naked eye, so even then, how could we verify? Maybe I don't understand what he meant by verification, but if maintained tarballs are generated from the owner's source code and is not on GitHub (or anywhere else, just a git repo), that is a problem in itself. Of course there was more to it than just pushing poisoned test files, but still. I do not see how Nix would have prevented it, if the git repo has those test files and with seemingly harmless code (and is reproducible). Perhaps what we can do is: if an (in)famous project has changed its main lead, then pay closer attention to the commits and check who it is? I don't know, TBH. Did I misunderstand the article, or am I missing something?
- gsliepen 2y agoWhile NixOS goes a bit further with it, most other distrubtions also compile everything from source, cryptographically verify that the sources they use are not tampered with, and have versioned dependencies between packages. Debian also has reproducible builds. The problem is just that the build systems did not strip pre-compiled object files before building from source. Even with that fixed, if nobody checks the source code then you can add all the backdoors you want, and there is nothing in NixOS or any other distro that would protect against that.
- attila-lendvai 2y agothere's a disappointing amount of firm conviction here by people who are clueless about NixOS and/or the xz attack...
- account42 2y agoStill better than vague rants that add nothing concrete to the discussion.
- deleted 2y ago[deleted]
- bdhcuidbebe 2y agoThis article suggests that Nix could have prevented xz backdoor, only to conclude that backdoor could be avoided by building from a git tag rather than source tarball. This is true for every distro and it grinds me that Nix is even mentioned.