26 ms·

This really deserves more love. Who remembers Ken Thompson's "Reflections on Trusting Trust"? The norm today is auto-updating, pre-built software. This place

This really deserves more love. Who remembers Ken Thompson's "Reflections on Trusting Trust"? The norm today is auto-updating, pre-built software. This places a ton of trust in the publisher. Even for open-source, well-vetted software, we all collectively cross our fingers and hope that whoever is building these binaries and running the servers that disseminate them, is honest and good at security. So far this has mostly worked out due to altruism (for open source maintainers) and self interest (companies do not want to attack their own users). But the failure modes are very serious. I predict that everyone's imagination on this topic will expand once there's a big enough incident in the news. Say some package manager gets compromised, nobody finds out, and 6mo later every computer on earth running `postgres:latest` from docker hub gets ransomwared. There are only two ways around this: - Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds. - Reproducible builds. Reproducible builds are way more important than is currently widely appreciated. I'm grateful to the nixos team for being beating a trail thru the jungle here. Retrofitting reproducibility onto a big software project that grew without it, is hard work.

zamadatix 5y ago
Unless you are going to be the equivalent of a full time maintainer doing code review for every piece of software you use you need to trust other software maintainers reproducible builds or not. Considering this is Linux and not even Linus can deeply review every change in just the kernel anymore that philosophy can't apply to meaningfully large software like Nixos.
- jnxx 5y ago
  That's too black-and-white. Being able to reproduce stuff makes some kind of attacks entirely uninteresting because malicious changes can be traced back. Which is what many types of attackers do not want. Debian, or the Linux kernel, for example, are not fool-proof, but both are in practice quite safe to work with.
  - zamadatix 5y ago
    Who are you going to trace it back to if not the maintainer anyways? If the delivery method then why is the delivery of the source from the maintainer inherently any safer?
    - jnxx 5y ago
      No, it is not always the maintainer. Imagine you download a binary software package via HTTPS. In theory, the integrity of the download is protected by the server certificate. However, it is possible that certificates get hacked, get stolen, or that nation states force CAs to give out back doors. In that case, your download could have been changed on the fly with arbitrary alterations. Reproducible builds make it possible to detect such changes.
      - zamadatix 5y ago
        Same as when you download the source instead of the binary and see it reproducibly builds the backdoored binary. And at this point we're back to "Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds." anyways. It's not that reproducible builds provide 0 value it's that they don't truly solve the trust problem as initially stated. They also have non-security value to boot which is often understated compared to the security value IMO.
        bigiain 5y ago
        I guess reproducible builds solve some of the problems in the same way TLS/SSL solves some of the problems. Most of the world is happy enough with the soft guarantee of: “This is _probably_ your bank’s real website. Unless a nation state is misusing their control over state owned certificate authorities, or GlobalSign or LetsEncrypt or whoever has been p0wned.” Expecting binary black and white solutions to trust problems isn’t all that useful, in my opinion. Often providing 50% more “value” in trust compared to the status quo is extremely valuable in the bigger picture.
- radicalcentrist 5y ago
  Reproducibility is what allows you to rely on other maintainers' reviews. Without reproducibility, you can't be certain that what you're running has been audited at all. It's true that no single person can audit their entire dependency tree. But many eyes make all bugs shallow.
- Taek 5y ago
  You can't solve this problem without having a full history of code to inspect (unless you are decompiling), reproducibility is the first step and bootstrapability is the second step. Then we refine the toolchains and review processes to ensure high impact code is properly scrutinized. What we can't do is throw our hands up and say anyone who compromises the toolchain deep enough is just allowed to win. It will happen at some point if we don't put the right barriers in place. It's the first step of a long journey, but it is a step we should be taking.
  - donio 5y ago
    https://github.com/fosslinux/live-bootstrap https://github.com/fosslinux/live-bootstrap is another approach, bootstrapping from a tiny binary seed that you could hand-assemble and type in as hex. But it doesn't address the dependency on the underling OS being trustworthy.
    - bmwiedemann 5y ago
      There is stage0 by Jeremiah Orians that is designed to be able to bootstrap on hardware that can be built from transistors. Currently it mostly runs in a small VM process that is somewhat harder to subvert.
- IgorPartola 5y ago
  No. I can review 0.1% of the code and verify that it compiles correctly and then let another 999 people review their own portion. It only takes one person to find a bit of malicious code, we don’t all need to review every single line.
  - remram 5y ago
    That only works if you coordinate. With even more people, you can pick randomly and be relatively sure you've read it all, but I posit that 1) you don't pick randomly, you pick a part that is accessible or interesting to you (and therefore probably others) and 2) reading code locally is not sufficient to find bugs or backdoors in the whole.
    - pabs3 5y ago
      The crev folks are working on a co-ordination system for incremental distributed code review: https://github.com/crev-dev/ https://github.com/crev-dev/
      - IgorPartola 5y ago
        I actually wonder if it’s possible to write code at such a macro level as to obfuscate, say, a keylogger in a huge codebase such that reviewing just a single module/unit would not reveal that something bad is going on.
        eru 5y ago
        Depends on how complicated the project itself is. A simple structure with the bare minimum of side-effects (think, functional programming) would make this effort harder. For something like C, all bets are off: http://www.underhanded-c.org/ http://www.underhanded-c.org/ or https://en.wikipedia.org/wiki/Underhanded_C_Contest https://en.wikipedia.org/wiki/Underhanded_C_Contest
      - remram 5y ago
        Crev is a great idea, unfortunately it is only really available for Rust right now.
jnxx 5y ago
This is great! The one fly in the ointment, pardon, is that Nix is a bit lax about trusting proprietary and binary-only stuff. It would be great if there were a FLOSS-only core system for NixOS which would be fully transparent.
- rejectedandsad 5y ago
  > It would be great if there were a FLOSS-only core system for NixOS Might be wrong but isn't this part of the premise for Guix/GuixSD?
  - Filligree 5y ago
    And it's good that it exists, I guess? But it can't do any of the things I bought my computer to do, so it's of limited value to me.
- quarantine 5y ago
  Nix/Nixpkgs blocks unfree packages by default, so I presume it would be relatively easy to disable packages with the `unFree` attribute.
  - jnxx 5y ago
    I totally believe it is possible, it is perhaps more of a cultural thing.
    - eptcyka 5y ago
      It's the pragmatic thing. I wouldn't use nixOS if I wasn't able to use it on a 16 core modern desktop. I don't think there's a performant and 100% FLOSS compatible computer that wouldn't make me want to gouge my eyes out with a rusty spoon when building stuff for ARM.
      - zamadatix 5y ago
        Talos has 44 core/176 thread server options which can take 2 TBs of DDR4 that are FSF certified. The board firmware is also open and has reproducible builds.
radicalcentrist 5y ago
Reproducibility is necessary, but unfortunately not sufficient, to stop a "Trusting Trust" attack. Nixpkgs still relies on a bootstrap tarball containing e.g. gcc and binutils, so theoretically such an attack could trace its lineage back to the original bootstrap tarball, if it was built with a compromised toolchain.
- mjg59 5y ago
  Diverse double compilation should allow a demonstration that the toolchain is trustworthy.
  - Foxboron 5y ago
    Indeed, and with the work done by Guix and the Reproducible Builds project we do have a real-world example of diverse double compilation which is not just a toy example utilizing the GNU Mes C compiler. https://dwheeler.com/trusting-trust/#real-world https://dwheeler.com/trusting-trust/#real-world
    - dane-pgp 5y ago
      Projects like GNU Mes are part of the Bootstrappable Builds effort[0]. Another great achievement in that area is the live-bootstrap project, which has automated a build pipeline that goes from a minimal binary seed up to tinycc then gcc 4 and beyond.[1] [0] https://www.bootstrappable.org/ https://www.bootstrappable.org/ [1] https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst#66gcc-404 https://github.com/fosslinux/live-bootstrap/blob/master/part...
      - Foxboron 5y ago
        I feel the need to point out that the "Bootstrappable Builds" project is a working group from a Reproducible Builds project which where interested in the next step beyond reproducing binaries. Obviously this project has seen most effort from Guix :) The GNU Mes C experiment mentioned above was also conducted during the 2019 Reproducible Builds summit in Marrakesh. https://reproducible-builds.org/events/Marrakesh2019/ https://reproducible-builds.org/events/Marrakesh2019/
hsbauauvhabzb 5y ago
I don’t have the resources to audit every component of my system. I favour enterprise distros who audit code which ends up in their repos and avoid pip, npm, etc. but there are some glaring trade offs on both productivity and scalability. The problem is unmaintainability, I can’t imagine it’d be easier for medium sized teams where security isn’t a priority, either.
0xbadcafebee 5y ago
Supply chain attacks are definitely important to deal with, but defense-in-depth saves us in the end. Even if a postgres container is backdoored, if the admins put postgres by itself in a network with no ingress or egress except the webserver querying it, an attack on the database itself would be very difficult. If on the other hand, the database is run on untrusted networks, and sensitive data kept on it... yeah, they're boned.
- dcposch 5y ago
  In the case of a supply chain attack, you don't even need ingress or egress. Say the posgres binary or image is set to encrypt the data on a certain date. Then it asks you to pay X ZEC to a shielded address to get your decryption key. This would work even if the actual database was airgapped.
  - 0xbadcafebee 5y ago
    That's true, I didn't think of that! D:
initplus 5y ago
Building from source doesn't have to be inaccessible, if the build tooling around it is strong. Modern compiled languages like Go (or modern toolchains on legacy languages like vcpkg) have a convention of building everything possible from source. So at least for software libraries building from source is definitely viable. Fro end user applications it's another story though, doubt we will ever be at a point where building your own browser from source makes sense...
- garmaine 5y ago
  Binary reproducible builds are still pretty inaccessible though.
- bigiain 5y ago
  Building from source also doesn’t buy you very much, if you haven’t inspected/audited the source. The upthread hypothetical of a compromised package manager equally applies to a compromised source repo. _Maybe _ you always check the hashes? _Maybe_ you always get the hashes from a different place to the code? _Maybe_ the hypothetical attacker couldn’t replace both the code you download and the hash you use to check it? (And as Ken pointed out decades ago, maybe the attacker didn’t fuck with your compiler so you had lost before you even started.)
tbrock 5y ago
Why does building from source help? It’s not like people are reading every line of the source before building it anyway 99.99% of the time.
- xvector 5y ago
  If the package maintainer's build pipeline is compromised (eg. Solarwinds), you are unlikely to be affected if you build from reviewed source yourself.
  - pjmlp 5y ago
    Except hardly anyone reviews a single line of code.
    - squiggleblaz 5y ago
      So? We are trying to protect against a malicious interloper damaging the machine of a trusted and trustworthy partner. You are bringing up red herrings about trusted partners being malicious and untrustworthy. Do you genuinely believe we should only solve a problem if it leads to a perfect outcome?
      - pjmlp 5y ago
        I genuinely believe to spend resources on issues where ROI is positive. So far exploits on FOSS kind of prove the point not everyone is using Gentoo, reading every line of code on their emerged packakges, let alone similar computing models. Now if we are speaking about driving the whole industry to where security bugs, caused by using languages like C that cannot save us from code reviews unless done by ISO C language lawyers and compiler experts in UB optimizations, are heavily punished like construction companies are for a fallen bridge, then that would be interesting.
        therealjumbo 5y ago
        > I genuinely believe to spend resources on issues where ROI is positive. How are you measuring the ROI of security efforts inside an OSS distro like debian or nixos? The effort in such orgs is freely given, so nobody knows how much it costs. And how would you calculate the return on attacks that have been prevented? Even if an attack wasn't prevented you don't know how much it cost, and you might not even know if it happened (or if it happened due to a lapse in debian.) >So far exploits on FOSS kind of prove the point not everyone is using Gentoo, reading every line of code on their emerged packakges, let alone similar computing models. Reproducible builds is attempting to mitigate a very specific type of attack, not all attacks in general. That is, it focuses on a specific threat model and countering that, nothing else. It's not a cure for cancer either. >Now if we are speaking about driving the whole industry to where security bugs, caused by using languages like C that cannot save us from code reviews unless done by ISO C language lawyers and compiler experts in UB optimizations, are heavily punished like construction companies are for a fallen bridge, then that would be interesting. This is just a word salad of red herrings. Different people can work on different stuff at the same time.
1vuio0pswjnm7 5y ago
"- Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds." I prefer compiling from source to binary packages. For me it is neither slow, incovenient nor inaccessible. Only with larger, more complex programs does compiling from source become a PITA. The "solution" I take is to prefer smaller, less complex programs over larger, more complex ones. If I cannot compile a program from source relatively quickly and easily, I do not voluntarily choose it as a program to use daily and depend on. For compiling OS, I use NetBSD so perhaps I am spoiled because it is relatively easy to compile. That said, I understand the value of reproducible builds and appreciate the work being done on such projects.
- kixiQu 5y ago
  "except to nerds" was conversationally phrased shorthand for "except to people with rarefied technical skills".
- deleted 5y ago
  [deleted]
- kaba0 5y ago
  You don’t use a browser or an office suite? Because those are a pain in the ass to compile (in terms of time).
  - 1vuio0pswjnm7 5y ago
    Not just time, IME. Also 1. highly resource intensive, e.g., cannot compile on small form factor computers (easier for me to compile a kernel than a "modern" browser) and 2. brittle.
- brigandish 5y ago
  Unfortunately, it's easy to break a lot of builds by things such as deciding not to install to /usr/local, or by building on a Mac. Pushing publishers to practices that aid reproducible builds would help both sides. I'd love to try building NetBSD, btw, I must try that!
- 5y ago
swiley 5y ago
>self interest (companies do not want to attack their own users). Anyone who has bought an Android phone in the past 5 years knows that's not true.
staticassertion 5y ago
> Reproducible builds are way more important than is currently widely appreciated. Why? How will this help with the problems you're talking about? I can't come up with a single benefit to security from reproducible builds. It seems nice for operational reasons and performance reasons though.
- pilif 5y ago
  > I can't come up with a single benefit to security from reproducible builds. It is a means to allow to detect a compromised supply chain. If people rebuilding a distro cannot get the same hash as the distro shipping from the distributor, then likely the distributors infrastructure has been compromised
  - staticassertion 5y ago
    How does this work in practice? The distro is owned, so where are you getting the hash from? I mean, specifically, what does the attacker have control of and how does a repeatable build help me stop them.
    - pilif 5y ago
      The idea is that multiple independent builders build the same distro. You expect all of them to have the same final hash. This doesn't help against the sources being owned, but it helps about build machines being owned. Accountability for source integrity is in theory provided by the source control system. Accountability for the build machine integrity can be provided by reproducible builds. To answer your specific questions: The attacker has access to the distro's build servers and is packaging and shipping altered binaries that do not correspond to the sources but instead contain added malware. Reproducible builds allow third parties to also build binaries from the same sources and once multiple third parties achieve consensus about the build output, it becomes apparent that the distro's build infrastructure could be compromised.
      - staticassertion 5y ago
User23 5y ago
> Who remembers Ken Thompson's "Reflections on Trusting Trust"? > The norm today is auto-updating, pre-built software. This is a little bit misleading. The actual paper[1] explains that you can't even trust source available code. [1] https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-tho...
esjeon 5y ago
> I'm grateful to the nixos team for being beating a trail thru the jungle here. Retrofitting reproducibility onto a big software project that grew without it, is hard work. Actually, it's Debian guys who pushed reproducible build hard in the early days. They upstreamed necessary changes and also spread the concept itself. This is a two-decade long community effort. In turn, NixOS is mostly just wrapping those projects with their own tooling, literally a cherry on the top. NixOS is disproportionately credited here.
- zucker42 5y ago
  I don't think NixOS is getting too much credit. This is an accomplishment, even if it was built on the shoulders of giants.
- mikepurvis 5y ago
  I think both efforts have been important and have benefitted each other. Nix has always had purity/reproducibility as tenets, but indeed it was Debian that got serious about it on a bit-for-bit basis, with changes to the compilers, tools like diffoscope, etc. The broader awareness and feasibility of reproducible builds then made it possible for Nix to finally realise the original design goal of a content-addressed rather than input-addressed store, where you don't need to actually sign your binary cache, but rather just sign a mapping between input hashes and content hashes.
  - Ericson2314 5y ago
    > where you don't need to actually sign your binary cache, but rather just sign a mapping between input hashes and content hashes. Though you can and should sign the mapping!
    - mikepurvis 5y ago
      Of course, yes— that was what I was saying. But the theory with content-addressability is that unlike a conventional distro where the binaries must all be built and then archived and distributed centrally, Nix could do things like age-out the cache and only archive the hashes, and a third party could later offer a rebuild-on-demand service where the binaries that come out of it are known to be identical to those which were originally signed. A similar guarantee is super useful when it comes to things like debug symbols.
marcosdumay 5y ago
> I predict that everyone's imagination on this topic will expand once there's a big enough incident in the news. How the Solarwinds incident, with about every large software vendor being silently compromised for years does not qualify? Because it does not, people's imagination is as closed as it always was.
- yeowMeng 5y ago
  Solarwinds is closed source so the choice to build from source is not really an option.
  - pabs3 5y ago
    They could have distributed the code to a few select parties for the purposes of doing a build and nothing more.
    - marcosdumay 5y ago
      Specifically Microsoft did distribute the code to several parties for the purposes of auditing. But they didn't allow building it.
cookiengineer 5y ago
Actually, being able to build projects much easier from GitHub is the sole reason why I'm currently using Arch as my main OS. Building a project is just a shell script with a couple of defined functions. Quite literally. I really admire NixOS's philosophy of pushing the boundaries as a distro where everything, including configurations and modifications, can be done in a reproducible manner. They're basically trying to automate the review process down the line, which is absurdly complex as a challenge. And given stability and desktop integrations improve over time, I really think that Nix has the potential to be the base for easily forkable distributions. Building a live/bootable distro will be so much easier, as everything is just a set of configuration files anyways.
- takeda 5y ago
  This is slightly different thing. Nix and NixOS are trying to solve multiple things, and that's what it might be a bit confusing. Many people don't realize that, but if you get for example mentioned project from github and I do and we compile it on our machines we get a different file (it'll work the same but it won't be exactly the same). Say we use the same dependencies, we still will get a different files, because maybe you used slightly different version of the compiler, or maybe those dependencies were compiled with different dependencies or compilers. Maybe the project while building inserts a date, or pulls some file. There are million ways that we would end up with different files. The goal here is to get bit by bit identical files and it's like a Holy Grail in this area. NixOS just appears to achieved that and all packages that come with the system are now fully reproducible.
  - eru 5y ago
    A rich source of non-reproducibility is non-determinism introduced by parallel building. Preserving parallel execution, but arriving at deterministic outputs, is an interesting and ongoing challenge. With a rich mathematical structure, too.
londons_explore 5y ago
> and 6mo later every computer ... gets ransomwared. I'm really surprised such an attack hasn't happened already. It seems so trivial for a determined attacker to take over an opensource project (plenty of very popular projects have just a single jaded maintainer). The malicious compiler could inject an extra timed event into the main loop for the time the attack is scheduled to begin, but only if it's >3 hours away, which simply retrieves a URL and executes whatever is received. Detecting this by chance is highly unlikely - because to find it, someone would have to have their clock set months ahead, be running the software for many days, and be monitoring the network. That code is probably only a few hundred bytes, so it probably won't be noticed in any disassembly, and is only executed once, so probably won't show up in debugging sessions or cpu profiling. It just baffles me that this hasn't been done already!
- schelling42 5y ago
  How do you know it hasn't been done already? (with a more silent payload than ransomware) /s
  - Tabular-Iceberg 5y ago
    What does the /s mean in this context?
    - ghoward 5y ago
      Not GP, but I think it indicates sarcasm?
    - Zetaphor 5y ago
      /s is internet parlance to show that the message should be read in a sarcastic tone.
      - Tabular-Iceberg 5y ago
        Yes, but what confused me is that as far as I can tell we really don’t know that it hasn’t been done before.
tester756 5y ago
>There are only two ways around this: >- Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds. if you trust the compiler :)
Accujack 5y ago
>The norm today is auto-updating, pre-built software. Only if you define "norm" as what's prevalent in consumer electronics and phones. Certainly, if you go by numbers, it's more common than anything else. That's not due to choice, though, it's because of the desires of corporations for ever more extensive control of their revenue streams.
nix23 5y ago
>Who remembers Ken Thompson's "Reflections on Trusting Trust"? That was his Turing Award ;) not Unix as one would assume.

zamadatix 5y ago

Unless you are going to be the equivalent of a full time maintainer doing code review for every piece of software you use you need to trust other software maintainers reproducible builds or not. Considering this is Linux and not even Linus can deeply review every change in just the kernel anymore that philosophy can't apply to meaningfully large software like Nixos.

jnxx 5y ago
That's too black-and-white. Being able to reproduce stuff makes some kind of attacks entirely uninteresting because malicious changes can be traced back. Which is what many types of attackers do not want. Debian, or the Linux kernel, for example, are not fool-proof, but both are in practice quite safe to work with.
- zamadatix 5y ago
  Who are you going to trace it back to if not the maintainer anyways? If the delivery method then why is the delivery of the source from the maintainer inherently any safer?
  - jnxx 5y ago
    No, it is not always the maintainer. Imagine you download a binary software package via HTTPS. In theory, the integrity of the download is protected by the server certificate. However, it is possible that certificates get hacked, get stolen, or that nation states force CAs to give out back doors. In that case, your download could have been changed on the fly with arbitrary alterations. Reproducible builds make it possible to detect such changes.
    - zamadatix 5y ago
      Same as when you download the source instead of the binary and see it reproducibly builds the backdoored binary. And at this point we're back to "Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds." anyways. It's not that reproducible builds provide 0 value it's that they don't truly solve the trust problem as initially stated. They also have non-security value to boot which is often understated compared to the security value IMO.
      - bigiain 5y ago
        I guess reproducible builds solve some of the problems in the same way TLS/SSL solves some of the problems. Most of the world is happy enough with the soft guarantee of: “This is _probably_ your bank’s real website. Unless a nation state is misusing their control over state owned certificate authorities, or GlobalSign or LetsEncrypt or whoever has been p0wned.” Expecting binary black and white solutions to trust problems isn’t all that useful, in my opinion. Often providing 50% more “value” in trust compared to the status quo is extremely valuable in the bigger picture.
radicalcentrist 5y ago
Reproducibility is what allows you to rely on other maintainers' reviews. Without reproducibility, you can't be certain that what you're running has been audited at all. It's true that no single person can audit their entire dependency tree. But many eyes make all bugs shallow.
Taek 5y ago
You can't solve this problem without having a full history of code to inspect (unless you are decompiling), reproducibility is the first step and bootstrapability is the second step. Then we refine the toolchains and review processes to ensure high impact code is properly scrutinized. What we can't do is throw our hands up and say anyone who compromises the toolchain deep enough is just allowed to win. It will happen at some point if we don't put the right barriers in place. It's the first step of a long journey, but it is a step we should be taking.
- donio 5y ago
  https://github.com/fosslinux/live-bootstrap https://github.com/fosslinux/live-bootstrap is another approach, bootstrapping from a tiny binary seed that you could hand-assemble and type in as hex. But it doesn't address the dependency on the underling OS being trustworthy.
  - bmwiedemann 5y ago
    There is stage0 by Jeremiah Orians that is designed to be able to bootstrap on hardware that can be built from transistors. Currently it mostly runs in a small VM process that is somewhat harder to subvert.
IgorPartola 5y ago
No. I can review 0.1% of the code and verify that it compiles correctly and then let another 999 people review their own portion. It only takes one person to find a bit of malicious code, we don’t all need to review every single line.
- remram 5y ago
  That only works if you coordinate. With even more people, you can pick randomly and be relatively sure you've read it all, but I posit that 1) you don't pick randomly, you pick a part that is accessible or interesting to you (and therefore probably others) and 2) reading code locally is not sufficient to find bugs or backdoors in the whole.
  - pabs3 5y ago
    The crev folks are working on a co-ordination system for incremental distributed code review: https://github.com/crev-dev/ https://github.com/crev-dev/
    - IgorPartola 5y ago
      I actually wonder if it’s possible to write code at such a macro level as to obfuscate, say, a keylogger in a huge codebase such that reviewing just a single module/unit would not reveal that something bad is going on.
      - eru 5y ago
        Depends on how complicated the project itself is. A simple structure with the bare minimum of side-effects (think, functional programming) would make this effort harder. For something like C, all bets are off: http://www.underhanded-c.org/ http://www.underhanded-c.org/ or https://en.wikipedia.org/wiki/Underhanded_C_Contest https://en.wikipedia.org/wiki/Underhanded_C_Contest
    - remram 5y ago
      Crev is a great idea, unfortunately it is only really available for Rust right now.

jnxx 5y ago

This is great! The one fly in the ointment, pardon, is that Nix is a bit lax about trusting proprietary and binary-only stuff. It would be great if there were a FLOSS-only core system for NixOS which would be fully transparent.

rejectedandsad 5y ago
> It would be great if there were a FLOSS-only core system for NixOS Might be wrong but isn't this part of the premise for Guix/GuixSD?
- Filligree 5y ago
  And it's good that it exists, I guess? But it can't do any of the things I bought my computer to do, so it's of limited value to me.
quarantine 5y ago
Nix/Nixpkgs blocks unfree packages by default, so I presume it would be relatively easy to disable packages with the `unFree` attribute.
- jnxx 5y ago
  I totally believe it is possible, it is perhaps more of a cultural thing.
  - eptcyka 5y ago
    It's the pragmatic thing. I wouldn't use nixOS if I wasn't able to use it on a 16 core modern desktop. I don't think there's a performant and 100% FLOSS compatible computer that wouldn't make me want to gouge my eyes out with a rusty spoon when building stuff for ARM.
    - zamadatix 5y ago
      Talos has 44 core/176 thread server options which can take 2 TBs of DDR4 that are FSF certified. The board firmware is also open and has reproducible builds.

radicalcentrist 5y ago

Reproducibility is necessary, but unfortunately not sufficient, to stop a "Trusting Trust" attack. Nixpkgs still relies on a bootstrap tarball containing e.g. gcc and binutils, so theoretically such an attack could trace its lineage back to the original bootstrap tarball, if it was built with a compromised toolchain.

mjg59 5y ago
Diverse double compilation should allow a demonstration that the toolchain is trustworthy.
- Foxboron 5y ago
  Indeed, and with the work done by Guix and the Reproducible Builds project we do have a real-world example of diverse double compilation which is not just a toy example utilizing the GNU Mes C compiler. https://dwheeler.com/trusting-trust/#real-world https://dwheeler.com/trusting-trust/#real-world
  - dane-pgp 5y ago
    Projects like GNU Mes are part of the Bootstrappable Builds effort[0]. Another great achievement in that area is the live-bootstrap project, which has automated a build pipeline that goes from a minimal binary seed up to tinycc then gcc 4 and beyond.[1] [0] https://www.bootstrappable.org/ https://www.bootstrappable.org/ [1] https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst#66gcc-404 https://github.com/fosslinux/live-bootstrap/blob/master/part...
    - Foxboron 5y ago
      I feel the need to point out that the "Bootstrappable Builds" project is a working group from a Reproducible Builds project which where interested in the next step beyond reproducing binaries. Obviously this project has seen most effort from Guix :) The GNU Mes C experiment mentioned above was also conducted during the 2019 Reproducible Builds summit in Marrakesh. https://reproducible-builds.org/events/Marrakesh2019/ https://reproducible-builds.org/events/Marrakesh2019/

hsbauauvhabzb 5y ago

I don’t have the resources to audit every component of my system. I favour enterprise distros who audit code which ends up in their repos and avoid pip, npm, etc. but there are some glaring trade offs on both productivity and scalability. The problem is unmaintainability, I can’t imagine it’d be easier for medium sized teams where security isn’t a priority, either.

0xbadcafebee 5y ago

Supply chain attacks are definitely important to deal with, but defense-in-depth saves us in the end. Even if a postgres container is backdoored, if the admins put postgres by itself in a network with no ingress or egress except the webserver querying it, an attack on the database itself would be very difficult. If on the other hand, the database is run on untrusted networks, and sensitive data kept on it... yeah, they're boned.

dcposch 5y ago
In the case of a supply chain attack, you don't even need ingress or egress. Say the posgres binary or image is set to encrypt the data on a certain date. Then it asks you to pay X ZEC to a shielded address to get your decryption key. This would work even if the actual database was airgapped.
- 0xbadcafebee 5y ago
  That's true, I didn't think of that! D:

initplus 5y ago

Building from source doesn't have to be inaccessible, if the build tooling around it is strong. Modern compiled languages like Go (or modern toolchains on legacy languages like vcpkg) have a convention of building everything possible from source. So at least for software libraries building from source is definitely viable. Fro end user applications it's another story though, doubt we will ever be at a point where building your own browser from source makes sense...

garmaine 5y ago
Binary reproducible builds are still pretty inaccessible though.
bigiain 5y ago
Building from source also doesn’t buy you very much, if you haven’t inspected/audited the source. The upthread hypothetical of a compromised package manager equally applies to a compromised source repo. _Maybe _ you always check the hashes? _Maybe_ you always get the hashes from a different place to the code? _Maybe_ the hypothetical attacker couldn’t replace both the code you download and the hash you use to check it? (And as Ken pointed out decades ago, maybe the attacker didn’t fuck with your compiler so you had lost before you even started.)

tbrock 5y ago

Why does building from source help? It’s not like people are reading every line of the source before building it anyway 99.99% of the time.

xvector 5y ago
If the package maintainer's build pipeline is compromised (eg. Solarwinds), you are unlikely to be affected if you build from reviewed source yourself.
- pjmlp 5y ago
  Except hardly anyone reviews a single line of code.
  - squiggleblaz 5y ago
    So? We are trying to protect against a malicious interloper damaging the machine of a trusted and trustworthy partner. You are bringing up red herrings about trusted partners being malicious and untrustworthy. Do you genuinely believe we should only solve a problem if it leads to a perfect outcome?
    - pjmlp 5y ago
      I genuinely believe to spend resources on issues where ROI is positive. So far exploits on FOSS kind of prove the point not everyone is using Gentoo, reading every line of code on their emerged packakges, let alone similar computing models. Now if we are speaking about driving the whole industry to where security bugs, caused by using languages like C that cannot save us from code reviews unless done by ISO C language lawyers and compiler experts in UB optimizations, are heavily punished like construction companies are for a fallen bridge, then that would be interesting.
      - therealjumbo 5y ago
        > I genuinely believe to spend resources on issues where ROI is positive. How are you measuring the ROI of security efforts inside an OSS distro like debian or nixos? The effort in such orgs is freely given, so nobody knows how much it costs. And how would you calculate the return on attacks that have been prevented? Even if an attack wasn't prevented you don't know how much it cost, and you might not even know if it happened (or if it happened due to a lapse in debian.) >So far exploits on FOSS kind of prove the point not everyone is using Gentoo, reading every line of code on their emerged packakges, let alone similar computing models. Reproducible builds is attempting to mitigate a very specific type of attack, not all attacks in general. That is, it focuses on a specific threat model and countering that, nothing else. It's not a cure for cancer either. >Now if we are speaking about driving the whole industry to where security bugs, caused by using languages like C that cannot save us from code reviews unless done by ISO C language lawyers and compiler experts in UB optimizations, are heavily punished like construction companies are for a fallen bridge, then that would be interesting. This is just a word salad of red herrings. Different people can work on different stuff at the same time.

1vuio0pswjnm7 5y ago

"- Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds." I prefer compiling from source to binary packages. For me it is neither slow, incovenient nor inaccessible. Only with larger, more complex programs does compiling from source become a PITA. The "solution" I take is to prefer smaller, less complex programs over larger, more complex ones. If I cannot compile a program from source relatively quickly and easily, I do not voluntarily choose it as a program to use daily and depend on. For compiling OS, I use NetBSD so perhaps I am spoiled because it is relatively easy to compile. That said, I understand the value of reproducible builds and appreciate the work being done on such projects.

kixiQu 5y ago
"except to nerds" was conversationally phrased shorthand for "except to people with rarefied technical skills".
deleted 5y ago
[deleted]
kaba0 5y ago
You don’t use a browser or an office suite? Because those are a pain in the ass to compile (in terms of time).
- 1vuio0pswjnm7 5y ago
  Not just time, IME. Also 1. highly resource intensive, e.g., cannot compile on small form factor computers (easier for me to compile a kernel than a "modern" browser) and 2. brittle.
brigandish 5y ago
Unfortunately, it's easy to break a lot of builds by things such as deciding not to install to /usr/local, or by building on a Mac. Pushing publishers to practices that aid reproducible builds would help both sides. I'd love to try building NetBSD, btw, I must try that!
5y ago

swiley 5y ago

>self interest (companies do not want to attack their own users). Anyone who has bought an Android phone in the past 5 years knows that's not true.

staticassertion 5y ago

> Reproducible builds are way more important than is currently widely appreciated. Why? How will this help with the problems you're talking about? I can't come up with a single benefit to security from reproducible builds. It seems nice for operational reasons and performance reasons though.

pilif 5y ago
> I can't come up with a single benefit to security from reproducible builds. It is a means to allow to detect a compromised supply chain. If people rebuilding a distro cannot get the same hash as the distro shipping from the distributor, then likely the distributors infrastructure has been compromised
- staticassertion 5y ago
  How does this work in practice? The distro is owned, so where are you getting the hash from? I mean, specifically, what does the attacker have control of and how does a repeatable build help me stop them.
  - pilif 5y ago
    The idea is that multiple independent builders build the same distro. You expect all of them to have the same final hash. This doesn't help against the sources being owned, but it helps about build machines being owned. Accountability for source integrity is in theory provided by the source control system. Accountability for the build machine integrity can be provided by reproducible builds. To answer your specific questions: The attacker has access to the distro's build servers and is packaging and shipping altered binaries that do not correspond to the sources but instead contain added malware. Reproducible builds allow third parties to also build binaries from the same sources and once multiple third parties achieve consensus about the build output, it becomes apparent that the distro's build infrastructure could be compromised.
    - staticassertion 5y ago

User23 5y ago

> Who remembers Ken Thompson's "Reflections on Trusting Trust"? > The norm today is auto-updating, pre-built software. This is a little bit misleading. The actual paper[1] explains that you can't even trust source available code. [1] https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-tho...

esjeon 5y ago

> I'm grateful to the nixos team for being beating a trail thru the jungle here. Retrofitting reproducibility onto a big software project that grew without it, is hard work. Actually, it's Debian guys who pushed reproducible build hard in the early days. They upstreamed necessary changes and also spread the concept itself. This is a two-decade long community effort. In turn, NixOS is mostly just wrapping those projects with their own tooling, literally a cherry on the top. NixOS is disproportionately credited here.

zucker42 5y ago
I don't think NixOS is getting too much credit. This is an accomplishment, even if it was built on the shoulders of giants.
mikepurvis 5y ago
I think both efforts have been important and have benefitted each other. Nix has always had purity/reproducibility as tenets, but indeed it was Debian that got serious about it on a bit-for-bit basis, with changes to the compilers, tools like diffoscope, etc. The broader awareness and feasibility of reproducible builds then made it possible for Nix to finally realise the original design goal of a content-addressed rather than input-addressed store, where you don't need to actually sign your binary cache, but rather just sign a mapping between input hashes and content hashes.
- Ericson2314 5y ago
  > where you don't need to actually sign your binary cache, but rather just sign a mapping between input hashes and content hashes. Though you can and should sign the mapping!
  - mikepurvis 5y ago
    Of course, yes— that was what I was saying. But the theory with content-addressability is that unlike a conventional distro where the binaries must all be built and then archived and distributed centrally, Nix could do things like age-out the cache and only archive the hashes, and a third party could later offer a rebuild-on-demand service where the binaries that come out of it are known to be identical to those which were originally signed. A similar guarantee is super useful when it comes to things like debug symbols.

marcosdumay 5y ago

> I predict that everyone's imagination on this topic will expand once there's a big enough incident in the news. How the Solarwinds incident, with about every large software vendor being silently compromised for years does not qualify? Because it does not, people's imagination is as closed as it always was.

yeowMeng 5y ago
Solarwinds is closed source so the choice to build from source is not really an option.
- pabs3 5y ago
  They could have distributed the code to a few select parties for the purposes of doing a build and nothing more.
  - marcosdumay 5y ago
    Specifically Microsoft did distribute the code to several parties for the purposes of auditing. But they didn't allow building it.

cookiengineer 5y ago

Actually, being able to build projects much easier from GitHub is the sole reason why I'm currently using Arch as my main OS. Building a project is just a shell script with a couple of defined functions. Quite literally. I really admire NixOS's philosophy of pushing the boundaries as a distro where everything, including configurations and modifications, can be done in a reproducible manner. They're basically trying to automate the review process down the line, which is absurdly complex as a challenge. And given stability and desktop integrations improve over time, I really think that Nix has the potential to be the base for easily forkable distributions. Building a live/bootable distro will be so much easier, as everything is just a set of configuration files anyways.

takeda 5y ago
This is slightly different thing. Nix and NixOS are trying to solve multiple things, and that's what it might be a bit confusing. Many people don't realize that, but if you get for example mentioned project from github and I do and we compile it on our machines we get a different file (it'll work the same but it won't be exactly the same). Say we use the same dependencies, we still will get a different files, because maybe you used slightly different version of the compiler, or maybe those dependencies were compiled with different dependencies or compilers. Maybe the project while building inserts a date, or pulls some file. There are million ways that we would end up with different files. The goal here is to get bit by bit identical files and it's like a Holy Grail in this area. NixOS just appears to achieved that and all packages that come with the system are now fully reproducible.
- eru 5y ago
  A rich source of non-reproducibility is non-determinism introduced by parallel building. Preserving parallel execution, but arriving at deterministic outputs, is an interesting and ongoing challenge. With a rich mathematical structure, too.

londons_explore 5y ago

> and 6mo later every computer ... gets ransomwared. I'm really surprised such an attack hasn't happened already. It seems so trivial for a determined attacker to take over an opensource project (plenty of very popular projects have just a single jaded maintainer). The malicious compiler could inject an extra timed event into the main loop for the time the attack is scheduled to begin, but only if it's >3 hours away, which simply retrieves a URL and executes whatever is received. Detecting this by chance is highly unlikely - because to find it, someone would have to have their clock set months ahead, be running the software for many days, and be monitoring the network. That code is probably only a few hundred bytes, so it probably won't be noticed in any disassembly, and is only executed once, so probably won't show up in debugging sessions or cpu profiling. It just baffles me that this hasn't been done already!

schelling42 5y ago
How do you know it hasn't been done already? (with a more silent payload than ransomware) /s
- Tabular-Iceberg 5y ago
  What does the /s mean in this context?
  - ghoward 5y ago
    Not GP, but I think it indicates sarcasm?
  - Zetaphor 5y ago
    /s is internet parlance to show that the message should be read in a sarcastic tone.
    - Tabular-Iceberg 5y ago
      Yes, but what confused me is that as far as I can tell we really don’t know that it hasn’t been done before.

tester756 5y ago

>There are only two ways around this: >- Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds. if you trust the compiler :)

Accujack 5y ago

>The norm today is auto-updating, pre-built software. Only if you define "norm" as what's prevalent in consumer electronics and phones. Certainly, if you go by numbers, it's more common than anything else. That's not due to choice, though, it's because of the desires of corporations for ever more extensive control of their revenue streams.

nix23 5y ago

>Who remembers Ken Thompson's "Reflections on Trusting Trust"? That was his Turing Award ;) not Unix as one would assume.