13 ms·

Woah. This is something that would cause me to switch over from Linux. Reproducible builds are so important, they prevent a build server or developer laptop fr

by trengrj 10y ago

Woah. This is something that would cause me to switch over from Linux. Reproducible builds are so important, they prevent a build server or developer laptop from being a single point of failure as a tainted build can now be detected by others.

crest 10y ago
In that case have a look at https://wiki.netbsd.org/guide/veriexec/ https://wiki.netbsd.org/guide/veriexec/.
- swhipple 10y ago
  I don't quite understand the real world use-case Veriexec is designed to solve. 1) Prevent tampering by making part of the system immutable? The fingerprint isn't necessary; unconditionally prevent modification to the relevant files instead. 2) Prevent tampering by using trusted files? Normally this should be done by having a set of trusted keys, not hardcoded hashes. That way you can still securely upgrade the system. 3) Accessing files from a remote untrusted filesystem? This doesn't seem to work either; see the caveats section in veriexec(9). Am I missing something here?
fnord123 10y ago
As mentioned in the post, this is built from some tooling developed for Debian's reproducible builds project. Valerie Young had a good progress report at Linux.conf.au this year: Reproducible Builds for a Better Future: https://www.youtube.com/watch?v=-nMVfmZL8zk https://www.youtube.com/watch?v=-nMVfmZL8zk
- zoulasc 10y ago
  No, the build system is completely unique to NetBSD. To my knowledge no other OS has the ability to cross-build any platform from any POSIX OS. NetBSD can build the whole OS from source tree to distribution media with a single command.
  - aseipp 10y ago
    > NetBSD can build the whole OS from source tree to distribution media with a single command. This one, at least, can be done in NixOS/Guix once you check out the source -- and the Nix package manager can technically be installed on any Linux distro, too (and some other ports to Cygwin/FreeBSD/Mac etc) and run a single command to get the ISO, or any kind of build product you want. The carefully tested and maintained portability/cross-compilation is another thing though: NetBSD has fantastic support here that is not easily replicated without just doing a ton of work. So its universal basically-always-works cross compilation, everywhere -- is rather unique here. You can't build NixOS ISOs natively e.g. Nix-on-Darwin, which is rather unfortunate.
  - ajdlinux 10y ago
    The article specifically mentions that they borrowed the reproducible-builds.org analysis infrastructure that was developed by Debian. "I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file."
    - fnord123 10y ago
      zoulasc is the author of the article. So I don't know what to think anymore.
      - liveoneggs 10y ago
        NetBSD is built using build.sh + bmake, including inside of the debian-provided jenkins. The output of those jenkins builds is then run through debian-provided test suites to validate the reproduce-ability of aforementioned NetBSD builds.
        fnord123 10y ago
maheart 10y ago
Debian developers have been working on Reproducible Builds for a number of years[0]. Currently the Debian amd64 distribution is ~93.7% reproducible[1]. It's possible that Debian amd64 systems will be fully reproducible in Stretch+1 (~2-3 years). [0] https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds [1] https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht...
- ymse 10y ago
  GNU Guix is also working towards full reproducibility, and has an up-to-date (but incomplete) list at https://www.gnu.org/software/guix/packages/reproducibility.html https://www.gnu.org/software/guix/packages/reproducibility.h...
- gwu78 10y ago
  The Debian wiki page appears to be about reproducible builds for packages. The NetBSD wiki entry is about progress on reproducible builds for tools, kernel and userland. As for packages, reproducible builds under pkgsrc is still a WIP. My use of third party binary packages and pkgsrc is minimal but I am continually building custom kernels and crunched binaries with build.sh and I do make use of the binary userlands from releng. For me at least, having reproducible builds for these alone is quite useful.
  - throwaway2048 10y ago
    On debian tools, kernel and userland are packages.
    - gwu78 10y ago
      I should have guessed. I use their libc6 package for q)k) under /emul/linux and in the past the live.debian.org images occasionally but that is about as much as I know about Debian. Apologies for my ignorance.
- qznc 10y ago
  93% of the distribution could be enough for almost everybody or for nearly nobody if some core packages are not reproducible yet. Is there any data on that? Maybe using the Debian popcon data?
nickpsecurity 10y ago
They really dont. The compiler can still be malicious since developers might be malicious or just screw up. You're just increasing probability the binary has same vulnerabilities as the source with nothing added during compilation. You can just as easily just download the source over one or more links to your OS then compile it locally. That was what TCSEC required for trusted distribution. TCSEC was partly designed by guy (Karger) who invented the compiling-compiler attack if you're wondering credibility. If your existing compiler is subverted, then your system is probably subverted with rootkit given that's easier than sniping one program you may never use the intended way with exotic attack. If you cant trust the repos for source or local compiler, then how did you trust your kernel or anything else to begin with? The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned.
- ymse 10y ago
  > The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned. Even if you have all that, how can you be sure no one tampered your system at some point? Reproducibility lets you compare your entire system against known-good states.
  - nickpsecurity 10y ago
    " Reproducibility lets you compare your entire system against known-good states." If you have a known, good state, then you don't have to. You simply use the source or binaries in the repo after checking hashes or signatures on a few mirrors. If your state is that bad, then you're already compromised with possible a rootkit. "Even if you have all that, how can you be sure no one tampered your system at some point?" That takes way more than reproducible builds. You need source code for a compiler that doesn't have 0-days in it in terms of backdoors, optimizations that eliminate security checks, or passes that create exploitable faults. That's called a verified compiler. Only one exists for C (CompCert) that's proprietary with quite a few open ones for ML languages. The SCM's distributing that source code must be secure so they aren't adding modifications to source or your OS binary you started with. During transfer, they have to be signed or the transport has to be secure. You then need some kind of local tool you can trust to bootstrap it. If you trust nothing, you're going to be using ancient hardware you bought in person with cash to run an interpreter you wrote by hand that executes something equivalent to a small C compiler you use to compile the others that you inspected by eye or trusted via 3rd party review w/ hash/sig confirmation. If exotic attacks like Karger's are in your threat model, so are 0-days that are accidental or deliberate along with endpoint and SCM attacks. Reproducible builds don't protect you from much in that model. The stronger methods, from endpoint protection to SCM to local tooling, will pay off over time in more situations given they're independently critical. It's why Karger et al put them in first standards for INFOSEC to begin with. The other problem is that making all binaries come out equal creates more of a monoculture that aids attacks on endpoints. Many good tools in the field automatically harden C programs by making them memory or data-flow safe. Softbound+CETS comes to mind. Others obfuscate the hell out of them on a per-user basis to reduce odds of one-size fits all attack. You can't use these techniques with everyone having same binary. So, it's a step back from existing compiler and system security methods in total assurance you can achieve.

by trengrj 10y ago

crest 10y ago

In that case have a look at https://wiki.netbsd.org/guide/veriexec/ https://wiki.netbsd.org/guide/veriexec/.

swhipple 10y ago
I don't quite understand the real world use-case Veriexec is designed to solve. 1) Prevent tampering by making part of the system immutable? The fingerprint isn't necessary; unconditionally prevent modification to the relevant files instead. 2) Prevent tampering by using trusted files? Normally this should be done by having a set of trusted keys, not hardcoded hashes. That way you can still securely upgrade the system. 3) Accessing files from a remote untrusted filesystem? This doesn't seem to work either; see the caveats section in veriexec(9). Am I missing something here?

fnord123 10y ago

As mentioned in the post, this is built from some tooling developed for Debian's reproducible builds project. Valerie Young had a good progress report at Linux.conf.au this year: Reproducible Builds for a Better Future: https://www.youtube.com/watch?v=-nMVfmZL8zk https://www.youtube.com/watch?v=-nMVfmZL8zk

zoulasc 10y ago
No, the build system is completely unique to NetBSD. To my knowledge no other OS has the ability to cross-build any platform from any POSIX OS. NetBSD can build the whole OS from source tree to distribution media with a single command.
- aseipp 10y ago
  > NetBSD can build the whole OS from source tree to distribution media with a single command. This one, at least, can be done in NixOS/Guix once you check out the source -- and the Nix package manager can technically be installed on any Linux distro, too (and some other ports to Cygwin/FreeBSD/Mac etc) and run a single command to get the ISO, or any kind of build product you want. The carefully tested and maintained portability/cross-compilation is another thing though: NetBSD has fantastic support here that is not easily replicated without just doing a ton of work. So its universal basically-always-works cross compilation, everywhere -- is rather unique here. You can't build NixOS ISOs natively e.g. Nix-on-Darwin, which is rather unfortunate.
- ajdlinux 10y ago
  The article specifically mentions that they borrowed the reproducible-builds.org analysis infrastructure that was developed by Debian. "I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file."
  - fnord123 10y ago
    zoulasc is the author of the article. So I don't know what to think anymore.
    - liveoneggs 10y ago
      NetBSD is built using build.sh + bmake, including inside of the debian-provided jenkins. The output of those jenkins builds is then run through debian-provided test suites to validate the reproduce-ability of aforementioned NetBSD builds.
      - fnord123 10y ago

maheart 10y ago

Debian developers have been working on Reproducible Builds for a number of years[0]. Currently the Debian amd64 distribution is ~93.7% reproducible[1]. It's possible that Debian amd64 systems will be fully reproducible in Stretch+1 (~2-3 years). [0] https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds [1] https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht...

ymse 10y ago
GNU Guix is also working towards full reproducibility, and has an up-to-date (but incomplete) list at https://www.gnu.org/software/guix/packages/reproducibility.html https://www.gnu.org/software/guix/packages/reproducibility.h...
gwu78 10y ago
The Debian wiki page appears to be about reproducible builds for packages. The NetBSD wiki entry is about progress on reproducible builds for tools, kernel and userland. As for packages, reproducible builds under pkgsrc is still a WIP. My use of third party binary packages and pkgsrc is minimal but I am continually building custom kernels and crunched binaries with build.sh and I do make use of the binary userlands from releng. For me at least, having reproducible builds for these alone is quite useful.
- throwaway2048 10y ago
  On debian tools, kernel and userland are packages.
  - gwu78 10y ago
    I should have guessed. I use their libc6 package for q)k) under /emul/linux and in the past the live.debian.org images occasionally but that is about as much as I know about Debian. Apologies for my ignorance.
qznc 10y ago
93% of the distribution could be enough for almost everybody or for nearly nobody if some core packages are not reproducible yet. Is there any data on that? Maybe using the Debian popcon data?

nickpsecurity 10y ago

They really dont. The compiler can still be malicious since developers might be malicious or just screw up. You're just increasing probability the binary has same vulnerabilities as the source with nothing added during compilation. You can just as easily just download the source over one or more links to your OS then compile it locally. That was what TCSEC required for trusted distribution. TCSEC was partly designed by guy (Karger) who invented the compiling-compiler attack if you're wondering credibility. If your existing compiler is subverted, then your system is probably subverted with rootkit given that's easier than sniping one program you may never use the intended way with exotic attack. If you cant trust the repos for source or local compiler, then how did you trust your kernel or anything else to begin with? The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned.

ymse 10y ago
> The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned. Even if you have all that, how can you be sure no one tampered your system at some point? Reproducibility lets you compare your entire system against known-good states.
- nickpsecurity 10y ago
  " Reproducibility lets you compare your entire system against known-good states." If you have a known, good state, then you don't have to. You simply use the source or binaries in the repo after checking hashes or signatures on a few mirrors. If your state is that bad, then you're already compromised with possible a rootkit. "Even if you have all that, how can you be sure no one tampered your system at some point?" That takes way more than reproducible builds. You need source code for a compiler that doesn't have 0-days in it in terms of backdoors, optimizations that eliminate security checks, or passes that create exploitable faults. That's called a verified compiler. Only one exists for C (CompCert) that's proprietary with quite a few open ones for ML languages. The SCM's distributing that source code must be secure so they aren't adding modifications to source or your OS binary you started with. During transfer, they have to be signed or the transport has to be secure. You then need some kind of local tool you can trust to bootstrap it. If you trust nothing, you're going to be using ancient hardware you bought in person with cash to run an interpreter you wrote by hand that executes something equivalent to a small C compiler you use to compile the others that you inspected by eye or trusted via 3rd party review w/ hash/sig confirmation. If exotic attacks like Karger's are in your threat model, so are 0-days that are accidental or deliberate along with endpoint and SCM attacks. Reproducible builds don't protect you from much in that model. The stronger methods, from endpoint protection to SCM to local tooling, will pay off over time in more situations given they're independently critical. It's why Karger et al put them in first standards for INFOSEC to begin with. The other problem is that making all binaries come out equal creates more of a monoculture that aids attacks on endpoints. Many good tools in the field automatically harden C programs by making them memory or data-flow safe. Softbound+CETS comes to mind. Others obfuscate the hell out of them on a per-user basis to reduce odds of one-size fits all attack. You can't use these techniques with everyone having same binary. So, it's a step back from existing compiler and system security methods in total assurance you can achieve.