8 ms·

NetBSD fully reproducible builds

281 points by conductor 10y ago

trengrj 10y ago
Woah. This is something that would cause me to switch over from Linux. Reproducible builds are so important, they prevent a build server or developer laptop from being a single point of failure as a tainted build can now be detected by others.
- crest 10y ago
  In that case have a look at https://wiki.netbsd.org/guide/veriexec/ https://wiki.netbsd.org/guide/veriexec/.
  - swhipple 10y ago
    I don't quite understand the real world use-case Veriexec is designed to solve. 1) Prevent tampering by making part of the system immutable? The fingerprint isn't necessary; unconditionally prevent modification to the relevant files instead. 2) Prevent tampering by using trusted files? Normally this should be done by having a set of trusted keys, not hardcoded hashes. That way you can still securely upgrade the system. 3) Accessing files from a remote untrusted filesystem? This doesn't seem to work either; see the caveats section in veriexec(9). Am I missing something here?
- fnord123 10y ago
  As mentioned in the post, this is built from some tooling developed for Debian's reproducible builds project. Valerie Young had a good progress report at Linux.conf.au this year: Reproducible Builds for a Better Future: https://www.youtube.com/watch?v=-nMVfmZL8zk https://www.youtube.com/watch?v=-nMVfmZL8zk
  - zoulasc 10y ago
    No, the build system is completely unique to NetBSD. To my knowledge no other OS has the ability to cross-build any platform from any POSIX OS. NetBSD can build the whole OS from source tree to distribution media with a single command.
    - aseipp 10y ago
      > NetBSD can build the whole OS from source tree to distribution media with a single command. This one, at least, can be done in NixOS/Guix once you check out the source -- and the Nix package manager can technically be installed on any Linux distro, too (and some other ports to Cygwin/FreeBSD/Mac etc) and run a single command to get the ISO, or any kind of build product you want. The carefully tested and maintained portability/cross-compilation is another thing though: NetBSD has fantastic support here that is not easily replicated without just doing a ton of work. So its universal basically-always-works cross compilation, everywhere -- is rather unique here. You can't build NixOS ISOs natively e.g. Nix-on-Darwin, which is rather unfortunate.
    - ajdlinux 10y ago
      The article specifically mentions that they borrowed the reproducible-builds.org analysis infrastructure that was developed by Debian. "I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file."
      - fnord123 10y ago
        zoulasc is the author of the article. So I don't know what to think anymore.
        liveoneggs 10y ago
        NetBSD is built using build.sh + bmake, including inside of the debian-provided jenkins. The output of those jenkins builds is then run through debian-provided test suites to validate the reproduce-ability of aforementioned NetBSD builds.
        fnord123 10y ago
- maheart 10y ago
  Debian developers have been working on Reproducible Builds for a number of years[0]. Currently the Debian amd64 distribution is ~93.7% reproducible[1]. It's possible that Debian amd64 systems will be fully reproducible in Stretch+1 (~2-3 years). [0] https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds [1] https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht...
  - ymse 10y ago
    GNU Guix is also working towards full reproducibility, and has an up-to-date (but incomplete) list at https://www.gnu.org/software/guix/packages/reproducibility.html https://www.gnu.org/software/guix/packages/reproducibility.h...
  - gwu78 10y ago
    The Debian wiki page appears to be about reproducible builds for packages. The NetBSD wiki entry is about progress on reproducible builds for tools, kernel and userland. As for packages, reproducible builds under pkgsrc is still a WIP. My use of third party binary packages and pkgsrc is minimal but I am continually building custom kernels and crunched binaries with build.sh and I do make use of the binary userlands from releng. For me at least, having reproducible builds for these alone is quite useful.
    - throwaway2048 10y ago
      On debian tools, kernel and userland are packages.
      - gwu78 10y ago
        I should have guessed. I use their libc6 package for q)k) under /emul/linux and in the past the live.debian.org images occasionally but that is about as much as I know about Debian. Apologies for my ignorance.
  - qznc 10y ago
    93% of the distribution could be enough for almost everybody or for nearly nobody if some core packages are not reproducible yet. Is there any data on that? Maybe using the Debian popcon data?
- nickpsecurity 10y ago
  They really dont. The compiler can still be malicious since developers might be malicious or just screw up. You're just increasing probability the binary has same vulnerabilities as the source with nothing added during compilation. You can just as easily just download the source over one or more links to your OS then compile it locally. That was what TCSEC required for trusted distribution. TCSEC was partly designed by guy (Karger) who invented the compiling-compiler attack if you're wondering credibility. If your existing compiler is subverted, then your system is probably subverted with rootkit given that's easier than sniping one program you may never use the intended way with exotic attack. If you cant trust the repos for source or local compiler, then how did you trust your kernel or anything else to begin with? The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned.
  - ymse 10y ago
    > The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned. Even if you have all that, how can you be sure no one tampered your system at some point? Reproducibility lets you compare your entire system against known-good states.
    - nickpsecurity 10y ago
      " Reproducibility lets you compare your entire system against known-good states." If you have a known, good state, then you don't have to. You simply use the source or binaries in the repo after checking hashes or signatures on a few mirrors. If your state is that bad, then you're already compromised with possible a rootkit. "Even if you have all that, how can you be sure no one tampered your system at some point?" That takes way more than reproducible builds. You need source code for a compiler that doesn't have 0-days in it in terms of backdoors, optimizations that eliminate security checks, or passes that create exploitable faults. That's called a verified compiler. Only one exists for C (CompCert) that's proprietary with quite a few open ones for ML languages. The SCM's distributing that source code must be secure so they aren't adding modifications to source or your OS binary you started with. During transfer, they have to be signed or the transport has to be secure. You then need some kind of local tool you can trust to bootstrap it. If you trust nothing, you're going to be using ancient hardware you bought in person with cash to run an interpreter you wrote by hand that executes something equivalent to a small C compiler you use to compile the others that you inspected by eye or trusted via 3rd party review w/ hash/sig confirmation. If exotic attacks like Karger's are in your threat model, so are 0-days that are accidental or deliberate along with endpoint and SCM attacks. Reproducible builds don't protect you from much in that model. The stronger methods, from endpoint protection to SCM to local tooling, will pay off over time in more situations given they're independently critical. It's why Karger et al put them in first standards for INFOSEC to begin with. The other problem is that making all binaries come out equal creates more of a monoculture that aids attacks on endpoints. Many good tools in the field automatically harden C programs by making them memory or data-flow safe. Softbound+CETS comes to mind. Others obfuscate the hell out of them on a per-user basis to reduce odds of one-size fits all attack. You can't use these techniques with everyone having same binary. So, it's a step back from existing compiler and system security methods in total assurance you can achieve.
ysleepy 10y ago
That is wonderful. I always dread dealing with build systems, mostly in the C land. Deterministic behaviour, especially in this rigorous fashion, is probably very helpful for much more cases than just trust. This looming assumption that make executes pure functions to produce output could actually become true. Now it really suffices if make triggers a target if one of the inputs changed.
- deleted 10y ago
  [deleted]
- ymse 10y ago
  > Now it really suffices if make triggers a target if one of the inputs changed. This is exactly how Nix and Guix works. If any inputs change, it will force a rebuild of all dependent packages.
- lamby 10y ago
  > helpful for much more cases than just trust. Indeed, there are a load of QA benefits for reproducible builds. Let alone the CO2 savings that result from cache hits instead of pointlessly rebuilding dependencies.
theamk 10y ago
FTR, Debian is like 80% reproducible https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds "Reproducible builds of Debian as a whole are still not a reality, though individual reproducible builds of packages are possible. So while we are making very good progress, it is a stretch to say that Debian is reproducible." In my experience, if you have a single home made package in C it is pretty easy to make it reproducible
- tachion 10y ago
  It's funny that to prove your argument you're quoting a paragraph that says exactly otherwise. Not to mention, NetBSD builds are about the OS, that's kernel, tooling, base and so on, and not about packages, as in Debian's case.
  - bcook 10y ago
    As another comment notes, Debian packages the kernel, tooling, base, etc.
eriknstr 10y ago
If author is here, there's a typo in the text. It says "iso6990" but should say "ISO 9660".
- zoulasc 10y ago
  thanks!
deleted 10y ago
[deleted]
sly010 10y ago
Anyone has a list of what are the typical sources of non-determinism (other than compiler and automake versions)?
- emaste 10y ago
  There's a fairly comprehensive list at https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/. Debian has a detailed list of reproducibility issues and affected packages at https://tests.reproducible-builds.org/debian/index_issues.html https://tests.reproducible-builds.org/debian/index_issues.ht...
  - ymse 10y ago
    Here's a smørbrødliste: * Non-isolated build environment. This is just asking for all sorts of trouble (users, hostname, network access, etc). * File system time stamps. * Recording times in the build process (although even gcc supports SOURCE_DATE_EPOCH since version 7). * Usage of CPU-specific instructions, e.g. -march=native to GCC.
- pserwylo 10y ago
  The post itself has a pretty good list of 10 sources of non-determinism: 1. timestamps 2. dates/times/authors etc. embedded in source files 3. timezone sensitive code 4. directory order/build order 5. non-sanitized data stored into files 6. symbolic links/paths 7. general tool inconsistencies 8. toolchain 9. build information / tunables / environment 10. making sure that the source tree has no local changes With a bit of info on each.
- vacri 10y ago
  That's exactly what the article is about.
jrcii 10y ago
Of all the great OSes, NetBSD seems to get the least fanfare but it's always been my favorite. It's fast, secure, and due to a well reasoned architecture it works on everything. pkgsrc is amazing. They also have a very helpful, friendly community. It especially warms my heart to run `ps -ax` on a new install and see all of about 10 processes. The OS feels lean, neat, and organized, and I feel like I know exactly what is going on, where to find a given file, etc. These special strengths -- vast hardware compatibility, rump kernels, now full reproducible builds, are all enabled by a greater underlying (and seemingly underrated) technical excellence.
- geff82 10y ago
  Also run "top" on the fresh install to see it uses about 10MB :D . I learned how to use Unix when I installed and used NetBSD for some years, so for me it is "Mother-UNIX". It's a pity that general hardware support is not that great anymore nowadays. But it runs quite well on the Lenovo X240/T440 range of laptops which can still be had new in some places.
hendry 10y ago
If you care about getting on the Web safely, Webconverger has reproducible builds at https://build.webconverger.com/ https://build.webconverger.com/
dolzenko 10y ago
I know, it's unrelated, but its 2017 and > Unfortunately this was not easy to find on NetBSD, because we are still using CVS as the source control system seems just weird.
gumby 10y ago
At Cygnus we had a customer from the telecom industry. They had SLAs with their own customers that included terms like "no more than 10 minutes of downtime per decade". They paid a LOT of money to have one, consistent release (no upgrades, only bug fixes); when they reported a big and got a fix they would diff the new binary and required that every change could be traced solely to the patch issued a nothing else. Satisfying this made GCC a lot better.
ksec 10y ago
Forgive my Ignorance, OpenBSD - Absolute Security FreeBSD - General purpose What is NetBSD aiming at?
- smhenderson 10y ago
  Maximum portability... "Of course it runs NetBSD" From their website: NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent for use in both production and research environments, and the source code is freely available under a business-friendly license.

281 points by conductor 10y ago

Woah. This is something that would cause me to switch over from Linux. Reproducible builds are so important, they prevent a build server or developer laptop from being a single point of failure as a tainted build can now be detected by others.

crest 10y ago
In that case have a look at https://wiki.netbsd.org/guide/veriexec/ https://wiki.netbsd.org/guide/veriexec/.
- swhipple 10y ago
  I don't quite understand the real world use-case Veriexec is designed to solve. 1) Prevent tampering by making part of the system immutable? The fingerprint isn't necessary; unconditionally prevent modification to the relevant files instead. 2) Prevent tampering by using trusted files? Normally this should be done by having a set of trusted keys, not hardcoded hashes. That way you can still securely upgrade the system. 3) Accessing files from a remote untrusted filesystem? This doesn't seem to work either; see the caveats section in veriexec(9). Am I missing something here?
fnord123 10y ago
As mentioned in the post, this is built from some tooling developed for Debian's reproducible builds project. Valerie Young had a good progress report at Linux.conf.au this year: Reproducible Builds for a Better Future: https://www.youtube.com/watch?v=-nMVfmZL8zk https://www.youtube.com/watch?v=-nMVfmZL8zk
- zoulasc 10y ago
  No, the build system is completely unique to NetBSD. To my knowledge no other OS has the ability to cross-build any platform from any POSIX OS. NetBSD can build the whole OS from source tree to distribution media with a single command.
  - aseipp 10y ago
    > NetBSD can build the whole OS from source tree to distribution media with a single command. This one, at least, can be done in NixOS/Guix once you check out the source -- and the Nix package manager can technically be installed on any Linux distro, too (and some other ports to Cygwin/FreeBSD/Mac etc) and run a single command to get the ISO, or any kind of build product you want. The carefully tested and maintained portability/cross-compilation is another thing though: NetBSD has fantastic support here that is not easily replicated without just doing a ton of work. So its universal basically-always-works cross compilation, everywhere -- is rather unique here. You can't build NixOS ISOs natively e.g. Nix-on-Darwin, which is rather unfortunate.
  - ajdlinux 10y ago
    The article specifically mentions that they borrowed the reproducible-builds.org analysis infrastructure that was developed by Debian. "I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file."
    - fnord123 10y ago
      zoulasc is the author of the article. So I don't know what to think anymore.
      - liveoneggs 10y ago
        NetBSD is built using build.sh + bmake, including inside of the debian-provided jenkins. The output of those jenkins builds is then run through debian-provided test suites to validate the reproduce-ability of aforementioned NetBSD builds.
        fnord123 10y ago
maheart 10y ago
Debian developers have been working on Reproducible Builds for a number of years[0]. Currently the Debian amd64 distribution is ~93.7% reproducible[1]. It's possible that Debian amd64 systems will be fully reproducible in Stretch+1 (~2-3 years). [0] https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds [1] https://tests.reproducible-builds.org/debian/reproducible.html https://tests.reproducible-builds.org/debian/reproducible.ht...
- ymse 10y ago
  GNU Guix is also working towards full reproducibility, and has an up-to-date (but incomplete) list at https://www.gnu.org/software/guix/packages/reproducibility.html https://www.gnu.org/software/guix/packages/reproducibility.h...
- gwu78 10y ago
  The Debian wiki page appears to be about reproducible builds for packages. The NetBSD wiki entry is about progress on reproducible builds for tools, kernel and userland. As for packages, reproducible builds under pkgsrc is still a WIP. My use of third party binary packages and pkgsrc is minimal but I am continually building custom kernels and crunched binaries with build.sh and I do make use of the binary userlands from releng. For me at least, having reproducible builds for these alone is quite useful.
  - throwaway2048 10y ago
    On debian tools, kernel and userland are packages.
    - gwu78 10y ago
      I should have guessed. I use their libc6 package for q)k) under /emul/linux and in the past the live.debian.org images occasionally but that is about as much as I know about Debian. Apologies for my ignorance.
- qznc 10y ago
  93% of the distribution could be enough for almost everybody or for nearly nobody if some core packages are not reproducible yet. Is there any data on that? Maybe using the Debian popcon data?
nickpsecurity 10y ago
They really dont. The compiler can still be malicious since developers might be malicious or just screw up. You're just increasing probability the binary has same vulnerabilities as the source with nothing added during compilation. You can just as easily just download the source over one or more links to your OS then compile it locally. That was what TCSEC required for trusted distribution. TCSEC was partly designed by guy (Karger) who invented the compiling-compiler attack if you're wondering credibility. If your existing compiler is subverted, then your system is probably subverted with rootkit given that's easier than sniping one program you may never use the intended way with exotic attack. If you cant trust the repos for source or local compiler, then how did you trust your kernel or anything else to begin with? The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned.
- ymse 10y ago
  > The real solution is secure SCM, ability to build from source, and a trusted point to start with. You have the latter unless your system is pre-backdoored. Reproducible builds might have other benefits such as during debugging. It's bringing you little security benefit over the stronger alternative I mentioned. Even if you have all that, how can you be sure no one tampered your system at some point? Reproducibility lets you compare your entire system against known-good states.
  - nickpsecurity 10y ago
    " Reproducibility lets you compare your entire system against known-good states." If you have a known, good state, then you don't have to. You simply use the source or binaries in the repo after checking hashes or signatures on a few mirrors. If your state is that bad, then you're already compromised with possible a rootkit. "Even if you have all that, how can you be sure no one tampered your system at some point?" That takes way more than reproducible builds. You need source code for a compiler that doesn't have 0-days in it in terms of backdoors, optimizations that eliminate security checks, or passes that create exploitable faults. That's called a verified compiler. Only one exists for C (CompCert) that's proprietary with quite a few open ones for ML languages. The SCM's distributing that source code must be secure so they aren't adding modifications to source or your OS binary you started with. During transfer, they have to be signed or the transport has to be secure. You then need some kind of local tool you can trust to bootstrap it. If you trust nothing, you're going to be using ancient hardware you bought in person with cash to run an interpreter you wrote by hand that executes something equivalent to a small C compiler you use to compile the others that you inspected by eye or trusted via 3rd party review w/ hash/sig confirmation. If exotic attacks like Karger's are in your threat model, so are 0-days that are accidental or deliberate along with endpoint and SCM attacks. Reproducible builds don't protect you from much in that model. The stronger methods, from endpoint protection to SCM to local tooling, will pay off over time in more situations given they're independently critical. It's why Karger et al put them in first standards for INFOSEC to begin with. The other problem is that making all binaries come out equal creates more of a monoculture that aids attacks on endpoints. Many good tools in the field automatically harden C programs by making them memory or data-flow safe. Softbound+CETS comes to mind. Others obfuscate the hell out of them on a per-user basis to reduce odds of one-size fits all attack. You can't use these techniques with everyone having same binary. So, it's a step back from existing compiler and system security methods in total assurance you can achieve.

ysleepy 10y ago

That is wonderful. I always dread dealing with build systems, mostly in the C land. Deterministic behaviour, especially in this rigorous fashion, is probably very helpful for much more cases than just trust. This looming assumption that make executes pure functions to produce output could actually become true. Now it really suffices if make triggers a target if one of the inputs changed.

deleted 10y ago
[deleted]
ymse 10y ago
> Now it really suffices if make triggers a target if one of the inputs changed. This is exactly how Nix and Guix works. If any inputs change, it will force a rebuild of all dependent packages.
lamby 10y ago
> helpful for much more cases than just trust. Indeed, there are a load of QA benefits for reproducible builds. Let alone the CO2 savings that result from cache hits instead of pointlessly rebuilding dependencies.

theamk 10y ago

FTR, Debian is like 80% reproducible https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds "Reproducible builds of Debian as a whole are still not a reality, though individual reproducible builds of packages are possible. So while we are making very good progress, it is a stretch to say that Debian is reproducible." In my experience, if you have a single home made package in C it is pretty easy to make it reproducible

tachion 10y ago
It's funny that to prove your argument you're quoting a paragraph that says exactly otherwise. Not to mention, NetBSD builds are about the OS, that's kernel, tooling, base and so on, and not about packages, as in Debian's case.
- bcook 10y ago
  As another comment notes, Debian packages the kernel, tooling, base, etc.

eriknstr 10y ago

If author is here, there's a typo in the text. It says "iso6990" but should say "ISO 9660".

zoulasc 10y ago
thanks!

deleted 10y ago

[deleted]

sly010 10y ago

Anyone has a list of what are the typical sources of non-determinism (other than compiler and automake versions)?

emaste 10y ago
There's a fairly comprehensive list at https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/. Debian has a detailed list of reproducibility issues and affected packages at https://tests.reproducible-builds.org/debian/index_issues.html https://tests.reproducible-builds.org/debian/index_issues.ht...
- ymse 10y ago
  Here's a smørbrødliste: * Non-isolated build environment. This is just asking for all sorts of trouble (users, hostname, network access, etc). * File system time stamps. * Recording times in the build process (although even gcc supports SOURCE_DATE_EPOCH since version 7). * Usage of CPU-specific instructions, e.g. -march=native to GCC.
pserwylo 10y ago
The post itself has a pretty good list of 10 sources of non-determinism: 1. timestamps 2. dates/times/authors etc. embedded in source files 3. timezone sensitive code 4. directory order/build order 5. non-sanitized data stored into files 6. symbolic links/paths 7. general tool inconsistencies 8. toolchain 9. build information / tunables / environment 10. making sure that the source tree has no local changes With a bit of info on each.
vacri 10y ago
That's exactly what the article is about.

jrcii 10y ago

Of all the great OSes, NetBSD seems to get the least fanfare but it's always been my favorite. It's fast, secure, and due to a well reasoned architecture it works on everything. pkgsrc is amazing. They also have a very helpful, friendly community. It especially warms my heart to run `ps -ax` on a new install and see all of about 10 processes. The OS feels lean, neat, and organized, and I feel like I know exactly what is going on, where to find a given file, etc. These special strengths -- vast hardware compatibility, rump kernels, now full reproducible builds, are all enabled by a greater underlying (and seemingly underrated) technical excellence.

geff82 10y ago
Also run "top" on the fresh install to see it uses about 10MB :D . I learned how to use Unix when I installed and used NetBSD for some years, so for me it is "Mother-UNIX". It's a pity that general hardware support is not that great anymore nowadays. But it runs quite well on the Lenovo X240/T440 range of laptops which can still be had new in some places.

hendry 10y ago

If you care about getting on the Web safely, Webconverger has reproducible builds at https://build.webconverger.com/ https://build.webconverger.com/

dolzenko 10y ago

I know, it's unrelated, but its 2017 and > Unfortunately this was not easy to find on NetBSD, because we are still using CVS as the source control system seems just weird.

gumby 10y ago

At Cygnus we had a customer from the telecom industry. They had SLAs with their own customers that included terms like "no more than 10 minutes of downtime per decade". They paid a LOT of money to have one, consistent release (no upgrades, only bug fixes); when they reported a big and got a fix they would diff the new binary and required that every change could be traced solely to the patch issued a nothing else. Satisfying this made GCC a lot better.

ksec 10y ago

Forgive my Ignorance, OpenBSD - Absolute Security FreeBSD - General purpose What is NetBSD aiming at?

smhenderson 10y ago
Maximum portability... "Of course it runs NetBSD" From their website: NetBSD is a free, fast, secure, and highly portable Unix-like Open Source operating system. It is available for a wide range of platforms, from large-scale servers and powerful desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent for use in both production and research environments, and the source code is freely available under a business-friendly license.