4 ms·

My big takeaway is that modern build systems are fundamentally broken. Autotools, even Makefiles, are (or can be) incredibly obtuse, even unreadable. If I read

by cletus 3y ago

My big takeaway is that modern build systems are fundamentally broken. Autotools, even Makefiles, are (or can be) incredibly obtuse, even unreadable. If I read this attack correctly, it relied on a payload in a test file (to obfuscate it). It shouldn't be possible to include test resources in a production build. As an aside, C/C++'s header system with conditional inclusion is also fundamentally broken. Even templates are just text substitution with a thin veneer of typing. I think about Google's build system, which is very much designed to avoid this kind of thing. The internal build tool is Blaze (Bazel is the open-source cousin). Many years ago, you could essentially write scripts in your BUILD files (called genrules) that were hugely problematic. There was no way to guarantee the output so they had to be constantly rebuilt. There was a long project to eliminate this kind of edge case. Blaze (and Bazel) are built around declaring hermetic units to be built with explicit dependencies only. Nothing shipped to production is built locally. It's all built by the build servers (a system called Forge). These outputs are packaged into Midas packages ("MPMs"). You could absolutely reconstruct the source used to build a particular library, binary or package as well as the build toolchain and version used. And any build is completely deterministic and verifiable. C/C++, Make, CMake, autotools, autoconf and all that tooling so common in Linux and its core dependencies absolutely needs to go.

__MatrixMan__ 3y ago
If I were starting something from scratch I'd do Bazel for internal deps, Nix for external ones. If that's a tall order, so be it: we train people. Investments in determinism usually end up paying off.
- threePointFive 3y ago
  I'm not familiar with Bazel, but Nix in it's current form wouldn't have solved this attack. First of all, the standard mkDerivation function calls the same configure; make; make install process that made this attack possible. Nixpkgs regularly pulls in external resources (fetchUrl and friends) that are equally vulnerable to a poisoned release tarball. Checkout the comment on the current xz entry in nixpkgs https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/compression/xz/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/comp...
  - diamondburned 3y ago
    If you're ever using Nix like you would Bazel, you likely would not want your derivation to be built via another Makefile. Indeed, that defeats the whole point of using Nix to fix this in the first place. As it is, mkDerivation + fetchurl is mostly used to build existing non-Nix software. Nix at the very least provides first-class(-ish) build support for modern languages like Rust, Go and Python, but I don't think anyone has written an actual Nix builder for C/C++. A combo of Bazel + Nix is fairly common, though. IMO, it's hard to say if Nix would "solve this attack", since we've only seen it being truly used on more modern things where the build complexity of the actual piece of code is not much more than the same few commands. As for pulling down a poisoned tarball, I think the discussion here is rather about upstream projects using Nix to guarantee reproducible builds rather than Nix trying to fend off attacks downstream in Nixpkgs. In modern Nix (with Flakes), this would look something like being able to clone and run `nix build .#` and end up with the same output every time.
- wocram 3y ago
  This sounds nice on paper, but there's a lot of missing glue to be written between nix and bazel. Ideally nix would move towards less imperative/genrule style package declarations and ultimately become more usable for internal builds.
sanderjd 3y ago
> modern build systems > Autotools, even Makefiles > C/C++'s header system with conditional inclusion Wouldn't it be more accurate to say something like "older build systems"? I don't think any of the things you listed are "modern". Which isn't a criticism of their legacy! They have been very useful for a long time, and that's to be applauded. But they have huge problems, which is a big part of why newer systems have been created. FWIW, I have been using pants[0] (v2) for a little under a year. We chose it after also evaluating it and bazel (but not nix, for better or worse). I think it's really really great! Also painful in some ways (as is inevitably the case with any software). And of course it's nearly impossible to entirely stomp out "genrules" use cases. But it's much easier to get much closer to true hermeticity, and I'm a big fan of that. 0: https://www.pantsbuild.org/ https://www.pantsbuild.org/
- justinpombrio 3y ago
  To be clear about the history: Make is from 1976, and Autotools is from 1991. Had me a chuckle about these being called modern, they're (some of?) the earliest build systems. https://en.wikipedia.org/wiki/Make_(software) https://en.wikipedia.org/wiki/Make_(software) https://en.wikipedia.org/wiki/Autoconf https://en.wikipedia.org/wiki/Autoconf
ants_everywhere 3y ago
> The internal build tool is Blaze (Bazel is the open-source cousin). I was under the impression that Blaze is Google's Bazel deployment, i.e. that they're the same code. Is that not correct?
- xen0 3y ago
  Mostly correct; there's some code that's Blaze only and some that's Bazel only, but the core is the same.
deleted 3y ago
[deleted]
dhx 3y ago
Ugly and broken build systems are most of the reason why Gentoo's 'sandbox' feature and equivalent for other distributions exists.[1] These sandboxing features have mostly been used in the past to prevent an ugly shell script in the build process for libuselesscruft from doing something similar to "rm -rf" on the build system. More recently these sandboxing features are helpful in encouraging reproducible builds by alerting maintainers to build processes which try and obtain non-deterministic information from the operating system environment such as the operating system name and version, host name and current time stamp. There are a few gaps I think xz-utils highlights: - Repositories containing a mixture of source code, build scripts, test frameworks, static resources and documentation generation scripts are all considered to be a single security domain with no isolation between them. If you look to Gentoo's src_prepare function as an example, we perhaps should instead split this into build_src_prepare, doc_src_prepare, test_src_prepare and install_src_prepare instead. If source code is being built and binaries generated, the sandboxed build directory should perhaps not contain test files and documentation generation scripts. If the package is being installed with "make install" (or equivalent) then static resources (such as a timezone database) should be available to copy to /usr/, but build scripts used to generate the binaries or documentation do not need to be available to "make install" (or equivalent). - Sandboxing used for package building hasn't traditionally been implemented for security reasons in the past. Sandboxing should perhaps be hardened further with modern and more complex approaches such as seccomp to further protect build systems from the likes of libbackdoored that are targeted towards package maintainers/Linux distribution build systems. As a further example to seccomp, Gentoo's 'sandbox' has Linux namespace isolation built in, but not yet enabled whilst it is tested. - There is a lack of automated package management tools (including dashboards / automatic bug creation) for comparing source trees in Git to released tarballs and making discrepancies more visible and easier for maintainers to review. - There is a lack of automatic package management tools (including dashboards / automatic bug creation) for detecting binary and high entropy files in source trees and confirming they are validly formatted (e.g. invalid tag in a TLV file format) and confirming that test and example files contain nothing-up-my-sleeve content. There has already been an accelerated move in recent years towards modern and safer build systems (such as meson and cargo) as 80's/90's C libraries get ripped out and replaced with modern Rust libraries, or other safer options. This is a lot of work that will take many years though, and many old 80's/90's C libraries and build systems will be needed for many more years to come. And for this reason, sandboxing/isolation/safety of old build systems seemingly needs to be improved as a priority, noting that old build systems will take years or decades to replace. [1] https://devmanual.gentoo.org/general-concepts/sandbox/index.html https://devmanual.gentoo.org/general-concepts/sandbox/index.... [2] https://devmanual.gentoo.org/ebuild-writing/functions/src_prepare/index.html https://devmanual.gentoo.org/ebuild-writing/functions/src_pr...
ufmace 3y ago
You're not wrong, but IMO that isn't the real problem. The real problem is the combination of highly popular core open-source utilities with near-zero maintenance resources. This was all possible because XZ, despite being effectively everywhere, has one actual part-time maintainer who had other things going on in his life. That also means that there's nowhere near enough resources to redo the build system to some more secure alternative. If they had, say, 10 enthusiastic and skilled volunteer contributors with plenty of free time, they could do that, but then a new person appearing out of nowhere with a few helpful commits would never have a chance at being made an official maintainer or sneaking sketchy tools and code past the other maintainers. Not that I'm blaming XZ or the real maintainer. Clearly whoever was behind this was looking for the weakest link, and if it wasn't XZ, it would have been something else. The real problem is the culture. So I guess what this really means is someone at a big corporation making Linux distros should audit their full dependency tree. Any tool in that tree that isn't actively maintained, say at least 3 long-time active contributors, they should take over one way or another - whether that's hiring the current maintainer as a full-time remote employee, offering to buy out the rights, or forking and running under their own team. I'm not necessarily super thrilled with that, but I guess it's the world we live in now.
- bhawks 3y ago
  The overworked solo maintainer is a problem. However I've seen way too many projects where individuals in the 'team' are able to carve out impenetrable fiefdoms where they can operate with wide latitude. I could see a Jia Tan being able to pull this off in a team context as well - bigger teams might even be weaker. (Everyone welcome Jia - he's going to write test cases and optimize our build process so everyone can focus on $LAUNCH_DAY)
- pch00 3y ago
  > So I guess what this really means is someone at a big corporation making Linux distros should audit their full dependency tree. This is it precisely. When you're paying Redhat for an "enterprise" Linux then that guarantee should extend down their entire software stack. Just getting the odd backported patch and so-so email support no longer cuts it.
- 3y ago
anthk 3y ago
Makefiles are not a Linux thing, but Unix. And if you confuse C with C++ and Make with CMake/Autotools/Autoconf, you have a lot to learn. Look: simple and portable makefiles work: git://bitreich.org/english_knight
peteradio 3y ago
Google gets to decide what platform they want to support for their backend. I wonder what their build code would look like if they tried to support the top-10 common OS from the last 30 years.

by cletus 3y ago

__MatrixMan__ 3y ago

If I were starting something from scratch I'd do Bazel for internal deps, Nix for external ones. If that's a tall order, so be it: we train people. Investments in determinism usually end up paying off.

threePointFive 3y ago
I'm not familiar with Bazel, but Nix in it's current form wouldn't have solved this attack. First of all, the standard mkDerivation function calls the same configure; make; make install process that made this attack possible. Nixpkgs regularly pulls in external resources (fetchUrl and friends) that are equally vulnerable to a poisoned release tarball. Checkout the comment on the current xz entry in nixpkgs https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/compression/xz/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/comp...
- diamondburned 3y ago
  If you're ever using Nix like you would Bazel, you likely would not want your derivation to be built via another Makefile. Indeed, that defeats the whole point of using Nix to fix this in the first place. As it is, mkDerivation + fetchurl is mostly used to build existing non-Nix software. Nix at the very least provides first-class(-ish) build support for modern languages like Rust, Go and Python, but I don't think anyone has written an actual Nix builder for C/C++. A combo of Bazel + Nix is fairly common, though. IMO, it's hard to say if Nix would "solve this attack", since we've only seen it being truly used on more modern things where the build complexity of the actual piece of code is not much more than the same few commands. As for pulling down a poisoned tarball, I think the discussion here is rather about upstream projects using Nix to guarantee reproducible builds rather than Nix trying to fend off attacks downstream in Nixpkgs. In modern Nix (with Flakes), this would look something like being able to clone and run `nix build .#` and end up with the same output every time.
wocram 3y ago
This sounds nice on paper, but there's a lot of missing glue to be written between nix and bazel. Ideally nix would move towards less imperative/genrule style package declarations and ultimately become more usable for internal builds.

sanderjd 3y ago

> modern build systems > Autotools, even Makefiles > C/C++'s header system with conditional inclusion Wouldn't it be more accurate to say something like "older build systems"? I don't think any of the things you listed are "modern". Which isn't a criticism of their legacy! They have been very useful for a long time, and that's to be applauded. But they have huge problems, which is a big part of why newer systems have been created. FWIW, I have been using pants[0] (v2) for a little under a year. We chose it after also evaluating it and bazel (but not nix, for better or worse). I think it's really really great! Also painful in some ways (as is inevitably the case with any software). And of course it's nearly impossible to entirely stomp out "genrules" use cases. But it's much easier to get much closer to true hermeticity, and I'm a big fan of that. 0: https://www.pantsbuild.org/ https://www.pantsbuild.org/

justinpombrio 3y ago
To be clear about the history: Make is from 1976, and Autotools is from 1991. Had me a chuckle about these being called modern, they're (some of?) the earliest build systems. https://en.wikipedia.org/wiki/Make_(software) https://en.wikipedia.org/wiki/Make_(software) https://en.wikipedia.org/wiki/Autoconf https://en.wikipedia.org/wiki/Autoconf

ants_everywhere 3y ago

> The internal build tool is Blaze (Bazel is the open-source cousin). I was under the impression that Blaze is Google's Bazel deployment, i.e. that they're the same code. Is that not correct?

xen0 3y ago
Mostly correct; there's some code that's Blaze only and some that's Bazel only, but the core is the same.

deleted 3y ago

[deleted]

dhx 3y ago

Ugly and broken build systems are most of the reason why Gentoo's 'sandbox' feature and equivalent for other distributions exists.[1] These sandboxing features have mostly been used in the past to prevent an ugly shell script in the build process for libuselesscruft from doing something similar to "rm -rf" on the build system. More recently these sandboxing features are helpful in encouraging reproducible builds by alerting maintainers to build processes which try and obtain non-deterministic information from the operating system environment such as the operating system name and version, host name and current time stamp. There are a few gaps I think xz-utils highlights: - Repositories containing a mixture of source code, build scripts, test frameworks, static resources and documentation generation scripts are all considered to be a single security domain with no isolation between them. If you look to Gentoo's src_prepare function as an example, we perhaps should instead split this into build_src_prepare, doc_src_prepare, test_src_prepare and install_src_prepare instead. If source code is being built and binaries generated, the sandboxed build directory should perhaps not contain test files and documentation generation scripts. If the package is being installed with "make install" (or equivalent) then static resources (such as a timezone database) should be available to copy to /usr/, but build scripts used to generate the binaries or documentation do not need to be available to "make install" (or equivalent). - Sandboxing used for package building hasn't traditionally been implemented for security reasons in the past. Sandboxing should perhaps be hardened further with modern and more complex approaches such as seccomp to further protect build systems from the likes of libbackdoored that are targeted towards package maintainers/Linux distribution build systems. As a further example to seccomp, Gentoo's 'sandbox' has Linux namespace isolation built in, but not yet enabled whilst it is tested. - There is a lack of automated package management tools (including dashboards / automatic bug creation) for comparing source trees in Git to released tarballs and making discrepancies more visible and easier for maintainers to review. - There is a lack of automatic package management tools (including dashboards / automatic bug creation) for detecting binary and high entropy files in source trees and confirming they are validly formatted (e.g. invalid tag in a TLV file format) and confirming that test and example files contain nothing-up-my-sleeve content. There has already been an accelerated move in recent years towards modern and safer build systems (such as meson and cargo) as 80's/90's C libraries get ripped out and replaced with modern Rust libraries, or other safer options. This is a lot of work that will take many years though, and many old 80's/90's C libraries and build systems will be needed for many more years to come. And for this reason, sandboxing/isolation/safety of old build systems seemingly needs to be improved as a priority, noting that old build systems will take years or decades to replace. [1] https://devmanual.gentoo.org/general-concepts/sandbox/index.html https://devmanual.gentoo.org/general-concepts/sandbox/index.... [2] https://devmanual.gentoo.org/ebuild-writing/functions/src_prepare/index.html https://devmanual.gentoo.org/ebuild-writing/functions/src_pr...

ufmace 3y ago

You're not wrong, but IMO that isn't the real problem. The real problem is the combination of highly popular core open-source utilities with near-zero maintenance resources. This was all possible because XZ, despite being effectively everywhere, has one actual part-time maintainer who had other things going on in his life. That also means that there's nowhere near enough resources to redo the build system to some more secure alternative. If they had, say, 10 enthusiastic and skilled volunteer contributors with plenty of free time, they could do that, but then a new person appearing out of nowhere with a few helpful commits would never have a chance at being made an official maintainer or sneaking sketchy tools and code past the other maintainers. Not that I'm blaming XZ or the real maintainer. Clearly whoever was behind this was looking for the weakest link, and if it wasn't XZ, it would have been something else. The real problem is the culture. So I guess what this really means is someone at a big corporation making Linux distros should audit their full dependency tree. Any tool in that tree that isn't actively maintained, say at least 3 long-time active contributors, they should take over one way or another - whether that's hiring the current maintainer as a full-time remote employee, offering to buy out the rights, or forking and running under their own team. I'm not necessarily super thrilled with that, but I guess it's the world we live in now.

bhawks 3y ago
The overworked solo maintainer is a problem. However I've seen way too many projects where individuals in the 'team' are able to carve out impenetrable fiefdoms where they can operate with wide latitude. I could see a Jia Tan being able to pull this off in a team context as well - bigger teams might even be weaker. (Everyone welcome Jia - he's going to write test cases and optimize our build process so everyone can focus on $LAUNCH_DAY)
pch00 3y ago
> So I guess what this really means is someone at a big corporation making Linux distros should audit their full dependency tree. This is it precisely. When you're paying Redhat for an "enterprise" Linux then that guarantee should extend down their entire software stack. Just getting the odd backported patch and so-so email support no longer cuts it.
3y ago

anthk 3y ago

Makefiles are not a Linux thing, but Unix. And if you confuse C with C++ and Make with CMake/Autotools/Autoconf, you have a lot to learn. Look: simple and portable makefiles work: git://bitreich.org/english_knight

peteradio 3y ago

Google gets to decide what platform they want to support for their backend. I wonder what their build code would look like if they tried to support the top-10 common OS from the last 30 years.