6 ms·

A vulnerability not mentioned in the article is the normalisation of executing code that has been especially targeted to a specific user or specific device with

by dhx 2y ago

A vulnerability not mentioned in the article is the normalisation of executing code that has been especially targeted to a specific user or specific device with no validation of reproducibility and no ability for anyone to verify this custom build and download service hasn't been generating backdoored builds. One should want to ensure use of the same build of xz-utils that Andres Freund is using, or at least a build of xz-utils that other security researchers can later obtain to figure out whether supply chain implants are present in open source software[1]. There's a write up at Mozilla[2] from years ago describing an abandoned attempt by Mozilla to ensure their release builds are publicly logged in a Merkle tree. Google has written up their implementation for Pixel firmware builds but apps delivered through the Google Play Store seem to be vulnerable (unless there is another log I have been unable to find).[3] Apple is seemingly worse than Google on binary transparency with Apple's firmware and app distribution system targeting builds to individual devices with no transparency of builds. For an example of binary transparency done well, Gentoo's ebuild repository (being a single Git repository/Merkle tree containing source checksums) possibly remains the largest and most distributed Merkle trees of open source software. [1] Post xz-utils backdoor, some researchers (including some posting to oss-security about their efforts) undertook automated/semi-automated scans of open source software builds to check for unexplained high entropy files which could contain hidden malicious code. This is not possible to achieve with customised per-user/per-device builds unless every single build is made publicly available for later analysis and a public log (Merkle tree) accompanies those published builds. [2] https://wiki.mozilla.org/Security/Binary_Transparency https://wiki.mozilla.org/Security/Binary_Transparency [3] https://developers.google.com/android/binary_transparency/overview https://developers.google.com/android/binary_transparency/ov...

dist-epoch 2y ago
> automated/semi-automated scans of open source software builds to check for unexplained high entropy files which could contain hidden malicious code that's easily defeated though, you just "spread-out" the entropy.
- dhx 2y ago
  It's easy to defeat right now because very few are currently thinking about secure build systems. As an example, systemd's latest source tarball has two Web Open Font Format (WOFF) files in a documentation folder, a favicon.png, two few small screenshots and error messages that have unexplained 128bit identifiers. There are also translated strings (PO files) which could include obscure writing systems few can quickly audit, and thus could be advantageous to an attacker wanting to hide malicious code. The problem with most build systems is the entire source tarball is extracted into a single location and any build scripts executed during build have access to everything in the source tarball. Gentoo's portage does apply sandboxing around an entire build process, just not internally between the different build stages. Continuing the Gentoo example (being one of the better examples of sandboxed builds), ideally src_unpack could take a compressed tarball distributed from an upstream project and split files into multiple separate paths such as: - source code files and build scripts needed to build binaries - supporting test files needed to test the the built binaries - supporting documentation/binary/high entropy data such as application icons, game data, documentation, etc that should be copied upon installation but aren't required to build binaries Then src_prepare, src_configure, src_compile, src_test and src_install all have different sandbox configurations restricting which types of files separated during src_unpack each build phase can interact with. For the systemd example above, some possible improvements could be: 1. Remove WOFF font files and use system default fonts, omit favicon.png and omit screenshots. Or only make WOFF font files, favicon.png and screenshots available for copying during the src_install phase, and ensure they are not readable by build scripts during src_configure and src_compile. 2. Generate error message identifiers using an explained approach such as hash_algorithm("systemd_error_" + name_of_error_constant) to generate nothing-up-my-sleeve identifiers. 3. Only provide access to and include in the build process any translation files of languages the user cares about. Or only make translation files available for copying during the src_install phase and possibly src_test too, and ensure they are not readable by build scripts. These build system security techniques are obviously more work, but are generally straightforward to understand and implement. These technqiues are in the realm of possibility for smaller embedded Linux systems that may just be kernel + BusyBox + a few small scripts + a bespoke application + a handful of dynamic libraries. And for more complex Linux systems, these techniques are within the realm of possibility when targeted towards high value software, such as software requiring or often executed with root permissions and software requiring simultaneous access to the Internet and access to user files.
jojobas 2y ago
Using a build service like that is apriori saying "i'm not valuable enough for a targeted attack".
tristor 2y ago
This is a nice idea, and one I also advocate for, however it's important to keep in mind that the idea of reproducibility relies on determinism. So much of what goes into a build pipeline is inherently nondeterministic, because we're making decisions at compile time which can differ from compilation run to compilation run, setting aside flags. In fact, that's the point of an optimizing compiler, as many reproducible build projects have discovered, turning on optimizations pretty much guarantees no reproducibility.
- actionfromafar 2y ago
  As long as the compiler is not optimizing by "let's give this 3 seconds of solving time, then continue if no better solution is found", then optimizing is not inherently nondeterministic.
- CorrectHorseBat 2y ago
  Counterpoint: Archlinux is 89% reproducible with optimizations enabled. The only thing I see which is difficult to make reproducible is optimizations with a timeout.
  - not2b 2y ago
    Instead of using a timeout, an optimization that can must be cut off if the cost is excessive can keep some kind of operation or size count, where the count is strictly a function of the input. For example, an optimization based on binary decision diagrams (BDDs) can put a ceiling on the number of nodes in the BDD.
- Groxx 2y ago
  "Reproducible" isn't necessary for "not modified from what everyone else gets", and that still makes some attacks FAR harder (and easier to identify, as you know what the "normal" one is). And a published Merkle tree just makes it easier to verify "none of this has changed", as opposed to SHAs on a website that could change any time.
  - tristor 2y ago
kccqzy 2y ago
Yeah I remember Google's certificate transparency team basically designing firmware transparency for all of Linux (not just Android) as well.
philjglass 2y ago
For Google Play: https://developer.android.com/guide/app-bundle/code-transparency https://developer.android.com/guide/app-bundle/code-transpar... As far as I know there's no centralised log, it's left up to app developers to publish their key/a log of transparency files.

by dhx 2y ago

dist-epoch 2y ago

> automated/semi-automated scans of open source software builds to check for unexplained high entropy files which could contain hidden malicious code that's easily defeated though, you just "spread-out" the entropy.

dhx 2y ago
It's easy to defeat right now because very few are currently thinking about secure build systems. As an example, systemd's latest source tarball has two Web Open Font Format (WOFF) files in a documentation folder, a favicon.png, two few small screenshots and error messages that have unexplained 128bit identifiers. There are also translated strings (PO files) which could include obscure writing systems few can quickly audit, and thus could be advantageous to an attacker wanting to hide malicious code. The problem with most build systems is the entire source tarball is extracted into a single location and any build scripts executed during build have access to everything in the source tarball. Gentoo's portage does apply sandboxing around an entire build process, just not internally between the different build stages. Continuing the Gentoo example (being one of the better examples of sandboxed builds), ideally src_unpack could take a compressed tarball distributed from an upstream project and split files into multiple separate paths such as: - source code files and build scripts needed to build binaries - supporting test files needed to test the the built binaries - supporting documentation/binary/high entropy data such as application icons, game data, documentation, etc that should be copied upon installation but aren't required to build binaries Then src_prepare, src_configure, src_compile, src_test and src_install all have different sandbox configurations restricting which types of files separated during src_unpack each build phase can interact with. For the systemd example above, some possible improvements could be: 1. Remove WOFF font files and use system default fonts, omit favicon.png and omit screenshots. Or only make WOFF font files, favicon.png and screenshots available for copying during the src_install phase, and ensure they are not readable by build scripts during src_configure and src_compile. 2. Generate error message identifiers using an explained approach such as hash_algorithm("systemd_error_" + name_of_error_constant) to generate nothing-up-my-sleeve identifiers. 3. Only provide access to and include in the build process any translation files of languages the user cares about. Or only make translation files available for copying during the src_install phase and possibly src_test too, and ensure they are not readable by build scripts. These build system security techniques are obviously more work, but are generally straightforward to understand and implement. These technqiues are in the realm of possibility for smaller embedded Linux systems that may just be kernel + BusyBox + a few small scripts + a bespoke application + a handful of dynamic libraries. And for more complex Linux systems, these techniques are within the realm of possibility when targeted towards high value software, such as software requiring or often executed with root permissions and software requiring simultaneous access to the Internet and access to user files.

jojobas 2y ago

Using a build service like that is apriori saying "i'm not valuable enough for a targeted attack".

tristor 2y ago

This is a nice idea, and one I also advocate for, however it's important to keep in mind that the idea of reproducibility relies on determinism. So much of what goes into a build pipeline is inherently nondeterministic, because we're making decisions at compile time which can differ from compilation run to compilation run, setting aside flags. In fact, that's the point of an optimizing compiler, as many reproducible build projects have discovered, turning on optimizations pretty much guarantees no reproducibility.

actionfromafar 2y ago
As long as the compiler is not optimizing by "let's give this 3 seconds of solving time, then continue if no better solution is found", then optimizing is not inherently nondeterministic.
CorrectHorseBat 2y ago
Counterpoint: Archlinux is 89% reproducible with optimizations enabled. The only thing I see which is difficult to make reproducible is optimizations with a timeout.
- not2b 2y ago
  Instead of using a timeout, an optimization that can must be cut off if the cost is excessive can keep some kind of operation or size count, where the count is strictly a function of the input. For example, an optimization based on binary decision diagrams (BDDs) can put a ceiling on the number of nodes in the BDD.
Groxx 2y ago
"Reproducible" isn't necessary for "not modified from what everyone else gets", and that still makes some attacks FAR harder (and easier to identify, as you know what the "normal" one is). And a published Merkle tree just makes it easier to verify "none of this has changed", as opposed to SHAs on a website that could change any time.
- tristor 2y ago

kccqzy 2y ago

Yeah I remember Google's certificate transparency team basically designing firmware transparency for all of Linux (not just Android) as well.

philjglass 2y ago

For Google Play: https://developer.android.com/guide/app-bundle/code-transparency https://developer.android.com/guide/app-bundle/code-transpar... As far as I know there's no centralised log, it's left up to app developers to publish their key/a log of transparency files.