5 ms·

But the xz backdoor didn’t involve a build script that tried to compromise the machine the build was running on. It involved a build script that compromised th

by comex 3y ago

But the xz backdoor didn’t involve a build script that tried to compromise the machine the build was running on. It involved a build script that compromised the code being built. Sandboxing the build script wouldn’t have helped much if at all. Depending on the implementation, it might have prevented it from overwriting .o files that were already compiled, maybe. But there would still be all sorts of shenanigans it could have played to sneakily inject code. I’d argue that Rust’s biggest advantage here is that build scripts and procedural macros are written in Rust themselves, making them easier to read and thus harder to hide things in than autotools’ m4-to-shell gunk. But that means it’s important to build those things from source! Part of the movement you cite consists of watt, which ships proc macros as precompiled WebAssembly binaries mainly to save build time. But if watt were more widely adopted, it would actually make an attacker’s life much easier since they could backdoor the precompiled binary. Historically I’ve been sympathetic to dtolnay’s various attempts to use precompiled binaries this way (not just watt but also the serde thing that caused a hullabaloo); I hate slow builds. But after the xz backdoor I think this is untenable.

noctune 3y ago
The xz backdoor did hide the payload in test data though. Proper sandboxing could have meant that eg. building the binary wouldn't have access to test data.
mjw1007 3y ago
Maybe not untenable. If everything is done carefully enough with reproducible builds, I think using a binary whose hash can be checked shouldn't be a great extension of trust. You could have multiple independent autobuilders verifying that particular source does indeed generate a binary with the claimed hash.
tadfisher 3y ago
Ultimately you're going to have to be adept at stuff like the Underhanded C Contest to spot this kind of thing in any Turing-complete language, so the idea of auditing the source is unreliable at worst. So I'd take another page from the Java/Maven ecosystem and require hashed+signed binaries, with the possible addition of requiring builds to be performed on a trusted remote host so that at least we can verify the binary is produced from the same source and results in the same output. But determined actors with access are always going to try to thwart these schemes, so verification and testing is going to need to step up.
- naikrovek 3y ago
  reproducible builds never made sense to me. if you trust the person giving you the hash, just get the binary from them. you don't need to reproduce the build at all. if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.
  - emn13 3y ago
    One thing a culture of reproducible builds (and thus stable hashes) however does provide is a lack of excuse as to why the build isn't reproducible. Almost nobody will build from source - but when there's bugs to be squashed and weird behavior, some people, sometimes, will. If hashes are the norm, then it's a little harder for attackers to pull of this kind of thing not because you trust their hash rather than their blob, but rather because they need to publish both at once - thus broadening the discovery window for shenanigans. To put it another way: if you don't have reproducible builds and you're trying to replicate an upstream artifact then it's very hard to tell what the cause is. It might just be some ephemeral state, a race, or some machine aspect that caused a few fairly meaningless differences. But as soon as you have a reproducible build, then failure to reproduce instantly marks upstream as being suspect. It's also useful when you're trying to do things like tweak a build - you can ensure you're importing upstream correctly by checking the hash with what you're making, and then even if your downstream version isn't a binary dependent on upstream (e.g. optimized differently, or with some extra plugins somewhere), you can be sure that changes between what you're building and upstream are intentional and not spurious. It's clearly not a silver bullet, sure. But it's not entirely useless either, and it probably could help as part of a larger ecosystem shift; especially if conventional tooling created and published these by default such that bad actors trying to hide build processes actually stick out more glaringly.
  - rstuart4133 3y ago
    Actually, this xz-utils was a sort of reproducible build issue. But the twist it is wasn't the binary not built reproducibly. It was the tar ball. The natural assumption is it just reflected the public git repository. It didn't. Debian's response is looking to be mandating the source must come from the git repository, not a tar ball. And it will be done using a script. And the script must produce the same output every time, ie be reproducible. Currently Debian's idea of "reproducible" means it reflects the source Debian distributes. This change means it will reproducibly reflect the upstream sources. That doesn't mean it will be the same - just that it's derived in a reproducible way. As for trusting the person who gave you the hash: that's not what it hinges on. It hinges on the majority installing a binary from a distro. That binary can be produced reproducibly, by an automated build system. Even now, Debian's binaries aren't created just once by Debian. There are Debian builders all around the planet creating the binary, and verifying it matches what's being distributed. Thus the odds of the binary you are running not being derived from the source are vanishingly low.

by comex 3y ago

noctune 3y ago

The xz backdoor did hide the payload in test data though. Proper sandboxing could have meant that eg. building the binary wouldn't have access to test data.

mjw1007 3y ago

Maybe not untenable. If everything is done carefully enough with reproducible builds, I think using a binary whose hash can be checked shouldn't be a great extension of trust. You could have multiple independent autobuilders verifying that particular source does indeed generate a binary with the claimed hash.

tadfisher 3y ago

Ultimately you're going to have to be adept at stuff like the Underhanded C Contest to spot this kind of thing in any Turing-complete language, so the idea of auditing the source is unreliable at worst. So I'd take another page from the Java/Maven ecosystem and require hashed+signed binaries, with the possible addition of requiring builds to be performed on a trusted remote host so that at least we can verify the binary is produced from the same source and results in the same output. But determined actors with access are always going to try to thwart these schemes, so verification and testing is going to need to step up.

naikrovek 3y ago
reproducible builds never made sense to me. if you trust the person giving you the hash, just get the binary from them. you don't need to reproduce the build at all. if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.
- emn13 3y ago
  One thing a culture of reproducible builds (and thus stable hashes) however does provide is a lack of excuse as to why the build isn't reproducible. Almost nobody will build from source - but when there's bugs to be squashed and weird behavior, some people, sometimes, will. If hashes are the norm, then it's a little harder for attackers to pull of this kind of thing not because you trust their hash rather than their blob, but rather because they need to publish both at once - thus broadening the discovery window for shenanigans. To put it another way: if you don't have reproducible builds and you're trying to replicate an upstream artifact then it's very hard to tell what the cause is. It might just be some ephemeral state, a race, or some machine aspect that caused a few fairly meaningless differences. But as soon as you have a reproducible build, then failure to reproduce instantly marks upstream as being suspect. It's also useful when you're trying to do things like tweak a build - you can ensure you're importing upstream correctly by checking the hash with what you're making, and then even if your downstream version isn't a binary dependent on upstream (e.g. optimized differently, or with some extra plugins somewhere), you can be sure that changes between what you're building and upstream are intentional and not spurious. It's clearly not a silver bullet, sure. But it's not entirely useless either, and it probably could help as part of a larger ecosystem shift; especially if conventional tooling created and published these by default such that bad actors trying to hide build processes actually stick out more glaringly.
- rstuart4133 3y ago
  Actually, this xz-utils was a sort of reproducible build issue. But the twist it is wasn't the binary not built reproducibly. It was the tar ball. The natural assumption is it just reflected the public git repository. It didn't. Debian's response is looking to be mandating the source must come from the git repository, not a tar ball. And it will be done using a script. And the script must produce the same output every time, ie be reproducible. Currently Debian's idea of "reproducible" means it reflects the source Debian distributes. This change means it will reproducibly reflect the upstream sources. That doesn't mean it will be the same - just that it's derived in a reproducible way. As for trusting the person who gave you the hash: that's not what it hinges on. It hinges on the majority installing a binary from a distro. That binary can be produced reproducibly, by an automated build system. Even now, Debian's binaries aren't created just once by Debian. There are Debian builders all around the planet creating the binary, and verifying it matches what's being distributed. Thus the odds of the binary you are running not being derived from the source are vanishingly low.