3 ms·

First of all, this is why reproducible builds (getting bit-for-bit identical binaries independent of the machine used to build the software) is something we sho

by cyphar 9y ago

First of all, this is why reproducible builds (getting bit-for-bit identical binaries independent of the machine used to build the software) is something we should be putting much more work into. The Debian folks are doing an amazing job there. Another important point is that distributions have already solved effectively all of these problems. We have automated building and signing systems that mean that installation and upgrades are done in ways that are not vulnerable even to fairly sophisticated attacks. You can build packages locally if you want to verify them, and modifying a package after it has been built invalidates the signature that all modern package managers require before installing a package. As part of the openSUSE project we even have a free-to-use (and free as in freedom) build project called the Open Build Service[1] which allows you to build packages (with automated dependency update rebuilds) for many different distributions (Arch Linux, Debian, Ubuntu, Fedora, RHEL and obviously openSUSE and SLES). I get that distributions aren't "sexy" but it's getting quite frustrating seeing all these communities make the same mistakes that distributions made (and learned from) more than 20 years ago. [Disclaimer: I work for SUSE and am an openSUSE community member.] [1]: https://build.opensuse.org/ https://build.opensuse.org/

cm2187 9y ago
That's ok for the NSA and a few large enterprises to check certain binaries but you can't ask any start up or small company to recompile themselves every version of every OS / Software / Library that they use. It's not really a practical solution (even assuming everything is open source).
- cyphar 9y ago
  As I posted above, we have a build service that does this automatically for you. It takes maybe 30 minutes to create a few spec files, and now it's supported by a bunch of different distributions. You can then point your users at the repo (or mirror the repo if you prefer). Also OBS lets you create forks of projects that just inherit the unmodified packages (and said forks can also be cross-host so you don't need to self-host a whole distribution). I agree that doing everything I mentioned manually is hard, that's again why I said that distributions have solved this problem and made it easy.
  - cm2187 9y ago
    But who would use the build service? The consumer of the software or the publisher? [edit] also that protects you against malware injecting binaries in an executable when compiling it, but not from malware injecting code into the source code of the executable.
    - cyphar 9y ago
      > But who would use the build service? The consumer of the software or the publisher? Publisher. A user could (if they were really paranoid) rebuild the packages locally, with two or three commands. > Also that protects you against malware injecting binaries in an executable when compiling it, but not from malware injecting code into the source code of the executable. You can download the source code that OBS used (both as a src RPM generated by the builder and the OBS repo that the builder was given read-only access to), and OBS supports cryptographic signatures of the originating source (with gpg-offline keys to avoid WoT attacks). If your developers are using sane source control practices (use GPG keys for every commit, but especially tags) then you are protected against that too. Of course, reproducible builds is something that would solve this problem even better (protecting against attacks on OBS that cause it to add source that are not in the repo). As a side point, our threat model doesn't fully trust the nodes compiling the software so such attacks are fairly limited in scope (but I'm not a developer of OBS so I'm really not the right person to be asked these questions).
- sdwisely 9y ago
  compile times are fast enough now that if the tools were friendly enough I don't know it would be so inconvenient to the end user anymore. is there any distros out there now that are source based like Gentoo with more friendly defaults?
  - Filligree 9y ago
    There's NixOS. By default it uses a binary cache, but it'd be easy to disable that if you want to. And reproducible builds are a work in progress.
  - emodendroket 9y ago
    And why exactly can't the source be compromised?
- pmontra 9y ago
  I don't know what they're planning to do, but we could need only a few parties doing the verifications. If a build yields the wrong binary then the release is flagged and nobody gets it. Apt should check against the expected results. Many things could go wrong with this (mainly attacks on the expected results db, it should be replicated) but the idea should work.
  - cm2187 9y ago
    The thing is if the threat is any machine on the publisher side being infected, then it sorts of need to be verified by the consumer of the software, a bit like compiling a checksum when you download software.
    - danieldk 9y ago
      Debian packages are signed. It is hard to make software distribution completely secure. But, one could imagine creating pretty strong guarantees. Imagine that you have N completely different parties/machines (isolated, on different infrastructure, different user accounts). - (N-1) machines build a new Debian package. - The (N-1) results are fetched by the fourth party. - The Nth party checks whether the build results are identical. - The Nth party signs the result with GPG on a machine that is not connected. - The signed package is distributed. An attacker would need to compromise either: (1) N-1 build machines; (2) the offline machine used for signing packages; or (3) the upstream source. I consider (3) to be the most serious thread. But in this scenario, only the distribution packagers need to inspect the source changes. In the common scenario where you download from a vendor (e.g. the Transmission or CCleaner website), every user has to inspect a binary blob. Another possibility would be to have strong sandboxing for applications. An application could still participate in DDoS attacks, etc. But it would at least not encrypt/destroy your data. tl;dr: reproducible builds are an extremely important development and Debian (and other distributions participating in this initiative) should be commended for their work!
  - cyphar 9y ago
    Right, and this is why we need reproducible builds.

by cyphar 9y ago

cm2187 9y ago

That's ok for the NSA and a few large enterprises to check certain binaries but you can't ask any start up or small company to recompile themselves every version of every OS / Software / Library that they use. It's not really a practical solution (even assuming everything is open source).

cyphar 9y ago
As I posted above, we have a build service that does this automatically for you. It takes maybe 30 minutes to create a few spec files, and now it's supported by a bunch of different distributions. You can then point your users at the repo (or mirror the repo if you prefer). Also OBS lets you create forks of projects that just inherit the unmodified packages (and said forks can also be cross-host so you don't need to self-host a whole distribution). I agree that doing everything I mentioned manually is hard, that's again why I said that distributions have solved this problem and made it easy.
- cm2187 9y ago
  But who would use the build service? The consumer of the software or the publisher? [edit] also that protects you against malware injecting binaries in an executable when compiling it, but not from malware injecting code into the source code of the executable.
  - cyphar 9y ago
    > But who would use the build service? The consumer of the software or the publisher? Publisher. A user could (if they were really paranoid) rebuild the packages locally, with two or three commands. > Also that protects you against malware injecting binaries in an executable when compiling it, but not from malware injecting code into the source code of the executable. You can download the source code that OBS used (both as a src RPM generated by the builder and the OBS repo that the builder was given read-only access to), and OBS supports cryptographic signatures of the originating source (with gpg-offline keys to avoid WoT attacks). If your developers are using sane source control practices (use GPG keys for every commit, but especially tags) then you are protected against that too. Of course, reproducible builds is something that would solve this problem even better (protecting against attacks on OBS that cause it to add source that are not in the repo). As a side point, our threat model doesn't fully trust the nodes compiling the software so such attacks are fairly limited in scope (but I'm not a developer of OBS so I'm really not the right person to be asked these questions).
sdwisely 9y ago
compile times are fast enough now that if the tools were friendly enough I don't know it would be so inconvenient to the end user anymore. is there any distros out there now that are source based like Gentoo with more friendly defaults?
- Filligree 9y ago
  There's NixOS. By default it uses a binary cache, but it'd be easy to disable that if you want to. And reproducible builds are a work in progress.
- emodendroket 9y ago
  And why exactly can't the source be compromised?
pmontra 9y ago
I don't know what they're planning to do, but we could need only a few parties doing the verifications. If a build yields the wrong binary then the release is flagged and nobody gets it. Apt should check against the expected results. Many things could go wrong with this (mainly attacks on the expected results db, it should be replicated) but the idea should work.
- cm2187 9y ago
  The thing is if the threat is any machine on the publisher side being infected, then it sorts of need to be verified by the consumer of the software, a bit like compiling a checksum when you download software.
  - danieldk 9y ago
    Debian packages are signed. It is hard to make software distribution completely secure. But, one could imagine creating pretty strong guarantees. Imagine that you have N completely different parties/machines (isolated, on different infrastructure, different user accounts). - (N-1) machines build a new Debian package. - The (N-1) results are fetched by the fourth party. - The Nth party checks whether the build results are identical. - The Nth party signs the result with GPG on a machine that is not connected. - The signed package is distributed. An attacker would need to compromise either: (1) N-1 build machines; (2) the offline machine used for signing packages; or (3) the upstream source. I consider (3) to be the most serious thread. But in this scenario, only the distribution packagers need to inspect the source changes. In the common scenario where you download from a vendor (e.g. the Transmission or CCleaner website), every user has to inspect a binary blob. Another possibility would be to have strong sandboxing for applications. An application could still participate in DDoS attacks, etc. But it would at least not encrypt/destroy your data. tl;dr: reproducible builds are an extremely important development and Debian (and other distributions participating in this initiative) should be commended for their work!
- cyphar 9y ago
  Right, and this is why we need reproducible builds.