4 ms·

one thing that I think got lost culturally in the move to linux was the emphasis on open source. yes, not open source in some abstract warm fuzzy way, but actua

one thing that I think got lost culturally in the move to linux was the emphasis on open source. yes, not open source in some abstract warm fuzzy way, but actual useful open source. if I cant run the build process, and I dont really know which patch sets have been applied on my system, and which version of the source was used, and I dont really have a handle on the dependencies. doesn't make it substantially more difficult to debug and work with the resulting system? its really quite odd that the dominant open source platform settled on opaque and unreproducible binaries as the distribution mechanism early on.

jrockway 8y ago
> its really quite odd that the dominant open source platform settled on opaque and unreproducible binaries as the distribution mechanism early on It's not that odd. Reproducible builds are very hard. "gcc foo.c -o foo" is very easy.
- giancarlostoro 8y ago
  Also as a sibling comment mentioned it's not priority number one. People were happy they didn't have to compile the whole damn thing anymore.
pjc50 8y ago
Move to Linux from where? Windows? SunOS? > its really quite odd that the dominant open source platform settled on opaque and unreproducible binaries as the distribution mechanism early on It was perfectly logical when I was installing slackware from floppies, and it remains more useful in most use cases - smaller downloads that installed faster. Debian's great innovation was dependency management of binary packages, after all.
- convolvatron 8y ago
  oh yes, sorry, yes I'm that old. before linux, except for maybe gcc, open source was largely distributed as source. and I agree that installing linux as a binary makes sense. but people didn't think it was important to maintain the reference back to the actual source used. I think to some extent (RedHat specifically), the patch sets used and the hidden build process was used to make it less trivial to reproduce.
  - uep 8y ago
    The distros do maintain a reference back to the source. Debian makes it trivial to download the source code for packages and to build it yourself. apt source htop apt build-dep htop cd htop-2.0.2 dpkg-buildpackage -us -uc -nc
    - uep 8y ago
      I should mention that reproducible builds are really about making bit-for-bit identical builds. There are reasons this is difficult, including individual projects inserting timestamps at the time that they build, or compile-time randomness, sort order of files in the filesystem based on the current locale, etc.
  - rstuart4133 8y ago
    > less trivial to reproduce. Seriously, this is nit picking, and you seem to acknowledge that here by using the term "less trivial". Instead of "tar -x" you run "rpmbuild -bp" to get the patched source RedHat will unleash the compiler on. For debian its "dpkg-source -x". I'm not even sure that it could be called "less trivial". Getting hold of the source in the first place it looks to be "more trivial" to me as there is no Wumpus hunt to find the URL. For Debian it's just "apt-get source package-name". The RedHat equivalent is "yumdownloader --source". In both cases that gets you the exact source used to build the binary on your machine. I don't know what you would call a strong reference back to the actual source used, but I'm struggling to see how it could be much stronger. As for the "hidden build process": it's only hidden because it's automated. You may as well be blaming "make" for hiding the kernel build process, which is more complex than most distro package builds. But every step the distro build processes (any distro) can be made very visible, naturally enough because otherwise it would be impossible to debug when things go wrong. And there is excellent doco on how to do all of this for both Red Hat and Debian.
cat199 8y ago
imho (and I wasn't there at the time) this is a result of proprietary unices constricting the growth of 'free unix' (e.g. source based BSD) in the commercial realm along with the rapid growth of space-constrained 90s PC's and "unix" users who came up in this commercial "unix" environment.. linux distributions were designed around the model of binary-only proprietary "unix" that just happened to have also have source available as a bonus instead of comprising a fully self-hosting source+binaries+documentation system like research unix and source-available derivatives/patchsets (like bsd, etc) always were from day 1 My personal opinion is that unix is a culture and not a trademark, the culture is the users and systems based around this self-hosting system with source lineage back to the original. Binary unix-derived distributions (and unix clones like linux) are therefore implicitly not unix since they are not out of the box self hosting and source included to one degree or another, and do not have a direct code lineage back to at&t v7. At best, they are some approximation of 'unix like', which can still of course be pretty useful.
jononor 8y ago
You can run the build process of any Debian package you want. It is quite easy to reproduce a functionally identical package. What is hard is too reproduce binaries that are byte-for-byte identical. This is not really needed to debug or work with a system, but needed for security to assure that no-one has meddled with the binary. If you think source package distribution are more useful than binary distribution, then Gentoo Linux is still a viable option.
blihp 8y ago
Assuming you're referring to Debian, you've missed the mark. Debian is more committed to open source, in the RMS 'free software' sense, than any other distro I'm aware of... it's the foundation of the distro. Any Debian user can absolutely rebuild from source any[1] part of the system. The problem that reproducible builds is attempting to solve is that my build, while being 100% functional, might not be bit identical to yours or what shipped with the distro for any number of valid reasons.[2] The problem this creates given the current world of increasingly sophisticated malware is that I can't prove a given build is identical to what was intended based on the binaries. So it's an issue of 'we've always been giving you the source, now we're going to ensure that it's possible to create bit-identical binaries' rather than 'we were giving you free binaries, now we're giving you the source'. If your argument is that one should always build from source, reproducible builds is part of getting to the same effective result without the overhead and headache that the vast majority of users are not willing to deal with. [1] In the cases when this turned out not to be possible, holy hell tends to get raised... see things like the Chromium BLOB issues (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786909 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786909) etc. These issues tend to have a lot of noise made and get resolved very quickly since it goes against the very core of what Debian is intended to be. In cases where you have things that are binary-only such as proprietary GPU drivers, these get segregated into a separate repo with the general advice from Debian of 'you shouldn't use these'. This doesn't tend to be realistic advice for many users given the state of many of the open source drivers... but they've tried their best to steer people away while not making the distro unusable for those who insist on using them. [2] A trivial example would be baking the build date/machine name into the binary... there's no expectation that one would ever reproduce that build again even on the same machine.
- mjw1007 8y ago
  I think the gripe is that it's never been as easy as it could be to say "OK, I'm going to make a change in this here source file, then run the result on my machine while keeping everything else the same". For example if I'm reporting a bug and I have a proposed patch, it would be nice to be able to say "I've been running with this patch for the last month and it's been fine". Ideally there'd be a coherent way to download the source for a particular package, rebuild it, and have the system use the rebuilt result. Bonus marks if it did something useful when a newer version of the package came along. I think https://packages.debian.org/sid/dgit https://packages.debian.org/sid/dgit is an attempt to make the download-and-rebuild part look more uniform, but I think there's still work to be done to make it easy to substitute the result for the distro package.
mikekchar 8y ago
One of the things I liked about Gentoo was the idea of building everything yourself. They even had a license manager that would allow you to decide if you wanted to use packages based on their license (and would warn you when the license changed). I always wondered why that never caught on. But binary distributions are just convenient. Even MCC was a binary distribution and that's going a long way back. I remember rebuilding everything by hand because I wanted a lean system that included exactly what I wanted and nothing more, but that was considered pretty extreme even at that time. One of the problems with having the flexibility of a pure source based system (including being about to recompile with exactly the flags and hence features you want) is making a decent dependency graph. If you install package X and it wants capability A, it's practically impossible to guarantee that it will "all just work" because the user can easily screw it up. In many ways, this was one of the problems with Gentoo (apart from it being unreasonable these days to keep a complete system up to date while recompiling every single package -- it just takes too much time). Of course Gentoo (and derivatives) have binary packages these days as well. I've moved over to Arch which has very minimal configuration for packages and so you have to take responsibility for making everything work. You can easily build things and the build scripts make it very easy to see what patches, etc you are using. However, its a lot more work than most people want. I think the reality of the situation is that most people do no debug their systems. They just use it. If it is "broken" then they complain that it is "broken" and wait for someone to fix it. Actually maintaining your system is akin to working on hotrods in your garage. It's mostly a hobby, even for professionals.
- groestl 8y ago
  The great thing about a reproducible build system and a large enough userbase is that it blurrs the line between a compile-from-source distro like Gentoo and a binary distro. This could be huge: flexibility when needed, and convenience everywhere else, all within a single toolchain. But the work required to reach that goal is massive, and probably not worth the effort if above outcome is the only reason to pursue it. I think trust/security/verifiability might be the more realistic driver, but the fallout could just be a new kind of distro.

jrockway 8y ago

> its really quite odd that the dominant open source platform settled on opaque and unreproducible binaries as the distribution mechanism early on It's not that odd. Reproducible builds are very hard. "gcc foo.c -o foo" is very easy.

giancarlostoro 8y ago
Also as a sibling comment mentioned it's not priority number one. People were happy they didn't have to compile the whole damn thing anymore.

pjc50 8y ago

Move to Linux from where? Windows? SunOS? > its really quite odd that the dominant open source platform settled on opaque and unreproducible binaries as the distribution mechanism early on It was perfectly logical when I was installing slackware from floppies, and it remains more useful in most use cases - smaller downloads that installed faster. Debian's great innovation was dependency management of binary packages, after all.

convolvatron 8y ago
oh yes, sorry, yes I'm that old. before linux, except for maybe gcc, open source was largely distributed as source. and I agree that installing linux as a binary makes sense. but people didn't think it was important to maintain the reference back to the actual source used. I think to some extent (RedHat specifically), the patch sets used and the hidden build process was used to make it less trivial to reproduce.
- uep 8y ago
  The distros do maintain a reference back to the source. Debian makes it trivial to download the source code for packages and to build it yourself. apt source htop apt build-dep htop cd htop-2.0.2 dpkg-buildpackage -us -uc -nc
  - uep 8y ago
    I should mention that reproducible builds are really about making bit-for-bit identical builds. There are reasons this is difficult, including individual projects inserting timestamps at the time that they build, or compile-time randomness, sort order of files in the filesystem based on the current locale, etc.
- rstuart4133 8y ago
  > less trivial to reproduce. Seriously, this is nit picking, and you seem to acknowledge that here by using the term "less trivial". Instead of "tar -x" you run "rpmbuild -bp" to get the patched source RedHat will unleash the compiler on. For debian its "dpkg-source -x". I'm not even sure that it could be called "less trivial". Getting hold of the source in the first place it looks to be "more trivial" to me as there is no Wumpus hunt to find the URL. For Debian it's just "apt-get source package-name". The RedHat equivalent is "yumdownloader --source". In both cases that gets you the exact source used to build the binary on your machine. I don't know what you would call a strong reference back to the actual source used, but I'm struggling to see how it could be much stronger. As for the "hidden build process": it's only hidden because it's automated. You may as well be blaming "make" for hiding the kernel build process, which is more complex than most distro package builds. But every step the distro build processes (any distro) can be made very visible, naturally enough because otherwise it would be impossible to debug when things go wrong. And there is excellent doco on how to do all of this for both Red Hat and Debian.

cat199 8y ago

imho (and I wasn't there at the time) this is a result of proprietary unices constricting the growth of 'free unix' (e.g. source based BSD) in the commercial realm along with the rapid growth of space-constrained 90s PC's and "unix" users who came up in this commercial "unix" environment.. linux distributions were designed around the model of binary-only proprietary "unix" that just happened to have also have source available as a bonus instead of comprising a fully self-hosting source+binaries+documentation system like research unix and source-available derivatives/patchsets (like bsd, etc) always were from day 1 My personal opinion is that unix is a culture and not a trademark, the culture is the users and systems based around this self-hosting system with source lineage back to the original. Binary unix-derived distributions (and unix clones like linux) are therefore implicitly not unix since they are not out of the box self hosting and source included to one degree or another, and do not have a direct code lineage back to at&t v7. At best, they are some approximation of 'unix like', which can still of course be pretty useful.

jononor 8y ago

You can run the build process of any Debian package you want. It is quite easy to reproduce a functionally identical package. What is hard is too reproduce binaries that are byte-for-byte identical. This is not really needed to debug or work with a system, but needed for security to assure that no-one has meddled with the binary. If you think source package distribution are more useful than binary distribution, then Gentoo Linux is still a viable option.

blihp 8y ago

Assuming you're referring to Debian, you've missed the mark. Debian is more committed to open source, in the RMS 'free software' sense, than any other distro I'm aware of... it's the foundation of the distro. Any Debian user can absolutely rebuild from source any[1] part of the system. The problem that reproducible builds is attempting to solve is that my build, while being 100% functional, might not be bit identical to yours or what shipped with the distro for any number of valid reasons.[2] The problem this creates given the current world of increasingly sophisticated malware is that I can't prove a given build is identical to what was intended based on the binaries. So it's an issue of 'we've always been giving you the source, now we're going to ensure that it's possible to create bit-identical binaries' rather than 'we were giving you free binaries, now we're giving you the source'. If your argument is that one should always build from source, reproducible builds is part of getting to the same effective result without the overhead and headache that the vast majority of users are not willing to deal with. [1] In the cases when this turned out not to be possible, holy hell tends to get raised... see things like the Chromium BLOB issues (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786909 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786909) etc. These issues tend to have a lot of noise made and get resolved very quickly since it goes against the very core of what Debian is intended to be. In cases where you have things that are binary-only such as proprietary GPU drivers, these get segregated into a separate repo with the general advice from Debian of 'you shouldn't use these'. This doesn't tend to be realistic advice for many users given the state of many of the open source drivers... but they've tried their best to steer people away while not making the distro unusable for those who insist on using them. [2] A trivial example would be baking the build date/machine name into the binary... there's no expectation that one would ever reproduce that build again even on the same machine.

mjw1007 8y ago
I think the gripe is that it's never been as easy as it could be to say "OK, I'm going to make a change in this here source file, then run the result on my machine while keeping everything else the same". For example if I'm reporting a bug and I have a proposed patch, it would be nice to be able to say "I've been running with this patch for the last month and it's been fine". Ideally there'd be a coherent way to download the source for a particular package, rebuild it, and have the system use the rebuilt result. Bonus marks if it did something useful when a newer version of the package came along. I think https://packages.debian.org/sid/dgit https://packages.debian.org/sid/dgit is an attempt to make the download-and-rebuild part look more uniform, but I think there's still work to be done to make it easy to substitute the result for the distro package.

mikekchar 8y ago

One of the things I liked about Gentoo was the idea of building everything yourself. They even had a license manager that would allow you to decide if you wanted to use packages based on their license (and would warn you when the license changed). I always wondered why that never caught on. But binary distributions are just convenient. Even MCC was a binary distribution and that's going a long way back. I remember rebuilding everything by hand because I wanted a lean system that included exactly what I wanted and nothing more, but that was considered pretty extreme even at that time. One of the problems with having the flexibility of a pure source based system (including being about to recompile with exactly the flags and hence features you want) is making a decent dependency graph. If you install package X and it wants capability A, it's practically impossible to guarantee that it will "all just work" because the user can easily screw it up. In many ways, this was one of the problems with Gentoo (apart from it being unreasonable these days to keep a complete system up to date while recompiling every single package -- it just takes too much time). Of course Gentoo (and derivatives) have binary packages these days as well. I've moved over to Arch which has very minimal configuration for packages and so you have to take responsibility for making everything work. You can easily build things and the build scripts make it very easy to see what patches, etc you are using. However, its a lot more work than most people want. I think the reality of the situation is that most people do no debug their systems. They just use it. If it is "broken" then they complain that it is "broken" and wait for someone to fix it. Actually maintaining your system is akin to working on hotrods in your garage. It's mostly a hobby, even for professionals.

groestl 8y ago
The great thing about a reproducible build system and a large enough userbase is that it blurrs the line between a compile-from-source distro like Gentoo and a binary distro. This could be huge: flexibility when needed, and convenience everywhere else, all within a single toolchain. But the work required to reach that goal is massive, and probably not worth the effort if above outcome is the only reason to pursue it. I think trust/security/verifiability might be the more realistic driver, but the fallout could just be a new kind of distro.