6 ms·

What does "reproducibility" mean? I understand and appreciate the importance of reproducibility in the context of scientific experiments, but I don't understand

by pen2l 9y ago

What does "reproducibility" mean? I understand and appreciate the importance of reproducibility in the context of scientific experiments, but I don't understand what it means in terms of computer programs. I am guessing it has to do with being able to build on different architectures without issue?

richdougherty 9y ago
I think it's for security. It means that there's a deterministic relationship between the source of a program and its final compiled artifacts. If software has reproducible builds that means that third-parties can independently verify that artifacts have been built safely from sources, without any sneaky stuff snuck in.
CJefferson 9y ago
No, the same architecture. It has taken, and continues to take, a suprising amount of work to make two builds of a program produce the same output. There are many sources of issues. For example: date and time stored in output, program's output effected by order in which files are read from a directory (and is not having a fixed ordering), hash tables based on the pointer and the high objects are stored having different ordering on different executions, parallel programs behaving differently on different runs, and others.
jldugger 9y ago
It's about being able to reproduce the binary from source. You might think this is pretty much impossible in the Debian context, but things like timestamps, and underspecified dependencies can end up shifting a build's result over time. If we want to insist that open source code is secure by source code analysis, we need a verifiable build chain, that the code and binaries an analysis uses are the same as what we get later.
cyphar 9y ago
It has a similar meaning to research. What it means is that you can reproduce (compile in most cases) from source code the same bit-for-bit identical binary independently. While this might sound like something that should be trivial to do, it turns out to be far from trivial (timestamps and other environment information leaks into binaries all the time). There's a website that describes this project in much more detail as well as how they worked around the various problems they found. https://reproducible-builds.org/ https://reproducible-builds.org/
detaro 9y ago
It means ensuring that anyone can verify that a binary was created from a given set of source code, by recompiling it and getting the exact same file.
wongarsu 9y ago
It means each time you build the same code in a known setup, you get bit for bit the same binary. That allows you to assure that the code that's shipped actually matches the source code. It sounds trivial, but the full paths and timestamps that get added at multiple points in the process are enough to screw this up, and those are the easy problems.
cesarb 9y ago
In the context of "reproducible builds", it means that if you compile the same source code with the same compiler and build system, the output will be completely identical, bit by bit. This is surprisingly hard to achieve in practice. Once they have reproducible builds, they can easily prove that each binary package was built from the corresponding source code package: just have a third party compile the source code again and generate the binary package, and it should be identical (except for the signature). This reduces the need to trust that the build machines haven't been compromised.
- kobeya 9y ago
  Just piggybacking on this comment, you can do a whole lot more than just trust that a few people have automated. Most people in Ubuntu get non-distro packages from Launchpad, for example, which uses their own build servers. With reproducible builds you can require BOTH launchpad and the developer's signature for a package to be valid, which tremendously improves the security situation.
  - kobeya 9y ago
    * trust that a few people have audited. (Damn autocorrect.)
e12e 9y ago
In addition to all the siblings; this is also important in research - which increasingly uses computers. If you provide a paper, source code, dataset and description of the system(s) used - can someone reproduce your research? It would certainly be convenient if you can point to a version/snapshot of Debian (or another distribution) - and it would then be possible to take your (say C) source code and compile and run the same binary used for research. It's true that often getting the algorithm more-or-less right is enough - but the more research is augmented by computing devices, the more important it becomes to maintain reproducibility - and the more complex and capable these computer devices (say a top-100 supercomputer, software stack in C++ on top of MPI, some Fortran numeric libraries etc) become - the harder it becomes to maintain it. Imagine verifying research done today by repeating experiments in 50 years.

by pen2l 9y ago

richdougherty 9y ago

I think it's for security. It means that there's a deterministic relationship between the source of a program and its final compiled artifacts. If software has reproducible builds that means that third-parties can independently verify that artifacts have been built safely from sources, without any sneaky stuff snuck in.

CJefferson 9y ago

No, the same architecture. It has taken, and continues to take, a suprising amount of work to make two builds of a program produce the same output. There are many sources of issues. For example: date and time stored in output, program's output effected by order in which files are read from a directory (and is not having a fixed ordering), hash tables based on the pointer and the high objects are stored having different ordering on different executions, parallel programs behaving differently on different runs, and others.

jldugger 9y ago

It's about being able to reproduce the binary from source. You might think this is pretty much impossible in the Debian context, but things like timestamps, and underspecified dependencies can end up shifting a build's result over time. If we want to insist that open source code is secure by source code analysis, we need a verifiable build chain, that the code and binaries an analysis uses are the same as what we get later.

cyphar 9y ago

It has a similar meaning to research. What it means is that you can reproduce (compile in most cases) from source code the same bit-for-bit identical binary independently. While this might sound like something that should be trivial to do, it turns out to be far from trivial (timestamps and other environment information leaks into binaries all the time). There's a website that describes this project in much more detail as well as how they worked around the various problems they found. https://reproducible-builds.org/ https://reproducible-builds.org/

detaro 9y ago

It means ensuring that anyone can verify that a binary was created from a given set of source code, by recompiling it and getting the exact same file.

wongarsu 9y ago

It means each time you build the same code in a known setup, you get bit for bit the same binary. That allows you to assure that the code that's shipped actually matches the source code. It sounds trivial, but the full paths and timestamps that get added at multiple points in the process are enough to screw this up, and those are the easy problems.

cesarb 9y ago

In the context of "reproducible builds", it means that if you compile the same source code with the same compiler and build system, the output will be completely identical, bit by bit. This is surprisingly hard to achieve in practice. Once they have reproducible builds, they can easily prove that each binary package was built from the corresponding source code package: just have a third party compile the source code again and generate the binary package, and it should be identical (except for the signature). This reduces the need to trust that the build machines haven't been compromised.

kobeya 9y ago
Just piggybacking on this comment, you can do a whole lot more than just trust that a few people have automated. Most people in Ubuntu get non-distro packages from Launchpad, for example, which uses their own build servers. With reproducible builds you can require BOTH launchpad and the developer's signature for a package to be valid, which tremendously improves the security situation.
- kobeya 9y ago
  * trust that a few people have audited. (Damn autocorrect.)

e12e 9y ago

In addition to all the siblings; this is also important in research - which increasingly uses computers. If you provide a paper, source code, dataset and description of the system(s) used - can someone reproduce your research? It would certainly be convenient if you can point to a version/snapshot of Debian (or another distribution) - and it would then be possible to take your (say C) source code and compile and run the same binary used for research. It's true that often getting the algorithm more-or-less right is enough - but the more research is augmented by computing devices, the more important it becomes to maintain reproducibility - and the more complex and capable these computer devices (say a top-100 supercomputer, software stack in C++ on top of MPI, some Fortran numeric libraries etc) become - the harder it becomes to maintain it. Imagine verifying research done today by repeating experiments in 50 years.