5 ms·

Take a look at nix-shell scripts. Much simpler, lightweight and faster alternative to Docker and friends for being able to build a project forever. Start of a b

Take a look at nix-shell scripts. Much simpler, lightweight and faster alternative to Docker and friends for being able to build a project forever. Start of a build script from a C project of mine: #!/usr/bin/env nix-shell #! nix-shell -i fish --pure #! nix-shell -I https://github.com/NixOS/nixpkgs/archive/4ecab3273592f27479a583fb6d975d4aba3486fe.tar.gz #! nix-shell --packages fish gcc gnumake gmp git cacert ...build script goes here... -i fish: I am using fish shell. --pure: Environment is as pure as it can reasonably get. --packages: Packages from nixpkgs to be made available in the environment. -I <github_commit_hash>: pinned to a specific git commit of nixpkgs. This script will give you the exact same environment with the exact same packages with exact same dependencies forever. You can drop into a shell with the exact same configuration by using the cli command nix-shell with the same options. Admittedly this is not as declarative as using flakes, since it's a script, but hey, I'm lazy, still didn't sit down to learn them once and for all. Reference: https://nixos.org/manual/nix/stable/command-ref/nix-shell https://nixos.org/manual/nix/stable/command-ref/nix-shell

cge 3y ago
> Take a look at nix-shell scripts. Much simpler, lightweight and faster alternative to Docker and friends for being able to build a project forever. As researchers using nix derivations to preserve usability of code meant to be archived with our papers, we found that it was not particularly effective for this task. Even with nixpkgs pinning, our code failed to compile after only a few years. Outside of nixos itself, it does not appear that the build environment is the same; aspects of compilation change, and we ended up needing to change compiler flags a few years later, while expecting that we might need to again at some point. Overall nix has been quite disappointing for us from an archival standpoint.
- Aerbil313 3y ago
  Nix-shell scripts are ad-hoc, what nix does here is to provide an environment. Nix derivations are different and can be fully declarative and pure. Nixpkgs pinning does not mean specific nixpkgs commit pinning or pinning a specific version of a package. You can do these as well. I wonder what was the underlying reason for your condition.
- jolux 3y ago
  Just curious, have you written anything up about these experiences with Nix? I've been learning it recently and the reproducibility angle is a huge part of why, if it's oversold then I feel like I should reevaluate how much time I'm putting into it.
- deredede 3y ago
  Things shouldn't change if the nixpkgs version didn't change, you didn't disable the sandbox, and you don't have master branches of random repos as dependencies. If that's the case, sounds like a major nix bug. If the sandbox is disabled (as it is by default outside of nixos I believe) then yes, you need to be very careful that there is no outside state that leaks into your build (usually through environment variables, PATH but not only - GCC is a big offender here), which sounds like what happened in your case? For archival purposes you really should be enabling the sandbox.
  - cge 3y ago
    Thanks: it does seem like the lack of sandboxing is the major problem here. A difficulty in this sort of archiving is that we're not trying to make it so that we can build the code: we're trying to make it so that any reader in the future can. Close to zero of those readers are likely to be using nix for anything else, or to have any experience with nix (or any other approach we might take). Our approach was to make a script for nix-shell that would put the readers in an environment where they could run the code, compiling specific (old) versions of two research software packages. But I think the problem here is that in writing the derivations here, we didn't put gcc in the build inputs. So the system's compiler ended up being used, and with the old, finicky code, compiler flags needed to change. A frustration here is that it appears sandboxing can't be enabled for nix-shell, a problem that has been known for close to a decade [2]. But in our case, if we can get the build inputs to be reasonably complete on our own, it should improve the situation for our readers in the future. [1]: https://github.com/DNA-and-Natural-Algorithms-Group/SST-sequence-designer https://github.com/DNA-and-Natural-Algorithms-Group/SST-sequ... [2]: https://github.com/NixOS/nix/issues/903 https://github.com/NixOS/nix/issues/903
- gray_-_wolf 3y ago
  You might give GNU Guix a try, combination of guix time-machine and guix shell -C should give you the exact same environment each time without fear of being affected by the rest of the system (with the obvious exception of the linux kernel). And it seems to be popular in science circles.
- ParetoOptimal 3y ago
  > Even with nixpkgs pinning, our code failed to compile after only a few years. Outside of nixos itself, it does not appear that the build environment is the same; aspects of compilation change, and we ended up needing to change compiler flags a few years later, while expecting that we might need to again at some point. Was this using flakes or no? Since you mention pinning, I'm guessing not. An experience report, even if brief, would be very valuable to a Nix/NixOS community working seriously on reproducibility.
oneshtein 3y ago
RPM .spec is more readable, IMHO. Archive of system repo + mock + SRPM does the same thing, but offline.
- bfrog 3y ago
  Rpm assumes deps and toolchain and doesn’t prevent accidental dependencies last I used it, admittedly many years ago. Are builds somehow done in a limited chroot now? Is the toolchain version dependency tracked and managed?
  - oneshtein 3y ago
    In Fedora, mock[0] is used to build packages in clean chroot for multiple of distros. In SuSE, Open Build Service[1] i used. [0]: https://fedoraproject.org/wiki/Using_Mock_to_test_package_builds https://fedoraproject.org/wiki/Using_Mock_to_test_package_bu... [1]: https://en.opensuse.org/openSUSE:Build_Service_Tutorial https://en.opensuse.org/openSUSE:Build_Service_Tutorial
    - bfrog 3y ago
      What about the toolchain being used, is that some how understood to be part of the result? This seems to be often overlooked. In nix the toolchain is considered an input I believe and taken into the output derivations hash. Meaning a toolchain change results in a new package (rightfully so!) What if you need different versions of the same package to support end applications?

cge 3y ago

> Take a look at nix-shell scripts. Much simpler, lightweight and faster alternative to Docker and friends for being able to build a project forever. As researchers using nix derivations to preserve usability of code meant to be archived with our papers, we found that it was not particularly effective for this task. Even with nixpkgs pinning, our code failed to compile after only a few years. Outside of nixos itself, it does not appear that the build environment is the same; aspects of compilation change, and we ended up needing to change compiler flags a few years later, while expecting that we might need to again at some point. Overall nix has been quite disappointing for us from an archival standpoint.

Aerbil313 3y ago
Nix-shell scripts are ad-hoc, what nix does here is to provide an environment. Nix derivations are different and can be fully declarative and pure. Nixpkgs pinning does not mean specific nixpkgs commit pinning or pinning a specific version of a package. You can do these as well. I wonder what was the underlying reason for your condition.
jolux 3y ago
Just curious, have you written anything up about these experiences with Nix? I've been learning it recently and the reproducibility angle is a huge part of why, if it's oversold then I feel like I should reevaluate how much time I'm putting into it.
deredede 3y ago
Things shouldn't change if the nixpkgs version didn't change, you didn't disable the sandbox, and you don't have master branches of random repos as dependencies. If that's the case, sounds like a major nix bug. If the sandbox is disabled (as it is by default outside of nixos I believe) then yes, you need to be very careful that there is no outside state that leaks into your build (usually through environment variables, PATH but not only - GCC is a big offender here), which sounds like what happened in your case? For archival purposes you really should be enabling the sandbox.
- cge 3y ago
  Thanks: it does seem like the lack of sandboxing is the major problem here. A difficulty in this sort of archiving is that we're not trying to make it so that we can build the code: we're trying to make it so that any reader in the future can. Close to zero of those readers are likely to be using nix for anything else, or to have any experience with nix (or any other approach we might take). Our approach was to make a script for nix-shell that would put the readers in an environment where they could run the code, compiling specific (old) versions of two research software packages. But I think the problem here is that in writing the derivations here, we didn't put gcc in the build inputs. So the system's compiler ended up being used, and with the old, finicky code, compiler flags needed to change. A frustration here is that it appears sandboxing can't be enabled for nix-shell, a problem that has been known for close to a decade [2]. But in our case, if we can get the build inputs to be reasonably complete on our own, it should improve the situation for our readers in the future. [1]: https://github.com/DNA-and-Natural-Algorithms-Group/SST-sequence-designer https://github.com/DNA-and-Natural-Algorithms-Group/SST-sequ... [2]: https://github.com/NixOS/nix/issues/903 https://github.com/NixOS/nix/issues/903
gray_-_wolf 3y ago
You might give GNU Guix a try, combination of guix time-machine and guix shell -C should give you the exact same environment each time without fear of being affected by the rest of the system (with the obvious exception of the linux kernel). And it seems to be popular in science circles.
ParetoOptimal 3y ago
> Even with nixpkgs pinning, our code failed to compile after only a few years. Outside of nixos itself, it does not appear that the build environment is the same; aspects of compilation change, and we ended up needing to change compiler flags a few years later, while expecting that we might need to again at some point. Was this using flakes or no? Since you mention pinning, I'm guessing not. An experience report, even if brief, would be very valuable to a Nix/NixOS community working seriously on reproducibility.

oneshtein 3y ago

RPM .spec is more readable, IMHO. Archive of system repo + mock + SRPM does the same thing, but offline.

bfrog 3y ago
Rpm assumes deps and toolchain and doesn’t prevent accidental dependencies last I used it, admittedly many years ago. Are builds somehow done in a limited chroot now? Is the toolchain version dependency tracked and managed?
- oneshtein 3y ago
  In Fedora, mock[0] is used to build packages in clean chroot for multiple of distros. In SuSE, Open Build Service[1] i used. [0]: https://fedoraproject.org/wiki/Using_Mock_to_test_package_builds https://fedoraproject.org/wiki/Using_Mock_to_test_package_bu... [1]: https://en.opensuse.org/openSUSE:Build_Service_Tutorial https://en.opensuse.org/openSUSE:Build_Service_Tutorial
  - bfrog 3y ago
    What about the toolchain being used, is that some how understood to be part of the result? This seems to be often overlooked. In nix the toolchain is considered an input I believe and taken into the output derivations hash. Meaning a toolchain change results in a new package (rightfully so!) What if you need different versions of the same package to support end applications?