5 ms·

Stack and Nix = portable reproducible builds

135 points by mcbuilder 11y ago

pmahoney 11y ago
I'm curious how this handles changes within nixpkgs itself. For example, if developer A has nixpkgs version 111, and developer B has nixpkgs version 222, and they both attempt to build a Haskell project depending on "libxml", they may compile against different versions of libxml (or worse, an older version of libxml might be incompatible with the Haskell code). Nixpkgs solves this itself by "owning" the entire dependency tree. That is, if there's a definition in nixpkgs that requires libxml, you can bet it works with the libxml derivation defined in that same nixpkgs. If not, there might be libxml1, libxml2 derivations defined. In practice, using the ~latest version of everything, or whatever nixpkgs happens to have, usually works well. I started moving toward building my own nixpkgs distribution, which included a vendored copy of nixpkgs at a specific version (rather than whatever version was available at the time you downloaded it), combined with a few customizations for specific (older) versions of, say, libxml. This ensured multiple developers could download things months apart and still expect everything to work. (nixpkgs changes somewhat rapidly.)
- cwp 11y ago
  Yeah, I've done something like this as well. It's wrapper around nixpkgs that provides: * baseline nixpkgs at a specific version * customized overrides of specific packages * a library of functions to aid packaging our own software It works great. It's actually not very much code, but it took a while to figure out what we wanted.
jallmann 11y ago
Probably the most succinct comparison of Nix and Docker I've seen. So this works by having Stack (a Haskell build tool?) emit a nix file to build a Haskell project? Very cool. The more natural approach for this seems to be to use your language's package manager spec as a base to build Nix packages. Since package manager spec files already contain build instructions, this frees you up from any dependency on a specific build system. It would then be straightforward to generate Nix packages from a repository and serve them through a Nix channel. All that being said, I should implement this for OCaml and OPAM. Nix is amazing, but it is "all-in." Native Nix builds the right way involve all your dependencies being Nix packages. (Nix builds using a language's package manager are impure and would likely require hand-holding with external deps.) This article seems like a nice step towards solving that, but I think the real solution would be tooling around the package manager itself.
- Ericson2314 11y ago
  > The more natural approach for this seems to be to use your language's package manager spec as a base to build Nix packages. Since package manager spec files already contain build instructions, this frees you up from any dependency on a specific build system. It would then be straightforward to generate Nix packages from a repository and serve them through a Nix channel. Yeah `Stack.yaml` is weird because it is both manually edited and generated. I'd rather see a split where the `Cabal-file + overrides/extra-info => build-plan` (not any central package db is not consulted). Then either Nix, or the built-in default can interpret that build plan (and just the build-plan) to actually build the thing. We can add a feedback loop `Cabal-file + overrides/extra-info + Maybe(solved-versions) => build-plan solved-versions` analogous to Cargo's and Bundler's lockfiles too. Basically the goal is to separate the less deterministic aspects of querying versions and solving constraints, vs the entirely determinstic simple-stupid steps of actually building things. Ideally then, nixpkgs would need no knowledge of upstream packages, and just the ability to interpret build-plans per language package manager. Caching would follow from just making sure the build-plan didn't very in stupid ways (of course the nix interpretation of the build-plan should be deterministic). For system packages relying on a langauge package manager (xmonad...) use a "dynamic import" (import result of derivation) to avoid the need of caching these steps in the nixpkgs source.
smilliken 11y ago
I highly recommend Nix. We used to use apt-get, pypi, rubygems, npm, make, and others, and now it's all replaced with a single tool: nix. It's dramatically simplified builds and dependency management for us at MixRank. Benefits: * It's declarative, so no one has to run anything to get to the correct state when a dependency has changed. Open a new nix-shell and automatically get everything. It's impossible to get in a bad state because it doesn't touch your system. * We know we're all running the same software because the build will fail if hashes don't match. * Portable across most unix's, and windows support is in progress. * Patching other people's code is as easy as referencing a .patch file from a nix expression. You don't have to wait for your patch to be accepted in the main project or fork the project to build a custom release. * There's no virtualization of IO like in containers. It effectively has no overhead. * We can easily use different versions of the same piece of software in parallel; they don't collide because the hashes don't match. * It has a very active community. The package repository, nixpkgs [1], has 73k commits, 650 contributors, and median pull request acceptance of 18 hours. * Our software won't silently break because one of our package managers replaced a package. We maintain a fork of nixpkgs which we rebase on our own schedule. If you've ever been frustrated with your package manager or build tools, give nix a shot! [1] https://github.com/nixos/nixpkgs https://github.com/nixos/nixpkgs
- wingless 11y ago
  Do you use NixOS or Nix on another distro?
  - smilliken 11y ago
    I've been using it with Ubuntu. We also have it working with OSX. I'd love to give NixOS a try, but haven't gotten around to it. Version controlling the entire system is very compelling.
- codemac 11y ago
  If you're like me and have a hankerin' for something other than the nix language, check out guix! http://gnu.org/s/guix http://gnu.org/s/guix All of the scheme your heart could want. I run archlinux + guix and it's a magical happy land. I'm sure the same could be accomplished with nix, but I enjoy writing scheme too much!
rnhmjoj 11y ago
I have never used stack: I went from cabal-install hell straight to NixOS. My current approach is: write the usual .cabal file, run cabal2nix --shell . > default.nix then build and install the package (with `nix-build` or `nix-env -if.`) or open a nix-shell (just `nix-shell` in the project root) where I can run cabal-install to do things like uploading to hackage, building haddock docs, using cabal repl. The nix expression is deterministically generated by the .cabal so unless you update the nix channel it won't change and you can safely delete it and regenerate it at anytime. While stack.yaml is not, or not entirely generate, if I understand correctly. Are there other advantages over cabal2nix beside parallelization and incremental recompilation? I hardly handle 2 package at once so it's not really an issue for me.
- mboes 11y ago
  Incremental recompilation is a big one, but beyond that, out-of-the-box handling of multi-package projects without the hassle of custom scripting or setting up .nix files just right. If all your projects are single package, `cabal repl` should work pretty well. Where stack really shines, and where it's interesting to simply reuse that, is for multi-package repos, which tend to be very common in private company Haskell developments.
gnu 11y ago
How are the security update/fixes handled in NixOS?

135 points by mcbuilder 11y ago

I'm curious how this handles changes within nixpkgs itself. For example, if developer A has nixpkgs version 111, and developer B has nixpkgs version 222, and they both attempt to build a Haskell project depending on "libxml", they may compile against different versions of libxml (or worse, an older version of libxml might be incompatible with the Haskell code). Nixpkgs solves this itself by "owning" the entire dependency tree. That is, if there's a definition in nixpkgs that requires libxml, you can bet it works with the libxml derivation defined in that same nixpkgs. If not, there might be libxml1, libxml2 derivations defined. In practice, using the ~latest version of everything, or whatever nixpkgs happens to have, usually works well. I started moving toward building my own nixpkgs distribution, which included a vendored copy of nixpkgs at a specific version (rather than whatever version was available at the time you downloaded it), combined with a few customizations for specific (older) versions of, say, libxml. This ensured multiple developers could download things months apart and still expect everything to work. (nixpkgs changes somewhat rapidly.)

cwp 11y ago
Yeah, I've done something like this as well. It's wrapper around nixpkgs that provides: * baseline nixpkgs at a specific version * customized overrides of specific packages * a library of functions to aid packaging our own software It works great. It's actually not very much code, but it took a while to figure out what we wanted.

jallmann 11y ago

Probably the most succinct comparison of Nix and Docker I've seen. So this works by having Stack (a Haskell build tool?) emit a nix file to build a Haskell project? Very cool. The more natural approach for this seems to be to use your language's package manager spec as a base to build Nix packages. Since package manager spec files already contain build instructions, this frees you up from any dependency on a specific build system. It would then be straightforward to generate Nix packages from a repository and serve them through a Nix channel. All that being said, I should implement this for OCaml and OPAM. Nix is amazing, but it is "all-in." Native Nix builds the right way involve all your dependencies being Nix packages. (Nix builds using a language's package manager are impure and would likely require hand-holding with external deps.) This article seems like a nice step towards solving that, but I think the real solution would be tooling around the package manager itself.

Ericson2314 11y ago
> The more natural approach for this seems to be to use your language's package manager spec as a base to build Nix packages. Since package manager spec files already contain build instructions, this frees you up from any dependency on a specific build system. It would then be straightforward to generate Nix packages from a repository and serve them through a Nix channel. Yeah `Stack.yaml` is weird because it is both manually edited and generated. I'd rather see a split where the `Cabal-file + overrides/extra-info => build-plan` (not any central package db is not consulted). Then either Nix, or the built-in default can interpret that build plan (and just the build-plan) to actually build the thing. We can add a feedback loop `Cabal-file + overrides/extra-info + Maybe(solved-versions) => build-plan solved-versions` analogous to Cargo's and Bundler's lockfiles too. Basically the goal is to separate the less deterministic aspects of querying versions and solving constraints, vs the entirely determinstic simple-stupid steps of actually building things. Ideally then, nixpkgs would need no knowledge of upstream packages, and just the ability to interpret build-plans per language package manager. Caching would follow from just making sure the build-plan didn't very in stupid ways (of course the nix interpretation of the build-plan should be deterministic). For system packages relying on a langauge package manager (xmonad...) use a "dynamic import" (import result of derivation) to avoid the need of caching these steps in the nixpkgs source.

smilliken 11y ago

I highly recommend Nix. We used to use apt-get, pypi, rubygems, npm, make, and others, and now it's all replaced with a single tool: nix. It's dramatically simplified builds and dependency management for us at MixRank. Benefits: * It's declarative, so no one has to run anything to get to the correct state when a dependency has changed. Open a new nix-shell and automatically get everything. It's impossible to get in a bad state because it doesn't touch your system. * We know we're all running the same software because the build will fail if hashes don't match. * Portable across most unix's, and windows support is in progress. * Patching other people's code is as easy as referencing a .patch file from a nix expression. You don't have to wait for your patch to be accepted in the main project or fork the project to build a custom release. * There's no virtualization of IO like in containers. It effectively has no overhead. * We can easily use different versions of the same piece of software in parallel; they don't collide because the hashes don't match. * It has a very active community. The package repository, nixpkgs [1], has 73k commits, 650 contributors, and median pull request acceptance of 18 hours. * Our software won't silently break because one of our package managers replaced a package. We maintain a fork of nixpkgs which we rebase on our own schedule. If you've ever been frustrated with your package manager or build tools, give nix a shot! [1] https://github.com/nixos/nixpkgs https://github.com/nixos/nixpkgs

wingless 11y ago
Do you use NixOS or Nix on another distro?
- smilliken 11y ago
  I've been using it with Ubuntu. We also have it working with OSX. I'd love to give NixOS a try, but haven't gotten around to it. Version controlling the entire system is very compelling.
codemac 11y ago
If you're like me and have a hankerin' for something other than the nix language, check out guix! http://gnu.org/s/guix http://gnu.org/s/guix All of the scheme your heart could want. I run archlinux + guix and it's a magical happy land. I'm sure the same could be accomplished with nix, but I enjoy writing scheme too much!

rnhmjoj 11y ago

I have never used stack: I went from cabal-install hell straight to NixOS. My current approach is: write the usual .cabal file, run cabal2nix --shell . > default.nix then build and install the package (with `nix-build` or `nix-env -if.`) or open a nix-shell (just `nix-shell` in the project root) where I can run cabal-install to do things like uploading to hackage, building haddock docs, using cabal repl. The nix expression is deterministically generated by the .cabal so unless you update the nix channel it won't change and you can safely delete it and regenerate it at anytime. While stack.yaml is not, or not entirely generate, if I understand correctly. Are there other advantages over cabal2nix beside parallelization and incremental recompilation? I hardly handle 2 package at once so it's not really an issue for me.

mboes 11y ago
Incremental recompilation is a big one, but beyond that, out-of-the-box handling of multi-package projects without the hassle of custom scripting or setting up .nix files just right. If all your projects are single package, `cabal repl` should work pretty well. Where stack really shines, and where it's interesting to simply reuse that, is for multi-package repos, which tend to be very common in private company Haskell developments.

gnu 11y ago

How are the security update/fixes handled in NixOS?