3 ms·

I use Nix for building projects and managing my laptops (NixOS at home, nix-darwin on macOS at work). This solves the 'dependency hell' in a much cleaner, more

I use Nix for building projects and managing my laptops (NixOS at home, nix-darwin on macOS at work). This solves the 'dependency hell' in a much cleaner, more efficient and IMHO simpler way than docker. For those who don't know, Nix doesn't install anything system-wide. Everything gets prefixed with a hash of its dependencies (e.g. /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1/bin/git). Since nothing is available system-wide, program A can only use program B by specifying its whole path (including the hash); this affects the hash of program A, and so on for anything that refers to it. This is a Merkle tree, giving the provenance of every dependency, all the way down to the 'bootstrap' tools (e.g. a prebuilt GCC that was used to build the GCC that was used to build the tar that was used to extract the bash source ....). Here's a snippet of the dependencies used to build the above git command: $ nix-store -q --requisites $(nix-store --deriver /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1) /nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv /nix/store/064jmylcq7h6fa5asg0rna9awcqz3765-0001-x86-Properly-merge-GNU_PROPERTY_X86_ISA_1_USED.patch /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh /nix/store/na8n4ndbmkc500jm7argsprzj94p27h4-libev-4.27.tar.gz.drv /nix/store/33sl3bqjcqzrdd9clgaad3ljlwyl1pkb-patch-shebangs.sh /nix/store/81ikflgpwzgjk8b5vmvg9gaw9mbkc86k-compress-man-pages.sh /nix/store/9ny6szla9dg61jv8q22qbnqsz37465n0-multiple-outputs.sh /nix/store/a92kz10cwkpa91k5239inl3fd61zp5dh-move-lib64.sh /nix/store/5g9j02amqihyg9b15m9bds9iqkjclcgr-bootstrap-tools.tar.xz.drv /nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv /nix/store/c0sr4qdy8halrdrh5dpm7hj05c6hyssa-unpack-bootstrap-tools.sh /nix/store/dccfayc70jsvzwq023smvqv04c57bsib-bootstrap-tools.drv ... Here's a snippet of the above git command's runtime dependencies (i.e. everything it references which, if it were our app, we'd need to include in its container): $ nix-store -q --requisites /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1 /nix/store/5gga7b7aavb4vcnia44mzd6m9vrx46gq-perl5.30.0-LWP-MediaTypes-6.04 /nix/store/9n0n6dg310azjmnyfxq6bsn0jsz9cvk8-perl5.30.0-URI-1.76 /nix/store/9zmmc8akdb2jlkapqf9p26jn84xvzbjr-perl5.30.0-HTTP-Date-6.02 /nix/store/ds92hvb2jx8yfh2j3wqy1fqr07654nkr-perl5.30.0-Encode-Locale-1.05 /nix/store/l43m8p3bkqar54krnj3jalbim7m64lid-perl5.30.0-IO-HTML-1.001 /nix/store/zbd643lb2a8iqgp1m36r1adzpqpwhr7s-perl5.30.0-HTTP-Message-6.18 /nix/store/0hkq3crf7357z5zywx4i9dkddi0qvb8k-perl5.30.0-HTTP-Daemon-6.01 /nix/store/5ka41zhii1bjss3f60rzd2npz9mxj060-glibc-2.27 /nix/store/7fvwr8la2k701hrx2w5xnvjr5kkc7ysv-gcc-8.3.0-lib /nix/store/clp0hgzq3pwx03lr8apg92nmswr4izvz-bash-4.4-p23 /nix/store/hgdh73wvrncfnmdmg1damcfrviv9cgp7-gnum4-1.4.18 /nix/store/ig8wzhnxxxrspcyb85wxqdbbn2q53088-bison-3.4.2 /nix/store/11z0140njah4dbrngg5bcx99nyw391kw-gettext-0.19.8.1 /nix/store/1ycvwhslagg7dn8rpc3simyfkirgjbp5-perl5.30.0-Net-HTTP-6.19 /nix/store/n3ja6nl5i91x5sn232f5fyhmx7lxcmj1-ncurses-6.1-20190112 /nix/store/47w0y1vavaxja3mm9gr5dmbgxrjgpw21-libedit-20190324-3.1 /nix/store/84jxhr8l3plkd6z2x4v941za9kvmv88g-zlib-1.2.11 /nix/store/dfgm23z5mvp7i3085dlq4rn0as41jp1p-openssl-1.1.1d ... All of these builds are performed in a sandbox, with no access to the network or global filesystem (e.g. no $HOME, /usr/bin, etc.), every file's timestamps are set to 1970-01-01T00:00:01 and the /nix/store filesystem is read-only. The Guix project takes this a bit further, with its `guix pack` command turning paths like this into a runnable docker image (amongst other options). Compare this to the way docker is used: we download and run some large binary blob ('image'), containing who-knows-what. These images are usually bloated with all sorts os unneeded cruft (due to the way docker builds images 'from the inside'); often a whole OS, package management tools, or at the very least a shell like busybox, all of which makes scaling harder and increases our attack surface. The build steps for those images are often insecure and unreproducible too, e.g. running commands like `apt-get install -y foo` or `pip install bar`, whose behaviour changes over time dependending on the contents of third-party servers. I really want to start using containers for their security and deployment advantages, but so much of their tooling and approach seems completely backwards to me :( (Note that it's possible to build docker images using Nix, but this requires running commands like 'docker load' which I can't get working on my Mac, and deploying to AWS ECS seems to require 'image registries' rather than just pointing to a file on S3. Urgh.)

codethief 6y ago
Totally unrelated question: Do Nix packages typically come with proper license & copyright notes (and source code, if necessary) and is there an easy way to extract them for the entire dependency tree? The reason I'm asking is that, as our team is getting closer to ship our $PRODUCT, we started worrying about the licenses of all the third-party software components we use. Needless to say, we're using some Docker image as base and now need to figure out what software it contains exactly…
- rnhmjoj 6y ago
  Yes, packages in Nixpkgs (the official repository of Nix packages) have mandatory copyright or license information. This can also be accessed programmatically, if you intend to scan all your dependencies: $ nix-env -qaA nixpkgs.vim --json | jq '."nixpkgs.vim".meta.license' { "fullName": "Vim License", "shortName": "vim", "spdxId": "Vim", "url": "https://spdx.org/licenses/Vim.html" } A large majority of the packages is built from source and so you can easily inspect it. For example, running `nix-build '<nixpkgs>' -A vim.src` will fetch the source tarball and link it in the current directory.
- chriswarbo 6y ago
  In addition to what rnhmjoj says, Nix can also check the license of packages as it's evaluating them (i.e. before fetching/building). The most obvious place this appears is the `allowUnfree` option, which defaults to `false` and refuses to evaluate proprietary packages. Details on how to whitelist/blacklist specific licenses, and how to define custom functions to decide this, are given at https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree
- chriswarbo 6y ago
  > and source code, if necessary Nix is completely based around source code, similar to Gentoo's emerge, BSD ports, etc. (in contrast to those based around binaries, like dpkg, RPM, etc.). More precisely, Nix works with "derivations", which have outputs (the paths in /nix/store which they define), inputs (the outputs of other derivations), and a "builder" (this is usually bash, with a build script given as argument). The files above which end in `.drv` define derivations. Fun fact: Nix doesn't care how these files are made; we can write our own tools to create .drv files, which is exactly what Guix and hnix do :) Notice that the runtime dependencies above don't contain any .drv files. That's because at runtime we only need (some of) the outputs of (some of) the derivations. Also note that the build dependencies contain lots of .drv files, since the build instructions are just as much a part of the 'source' as anything else. Some of those derivations have names like `.tar.gz.drv` and `.xz.drv`; those define how to fetch a particular archive, e.g. containing a URL, maybe a bunch of mirrors, a SHA256 hash to compare it against, etc. Internally, Nix doesn't distinguish between such "sources" and anything else; it's all just derivations. Since Nix builds tend to be reproducible (although that's not guaranteed), it's able to use a clever optimisation trick: packages (i.e. the outputs of derivations) are completely specified by their hash, so we can copy them from someone else (if they already have it), rather than building them ourselves, and the result should be the same. Similar to how we can checkout a particular git commit from someone who already has it, rather than having to recreate the tree ourselves by make all the same edits in the same order. (It's not exactly the same, since we need to trust that person isn't sending us something malicious). Hence Nix can use a (trusted) binary cache: this is just a folder full of derivation outputs, which we can query to see if our desired output has already been built, before we bother doing it ourselves. Binary caches are the reason I can do `nix-shell -p git` and have a git binary ready to go in a couple of seconds, despite that binary being defined in terms of those massive source-based dependency graphs.

codethief 6y ago

Totally unrelated question: Do Nix packages typically come with proper license & copyright notes (and source code, if necessary) and is there an easy way to extract them for the entire dependency tree? The reason I'm asking is that, as our team is getting closer to ship our $PRODUCT, we started worrying about the licenses of all the third-party software components we use. Needless to say, we're using some Docker image as base and now need to figure out what software it contains exactly…

rnhmjoj 6y ago
Yes, packages in Nixpkgs (the official repository of Nix packages) have mandatory copyright or license information. This can also be accessed programmatically, if you intend to scan all your dependencies: $ nix-env -qaA nixpkgs.vim --json | jq '."nixpkgs.vim".meta.license' { "fullName": "Vim License", "shortName": "vim", "spdxId": "Vim", "url": "https://spdx.org/licenses/Vim.html" } A large majority of the packages is built from source and so you can easily inspect it. For example, running `nix-build '<nixpkgs>' -A vim.src` will fetch the source tarball and link it in the current directory.
chriswarbo 6y ago
In addition to what rnhmjoj says, Nix can also check the license of packages as it's evaluating them (i.e. before fetching/building). The most obvious place this appears is the `allowUnfree` option, which defaults to `false` and refuses to evaluate proprietary packages. Details on how to whitelist/blacklist specific licenses, and how to define custom functions to decide this, are given at https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree
chriswarbo 6y ago
> and source code, if necessary Nix is completely based around source code, similar to Gentoo's emerge, BSD ports, etc. (in contrast to those based around binaries, like dpkg, RPM, etc.). More precisely, Nix works with "derivations", which have outputs (the paths in /nix/store which they define), inputs (the outputs of other derivations), and a "builder" (this is usually bash, with a build script given as argument). The files above which end in `.drv` define derivations. Fun fact: Nix doesn't care how these files are made; we can write our own tools to create .drv files, which is exactly what Guix and hnix do :) Notice that the runtime dependencies above don't contain any .drv files. That's because at runtime we only need (some of) the outputs of (some of) the derivations. Also note that the build dependencies contain lots of .drv files, since the build instructions are just as much a part of the 'source' as anything else. Some of those derivations have names like `.tar.gz.drv` and `.xz.drv`; those define how to fetch a particular archive, e.g. containing a URL, maybe a bunch of mirrors, a SHA256 hash to compare it against, etc. Internally, Nix doesn't distinguish between such "sources" and anything else; it's all just derivations. Since Nix builds tend to be reproducible (although that's not guaranteed), it's able to use a clever optimisation trick: packages (i.e. the outputs of derivations) are completely specified by their hash, so we can copy them from someone else (if they already have it), rather than building them ourselves, and the result should be the same. Similar to how we can checkout a particular git commit from someone who already has it, rather than having to recreate the tree ourselves by make all the same edits in the same order. (It's not exactly the same, since we need to trust that person isn't sending us something malicious). Hence Nix can use a (trusted) binary cache: this is just a folder full of derivation outputs, which we can query to see if our desired output has already been built, before we bother doing it ourselves. Binary caches are the reason I can do `nix-shell -p git` and have a git binary ready to go in a couple of seconds, despite that binary being defined in terms of those massive source-based dependency graphs.