11 ms·

A Gentle Introduction to the Nix Family (2019)

175 points by gurjeet 6y ago

gurjeet 6y ago
I have struggled with understanding Nix package manager for a long time. I use it on my macOS as a daily driver, but never understood the various pieces involved for me to comfortable make changes to my configuration. `nix-env -i` and `nix-env -e` were all I was comfortable with. This blog post does a good job of breaking down the Nix package manager's knowledge into decent sized chunks that one can easily understand and reason about.
avindroth 6y ago
I love the mood in this writing
burke 6y ago
If this is of interest, I’ve been publicly releasing a series of educational Nix videos I originally recorded for developers at Shopify that you may also enjoy: https://www.youtube.com/playlist?list=PLRGI9KQ3_HP_OFRG6R-p4iFgMSK1t5BHs https://www.youtube.com/playlist?list=PLRGI9KQ3_HP_OFRG6R-p4...
- ssivark 6y ago
  That’s great! Is any of that stuff Mac specific, or would all of it apply on Linux also? PS: These are about using the Nix package manager, rather than the Nix OS, right?
  - burke 6y ago
    It’s all about using Nix on macOS for now, but a vast majority of the topics apply on Linux as well. And yes, very very little specific coverage of NixOS.
- chaorace 6y ago
  Just wanted to comment that this playlist is fantastic. Thank you very much for contributing these tutorials!
- ShamelessC 6y ago
  Just got started and I'm already fascinated with generations and rolling back. How difficult is it to use as a complete replacement for Homebrew? Am I likely to find most of the packages provided by Homebrew?
  - danieldk 6y ago
    It's fairly realistic. On my Mac, I only use Homebrew for casks. I have everything else installed through Nix. One of the nicest things is that I have nearly completely overridden macOS' outdated utilities with a GNU userland (without any annoying g- prefix). I also define most of my user configuration with home-manager (both on Mac, NixOS, and work's Ubuntu servers). On a new machine I just run nix run nixpkgs.home-manager home-manager switch My home-manager configuration: https://github.com/danieldk/nix-home https://github.com/danieldk/nix-home
    - ShamelessC 6y ago
      That's awesome. Just finished the video on home manager which seems very useful. Your videos being directed at co-workers actually serves to highlight the usefulness of this tool. "Oh, you're missing dependency x, y and z? Here, just copy and paste this nix command to get my Nix environment." Then you can immediately continue discussing more productive things. I'm curious about generations. If you roll back to a previous generation and then continue to modify that, will the generation be overwritten? If not, is there a way to track only the history of generations that contribute a change to their current generation? Also, I saw you run NixOS in a droplet. Do you have any experience running it locally as a desktop environment? I'm curious how mature it is as a full blown Linux Desktop.
      - burke 6y ago
        Generations are linear, so if you are on 6, roll back to 3, and make a new change (I'm 90% sure) it will create 7. I've used NixOS a liiiiitle bit on a laptop but not enough to really get into it. We're toying with the idea of exploring supporting NixOS as a primary local development environment (and a host for cloud-based development environments) but there's lot of work to do to make that work well.
- JamesSwift 6y ago
  I had to log in just so I could upvote and thank you for doing these! My biggest issue with nix on MacOS (besides not grokking the DSL even after a couple years) is that there are a decent handful of packages that dont support Darwin. How do you handle those?
  - burke 6y ago
    Some of them actually do support darwin and we have a function in our company overlay: withDarwinEnabled = drv: drv.overrideAttrs (oldAttrs: { meta = with super.stdenv.lib.platforms; { platforms = darwin ++ oldAttrs.meta.platforms; }; }); but really, we've only run into a couple of things we've cared about that are Linux-only, and just PR'd fixes upstream and duplicated them to our overlay temporarily.
- gurjeet 6y ago
  Thanks for making those videos! These are the kinds of videos, I think, the newcomers need to see to get value out of Nix. Can you please share what screencasting software/setup you're using? It seems pretty effortless on your part to get your picture-in-picture move around.
k__ 6y ago
Everyone read this! This Nix/Guix level of package management is what every distribution should aim for.
- j88439h84 6y ago
  NixOS is available, not sure what other distros offer that it doesn't.
  - k__ 6y ago
    Me neither. But I see some heavy Slack, Arch and Void proponents.
  - solarkraft 6y ago
    An easy, relatively worry free installation/out of the box experience? The option to install uncommon software in a way that may be crappy but would work (make install?)? That's what's holding me back, although I'm fascinated by the concept.
    - chpatrick 6y ago
      I would say the installation experience might be a bit more involved, but once it's installed it's much more worry-free for me. On every other distribution I've used (including Ubuntu and co), it's very easy to break the system completely. On Nix, it's almost impossible because you can always boot directly into the state before your update. Once you've set up your configuration, you can install it instantly on other machines by copying one file. On other distributions, you would have to re-install and configure everything by hand. Many of the tweaks for which you usually have to read Arch Wiki are also built into Nix, so you just have to write services.myService.myOption = foo; and you're done. For the second point, if you've got a package that just builds with the usual ./configure, make, make install, here is all you need to get a nix package for it: { stdenv }: stdenv.mkDerivation { name = "my-package"; src = ./.; buildInputs = [ ... whatever dependencies you need ...]; } And the standard build commands will be run automatically. The only thing that's kind of annoying is installing binary distributions of packages since they need to be patched to work on Nix, but it's not the end of the world. Also, nixpkgs is so universal that I very rarely find myself having to add a package that's not in it.
xena 6y ago
The Esperanto translation is fantastic
- senorsmile 6y ago
  I did not notice that at first! It appears to be complete too.
koolba 6y ago
This is a fun read. Curious how this works for large systems: > This shell instance is special because it only contains sufficient information just to make GNU Hello, available. We can even inspect the value of $PATH, here: > (list of 20 or so paths to binaries) Do binary path lookups slow to a crawl when you’ve got 100s or 1000s of separate directory paths to look through? IIUC, there’s a separate directory added to PATH for each binary right? Which presumably would be a disk I/O to read the file list for PATH matching for each command.
- xyzzy_plugh 6y ago
  Good question, you can of course test this yourself with `time`. In the case of Bash, non-built-in commands are stored in a lookup table, exposed via the hash built-in (`help hash`). So, there should only be a one-time performance hit per shell instance. Generally speaking, stat'ing hundreds of directories is surprisingly quick, most binaries do it all the time, when loading libraries for example. You can `strace` to see the numerous file tree crawls.
- singron 6y ago
  It usually doesnt get that bad because transitive dependencies don't get added to the path. I.e. you actually need to have a list of 1000 direct dependencies to get 1000 path entries. If for some reason you have a crazy amount of paths like that and path lookups were a problem, you could use buildEnv to create a single bin directory that had symlinks to all the binaries in it.
enriquto 6y ago
> We can even inspect the value of $PATH, here: > (...) > Your output is going to be different from mine because of the hashes in the store paths. Does this mean that a nix install is not reproducible? If the same installation is performed twice, the file names and contents will be different due to some randomness?
- exdsq 6y ago
  From my understanding it is reproducible, although the hashes might be different they'll still be linked in the same way. Give someone your nix.conf and they will be able to build your machine. You reference things by variable name anyway.
- pingiun 6y ago
  The hashes will only be different if you install different versions of the derivation (package). All the inputs of a derivation, so even selected plugins and compile options influence the hash. That said, if me and you agreed on a specific git commit of the nixpkgs repo and build some packages from that, our hashes would be the same. This knowledge is also used to download packages from the binary cache. You can see the automatic building of the packages on hydra.nixos.org, from there they will be made avaiable on the binary cache.
- ashtonkem 6y ago
  It is reproducible, but only for the same exact version, and with no overlays at all. If you’re running this with a different channel, or months later, you’ll get drastically different hashes, since the software will be different.
rq1 6y ago
I use NixOS as my main OS on my Blade 15. It was a bit confusing to set up all the bumblebee stuff to use the RTX2080 on demand for my ML tasks (nixGL is the solution). All my projects include a `default.nix` to start a shell with all the necessary dependencies. I’m very happy with it now. I’ll never look back.
ShamelessC 6y ago
Would love to see NixOS available as an image for Windows Subsystem for Linux. Looks like it's been an open issue for awhile. https://github.com/NixOS/nixpkgs/issues/30391 https://github.com/NixOS/nixpkgs/issues/30391 Seems someone may have figured it out - https://github.com/Trundle/NixOS-WSL https://github.com/Trundle/NixOS-WSL

175 points by gurjeet 6y ago

I have struggled with understanding Nix package manager for a long time. I use it on my macOS as a daily driver, but never understood the various pieces involved for me to comfortable make changes to my configuration. `nix-env -i` and `nix-env -e` were all I was comfortable with. This blog post does a good job of breaking down the Nix package manager's knowledge into decent sized chunks that one can easily understand and reason about.

avindroth 6y ago

I love the mood in this writing

burke 6y ago

If this is of interest, I’ve been publicly releasing a series of educational Nix videos I originally recorded for developers at Shopify that you may also enjoy: https://www.youtube.com/playlist?list=PLRGI9KQ3_HP_OFRG6R-p4iFgMSK1t5BHs https://www.youtube.com/playlist?list=PLRGI9KQ3_HP_OFRG6R-p4...

ssivark 6y ago
That’s great! Is any of that stuff Mac specific, or would all of it apply on Linux also? PS: These are about using the Nix package manager, rather than the Nix OS, right?
- burke 6y ago
  It’s all about using Nix on macOS for now, but a vast majority of the topics apply on Linux as well. And yes, very very little specific coverage of NixOS.
chaorace 6y ago
Just wanted to comment that this playlist is fantastic. Thank you very much for contributing these tutorials!
ShamelessC 6y ago
Just got started and I'm already fascinated with generations and rolling back. How difficult is it to use as a complete replacement for Homebrew? Am I likely to find most of the packages provided by Homebrew?
- danieldk 6y ago
  It's fairly realistic. On my Mac, I only use Homebrew for casks. I have everything else installed through Nix. One of the nicest things is that I have nearly completely overridden macOS' outdated utilities with a GNU userland (without any annoying g- prefix). I also define most of my user configuration with home-manager (both on Mac, NixOS, and work's Ubuntu servers). On a new machine I just run nix run nixpkgs.home-manager home-manager switch My home-manager configuration: https://github.com/danieldk/nix-home https://github.com/danieldk/nix-home
  - ShamelessC 6y ago
    That's awesome. Just finished the video on home manager which seems very useful. Your videos being directed at co-workers actually serves to highlight the usefulness of this tool. "Oh, you're missing dependency x, y and z? Here, just copy and paste this nix command to get my Nix environment." Then you can immediately continue discussing more productive things. I'm curious about generations. If you roll back to a previous generation and then continue to modify that, will the generation be overwritten? If not, is there a way to track only the history of generations that contribute a change to their current generation? Also, I saw you run NixOS in a droplet. Do you have any experience running it locally as a desktop environment? I'm curious how mature it is as a full blown Linux Desktop.
    - burke 6y ago
      Generations are linear, so if you are on 6, roll back to 3, and make a new change (I'm 90% sure) it will create 7. I've used NixOS a liiiiitle bit on a laptop but not enough to really get into it. We're toying with the idea of exploring supporting NixOS as a primary local development environment (and a host for cloud-based development environments) but there's lot of work to do to make that work well.
JamesSwift 6y ago
I had to log in just so I could upvote and thank you for doing these! My biggest issue with nix on MacOS (besides not grokking the DSL even after a couple years) is that there are a decent handful of packages that dont support Darwin. How do you handle those?
- burke 6y ago
  Some of them actually do support darwin and we have a function in our company overlay: withDarwinEnabled = drv: drv.overrideAttrs (oldAttrs: { meta = with super.stdenv.lib.platforms; { platforms = darwin ++ oldAttrs.meta.platforms; }; }); but really, we've only run into a couple of things we've cared about that are Linux-only, and just PR'd fixes upstream and duplicated them to our overlay temporarily.
gurjeet 6y ago
Thanks for making those videos! These are the kinds of videos, I think, the newcomers need to see to get value out of Nix. Can you please share what screencasting software/setup you're using? It seems pretty effortless on your part to get your picture-in-picture move around.

k__ 6y ago

Everyone read this! This Nix/Guix level of package management is what every distribution should aim for.

j88439h84 6y ago
NixOS is available, not sure what other distros offer that it doesn't.
- k__ 6y ago
  Me neither. But I see some heavy Slack, Arch and Void proponents.
- solarkraft 6y ago
  An easy, relatively worry free installation/out of the box experience? The option to install uncommon software in a way that may be crappy but would work (make install?)? That's what's holding me back, although I'm fascinated by the concept.
  - chpatrick 6y ago
    I would say the installation experience might be a bit more involved, but once it's installed it's much more worry-free for me. On every other distribution I've used (including Ubuntu and co), it's very easy to break the system completely. On Nix, it's almost impossible because you can always boot directly into the state before your update. Once you've set up your configuration, you can install it instantly on other machines by copying one file. On other distributions, you would have to re-install and configure everything by hand. Many of the tweaks for which you usually have to read Arch Wiki are also built into Nix, so you just have to write services.myService.myOption = foo; and you're done. For the second point, if you've got a package that just builds with the usual ./configure, make, make install, here is all you need to get a nix package for it: { stdenv }: stdenv.mkDerivation { name = "my-package"; src = ./.; buildInputs = [ ... whatever dependencies you need ...]; } And the standard build commands will be run automatically. The only thing that's kind of annoying is installing binary distributions of packages since they need to be patched to work on Nix, but it's not the end of the world. Also, nixpkgs is so universal that I very rarely find myself having to add a package that's not in it.

xena 6y ago

The Esperanto translation is fantastic

senorsmile 6y ago
I did not notice that at first! It appears to be complete too.

koolba 6y ago

This is a fun read. Curious how this works for large systems: > This shell instance is special because it only contains sufficient information just to make GNU Hello, available. We can even inspect the value of $PATH, here: > (list of 20 or so paths to binaries) Do binary path lookups slow to a crawl when you’ve got 100s or 1000s of separate directory paths to look through? IIUC, there’s a separate directory added to PATH for each binary right? Which presumably would be a disk I/O to read the file list for PATH matching for each command.

xyzzy_plugh 6y ago
Good question, you can of course test this yourself with `time`. In the case of Bash, non-built-in commands are stored in a lookup table, exposed via the hash built-in (`help hash`). So, there should only be a one-time performance hit per shell instance. Generally speaking, stat'ing hundreds of directories is surprisingly quick, most binaries do it all the time, when loading libraries for example. You can `strace` to see the numerous file tree crawls.
singron 6y ago
It usually doesnt get that bad because transitive dependencies don't get added to the path. I.e. you actually need to have a list of 1000 direct dependencies to get 1000 path entries. If for some reason you have a crazy amount of paths like that and path lookups were a problem, you could use buildEnv to create a single bin directory that had symlinks to all the binaries in it.

enriquto 6y ago

> We can even inspect the value of $PATH, here: > (...) > Your output is going to be different from mine because of the hashes in the store paths. Does this mean that a nix install is not reproducible? If the same installation is performed twice, the file names and contents will be different due to some randomness?

exdsq 6y ago
From my understanding it is reproducible, although the hashes might be different they'll still be linked in the same way. Give someone your nix.conf and they will be able to build your machine. You reference things by variable name anyway.
pingiun 6y ago
The hashes will only be different if you install different versions of the derivation (package). All the inputs of a derivation, so even selected plugins and compile options influence the hash. That said, if me and you agreed on a specific git commit of the nixpkgs repo and build some packages from that, our hashes would be the same. This knowledge is also used to download packages from the binary cache. You can see the automatic building of the packages on hydra.nixos.org, from there they will be made avaiable on the binary cache.
ashtonkem 6y ago
It is reproducible, but only for the same exact version, and with no overlays at all. If you’re running this with a different channel, or months later, you’ll get drastically different hashes, since the software will be different.

rq1 6y ago

I use NixOS as my main OS on my Blade 15. It was a bit confusing to set up all the bumblebee stuff to use the RTX2080 on demand for my ML tasks (nixGL is the solution). All my projects include a `default.nix` to start a shell with all the necessary dependencies. I’m very happy with it now. I’ll never look back.

ShamelessC 6y ago

Would love to see NixOS available as an image for Windows Subsystem for Linux. Looks like it's been an open issue for awhile. https://github.com/NixOS/nixpkgs/issues/30391 https://github.com/NixOS/nixpkgs/issues/30391 Seems someone may have figured it out - https://github.com/Trundle/NixOS-WSL https://github.com/Trundle/NixOS-WSL