5 ms·

Nix Derivations, Without Guessing

136 points by surprisetalk 2y ago

deleted 2y ago
[deleted]
Mayora13 2y ago
Now we are really interested in why they do this—probably for backward compatibility with a 20‑year‑old implementation. It’s not messy, just particular.
klysm 2y ago
What a bizarre and arcane incantation
kennethallen 2y ago
I walk away from every article on or attempt to use Nix more mystified
- p1necone 2y ago
  Yeah Nix seems like an insanely useful concept (declarative, reproducable, diffable, source controllable definitions of linux environments). But actually using it is a nightmare. I think there's some point on the spectrum between "commit your dotfiles to git" and Nix that would be useful, I don't know what it is though. Containerization is kinda like this, but they're entirely imperatively defined rather than declarative and I wouldn't really want to use a docker container as my main system environment.
  - o11c 2y ago
    I'm pretty sure the best middle ground that currently exists is the various "immutable, snapshot to upgrade" distros out there.
    - pxc 2y ago
      Depends on your use cases. I use Nix all the time at work but I don't use NixOS there at all. (I'd like to, but there are barriers and it's not a priority.) Distros like that don't address my use cases at all.
    - microtonal 2y ago
      Yes and no. It has some of the same advantages, but not the declarative definition of a system. For me (even though I use NixOS the desktop) there is a difference between desktops and servers. I can set up a desktop pretty quickly - install a bunch of flatpaks, checkout a bunch of dotfiles and I don't do it often. Also, popular desktops like Fedora tend to be far more polished than the NixOS desktop, which has all kinds of glitches. Just to name two current ones: (1) gdm login will fail if you are too fast and log in before WiFi is up (usually you are thrown back in gdm, sometimes the session freezes up completly); (2) fwupd firmware updates usually fail. On the other hand, on servers and remote development VMs, the setup work is annoying because I spin up/down machines far more frequently and managing them as pets gets old pretty quickly. So NixOS is much nicer, because you can have a system up and identical in 5 minutes. You could of course approximate it with something like Ansible on non-NixOS. Though, I think the differences will become smaller since Fedora-based immutable systems will switch from OSTree to bootable containers soon [1]. Of course, you can use Nix on another immutable distro than NixOS. [1] https://docs.fedoraproject.org/en-US/bootc/getting-started/ https://docs.fedoraproject.org/en-US/bootc/getting-started/
  - chubot 2y ago
    I honestly wonder why there are such divergent opinions between say git and Nix? (I've never used Nix, but I use git all the time) Is Nix so much harder to use than git? git also has a clear model (the Merkle tree of file and directory nodes) but a famously unfriendly UI (git checkout does 5 or more unrelated things) Some people do not like git, but I get the feeling that most people just use it, and get on with their day. Why the difference with Nix? Maybe because building packages is inherently slower. Whereas you can quickly get yourself intro trouble with git, but you can also quickly get out of trouble (rm -rf, git clone) Maybe Nix is more stateful? Although the git index/staging area I find to be a fabulously odd piece of state, and honestly breaks all the rules when I think about CLIs Also Nix does rely on a very big global repo, whereas git doesn't It also seems that Nix's model is less clean, and perhaps doesn't match the problem domain as well ... there are disagreements on whether it is "reproducible", etc. Or maybe it's just a harder problem
    - amarshall 2y ago
      Git doesn’t really care at all about the content. Nix does. Really they’re not reasonably comparable at all.
    - pxc 2y ago
      You may not use Nix, but some pretty cool Nix tools use your work. :) Just today I was working on some integrations with a runtime dependency resolver for shell programs that uses OSH for parsing on the job. We use Nix to manage our development environments, and we use it to make some small wrappers and tools written in Bash into something portable and reproducible that we can include in development environments that run both on Linux and macOS. Historically we've just included them inline in our Nix files, but thanks to resholve, we're switching to a nicer system so that they live in separate files that are more pleasant to edit. The "source code" of the scripts lives in the repo as normal bash scripts, but when they get built into the development environment, all command invocations get automatically replaced with hard-coded references to the paths of the relevant executables in the Nix store, the shebang gets pinned to a specific bash version in the same way, and they also get run through ShellCheck. Now we have not only a really nice and quick way to define portable wrappers in our Nix code, but a sane way to manage longer scripts without any portability issues. So thanks for your cool shell and associated tools and libs! > Or maybe it's just a harder problem Fwiw, I think this is true. > Some people do not like git, but I get the feeling that most people just use it, and get on with their day. I think one of the other devs on my team probably feels this way. He thinks it's conceptually cool but is somewhat horrified by the complexity and UI. He makes simple uses of it in ways that have precedent on our team, but never really dives in. > Why the difference with Nix? Nix didn't have a celebrity author to spur adoption early on, in some ways it can be slow, and I think maybe it isn't as much better than entrenched alternative stacks than Git was as opposed to SVN. The pain of SVN was very acute for the average developer. I'm not sure that any of the pains of dependency hell, stateful configuration management, distribution of portable executables, etc., are quite as acute for the average operator or developer as that. People who feel Nix makes their professional lives easier tend to have come to it after their career has inflicted more specialized pain upon them. Using Git also doesn't generally (ever?) involve writing code in a Turing-complete language, but to make the best use of Nix, you do have to do that. The paradigm of that language is not very mainstream, either, and although it's generally suited to its domain imo, it certainly has some warts.
  - tombert 2y ago
    I run NixOS on every computer I'm allowed to install it and I really don't think it's hard to use, just different. Adapting to a new workflow is hard, but I don't know that NixOS is intrinsically more difficult than any other Linux. I wouldn't want a docker container as my main environment, but I do like having NixOS managing my main environment for a few reasons. First, the declarative nature of everything makes it clear and easy to know what is actually installed on my computer. I can simply look at my configuration file and see if a program is installed, and if it's not. If I want to uninstall something, I delete the program from the configuration.nix and rebuild. This might not seem insignificant, but to me it's game changing. How many times have you uninstalled things in Ubuntu or something and had a ton of lingering transitive dependencies? How many times have you been trying to debug a problem, end up installing a million things, and then painstakingly having to track down every unnecessary dependency and transitive dependency that you ended up installing, only to miss something for months? Maybe most people here are better at this than I am, but these things happened to me all the time. Second, the declarative nature of NixOS makes snapshotting trivial. Again, this is game-changing for me, and it makes fixing stuff on my computer more fun and less scary. If I break something, all I have to do is reboot and choose the last generation, then fix it. This might not seem like a big deal, and again maybe for people smarter than me it's not, but for me it completely changed the way I deal with computers. When I first started using Ubuntu, when I would do something like break the video driver or the WiFi driver, I would end up having to nuke the system and start again, because I would get into a state where I didn't know how to fix it. I probably could fix these things now, I've been doing this stuff for awhile, but even still, it's nice to be able to not ever have to worry about getting into a state like that.
    - ChocolateGod 2y ago
      My biggest gripe with Nix (from real world experience) is that my .nix files randomly break due to changes and I have to spend my time going through Github commits to see what changed in the settings I used to fix it. That and when things do error, the error messages may as well be generated from /dev/random
      - adamtulinius 2y ago
  - eru 2y ago
    > [...] but they're entirely imperatively defined rather than declarative [...] The conceptual problem with Docker isn't imperative vs declarative. It's that Docker doesn't even try to be reproducible. Executing the same Dockerfile twice doesn't necessarily build you the same container. (Imperative vs declarative is still an interesting problem to think about, it's just independent of reproducibility in the abstract.)
  - darthrupert 2y ago
    I'm also one of the people who have tried Nix(OS) a couple of times and found it too much of a hassle, but nightmare is exaggerating a bit, I feel. Nix's strength and weakness is that it wants to take over everything, and if you want to do something without it, you might be in a world of pain. And after doing more or less standard unixy things for 20+ years, it's difficult to hand over control to a new thing like that.
  - xorcist 2y ago
    > some point on the spectrum between "commit your dotfiles to git" and Nix That would be configuration management tools such as Salt/Puppet/Ansible/Chef. They were popular ten years ago, and gained a lot of exposure as the devops movement gained ground, but they never stopped being useful. Having your non-running state defined declarative is powerful, and if you can define a single source of truth for entire distributed systems, that suddenly makes you able to reason about state on whole systems.
    - XorNot 2y ago
      What I can't currently figure out with Nix though is how I kill off dependency explosions though. I want to reach into a big tree of derivations and force library and compiler versions to a single common one so we don't, for example, have 6 rust compilers being used.
      - chriswarbo 2y ago
        The usual approach is to give Nixpkgs some `overlays`, which override the attributes you want. This can be handy in conjunction with attributes like `.override`, `.overrideAttrs`, etc. for swapping-out things deeper in the dependency graph. The https://codeberg.org/amjoseph/infuse.nix https://codeberg.org/amjoseph/infuse.nix project looks nice as a way to simplify annoying chains of overrides; though I haven't used it personally.
- amarshall 2y ago
  For what it’s worth, nothing in this article is really necessarily for general usage of Nix, as the derivation format is mostly abstracted-away, like how the OCI image format is irrelevant to everyday authoring of Dockerfiles.
- drowsspa 2y ago
  Yeah, it's like those famous posts comparing monads to burritos or something
tikhonj 2y ago
For context, this particular article is a cool deep dive into how Nix works, but it doesn't represent what using Nix + Nixpkgs is like in practice. I've been using Nix personally and professionally for almost 10 years now (yikes has time passed quickly!) and I have never needed to operate at the level of derivations like this.
- setheron 2y ago
  Professionally for 10 years ? I would like to hear more. I remember Target funded some earlier Nix work like lorri. Interestingly, I found operating at this level to have been a necessity if I wanted to fix or write a derivation. Otherwise I would face random errors and be left clueless as to what the problem was. Writing a derivation builder (nix pill) was illuminating.
  - tikhonj 2y ago
    Yep, I was one of the people who originally started using Nix on that particular team at Target back in 2016. I wasn't the only person interested in Nix but, if I'm recalling correctly, I was the person who first set it up and supported our Nix environment on Linux and macOS for the first year or two. After we scaled up a bit we hired some more Nix-focused people and some consultants (for stuff like lorri), so I stopped having to do much Nix stuff day-to-day. That was a really special team. Nobody "let" me use Nix, we just did it because it made sense to us, and I went out of my way to support it internally not because I felt pressured to do it, but because I was enthusiastic about making it work. The culture there reminded me of this story[1] about getting Python into production at Goldman—and it turns out that the VP who built the supply chain team at Target started his career in strats at Goldman and was heavily inspired by the same Armen from that story :) [1]: https://news.ycombinator.com/item?id=29104401 https://news.ycombinator.com/item?id=29104401 After leaving Target, I also used Nix for my own work setup at CX Score. That was a seed-stage startup and I didn't want to push Nix on anybody else, but it made my own work setup much nicer. Most recently I moved to Mercury where we use Nix at a pretty big scale (several hundred developers). We have a team that handles most of it so I haven't had to do much myself, but I've helped a few people work through Nix issues just because I could :) I'm also one of the weirdos who uses NixOS on my work laptop instead of getting a macbook. And, in parallel, I've been using NixOS on all my personal machines ever since I first discovered it. That was relatively shortly before joining Target, so, if I had to guess, late 2015 or early 2016. Coming up on 10 years shortly, even though it really does not feel like it...
setheron 2y ago
What a fun read. Thanks for the nice call out on my blog. If you're also interested I (Farid) also had a follow up where I follow up on how the hashes are calculated. (Using the error to get the hashes also bugged me) I use it to also create vanity hashes :) https://fzakaria.com/2025/03/28/what-s-in-a-nix-store-path https://fzakaria.com/2025/03/28/what-s-in-a-nix-store-path https://fzakaria.com/2025/03/27/nix-vanity-store-paths https://fzakaria.com/2025/03/27/nix-vanity-store-paths
pveierland 2y ago
Nix derivations are pretty neat! I've been building a Nix store navigator for MyNixOS v2. It can help you get a feel for how derivations are connected in Nix: https://v2.mynixos.com/nix/store/lsk1c4v03y4lmpxdcwal99nm5nwz2rj7-texlive-2024-env.drv https://v2.mynixos.com/nix/store/lsk1c4v03y4lmpxdcwal99nm5nw... PS: The controls to the upper right can be used to limit what is downloaded.
Joker_vD 2y ago
Ah, don't you just like it when projects that use hashsums, calculate them in such a way that you can't actually recalculate them on your own? And when you start digging, you find not only that it uses the same basic design of XML-DSig while barely escaping its fatal flaws, it also uses some bizarre data encodings nobody else uses for anything. And then the resulting hash is not even the truncation of the actual hash, it has an additional (again, entirely undocumented) strange post-processing step for unspecified reasons.
danlitt 2y ago
c.f. guix, where the command is simply `guix hash`.
chombier 2y ago
What I found hard with Nix is the sheer amount of things I had to get familiar with before it started to really click: - nix, the command-line tool - nix, the language - nixpkgs with the general API/idioms (overriding, overlays) - individual nixpkgs packages that sometimes deviate from common practices - flakes (which I haven't properly looked into yet) The Nix pills series [1] and nixpkgs documentation [2] do help a lot, but that is quite a lot to process. [1] https://nixos.org/guides/nix-pills https://nixos.org/guides/nix-pills [2] https://nixos.org/manual/nixpkgs/stable/ https://nixos.org/manual/nixpkgs/stable/
- woile 2y ago
  they are a lot of things, but it's hard to avoid otherwise. Overall I'm glad it's relatively cohesive how it all works.
- angra_mainyu 2y ago
  Controversial, but I found just going through the docs, attempting to write my own flake, throwing it at ChatGPT and asking it to review it and give me feedback was enough to get started. I never ask it for code, just to review code - keeps me sharp and makes the iteration cycles faster.
  - pxc 2y ago
    ChatGPT is generally pretty bad at writing Nix code IME, and has a much greater tendency to hallucinate than when writing code in other languages. But it's better at advice and review than writing code. And if you want to ask it about Nixpkgs conventions or the behavior of some Nix codebase, it does much better if you dump the relevant source files in the chat for it to consume. It's important with Nix to seriously understand what you're doing, so I think the way you've described is the most appropriate way to use an LLM with Nix. Use it to point you to docs, ask it questions about some code and then read the code closely, ask it which docs to read, ask it to point you to example repos on GitHub. If you use it this way it can actually be helpful. If you have a longer time horizon, interacting with real community members is even better, and of course can also enrich the corpus for LLMs like ChatGPT. Discourse.nixos.org has a lot of smart and helpful people whose insight is more reliable than the advice of an LLM, so it's worth visiting as well.
klysm 2y ago
Is there any technical pathway for changing the underlying hash function? Similar to how git is kind of stuck on SHA1, what would it take to get this onto SHA512?
- chriswarbo 2y ago
  I think it could be done in a backwards-compatible way, since existing hashes fit a known pattern; e.g. (silly example) store paths using SHA512 could be distinguished using prefix like 'sha512_'. One problem would be cache misses and duplication of files. I'm not completely up to date on Git's hash situation, but it seems to me like SHA256 is usable; but only for new repos. I imagine lots of existing infrastructure assumes SHA1 hashes though, and would break for silly reasons like different lengths. Relatedly, Nix has a new feature called `git-hashing` which can use Git tree hashes to validate fixed-output derivations, rather than requiring two separate hashes; e.g. the following `fetchTreeFromGitHub` function uses a single `tree` argument for a Git tree ID, rather than needing separate `rev` and `hash` arguments: { owner, repo, tree }: (pkgs.fetchFromGitHub { inherit owner repo; rev = tree; hash = builtins.convertHash { hash = tree; hashAlgo = "sha1"; toHashFormat = "sri"; }; }).overrideAttrs (_: { outputHashMode = "git"; }) Unfortunately, this git-hashing functionality only works for SHA1; so it's a good example of the infrastructure around Git not allowing us to move away from SHA1!

136 points by surprisetalk 2y ago

deleted 2y ago

[deleted]

Now we are really interested in why they do this—probably for backward compatibility with a 20‑year‑old implementation. It’s not messy, just particular.

klysm 2y ago

What a bizarre and arcane incantation

kennethallen 2y ago

I walk away from every article on or attempt to use Nix more mystified

p1necone 2y ago
Yeah Nix seems like an insanely useful concept (declarative, reproducable, diffable, source controllable definitions of linux environments). But actually using it is a nightmare. I think there's some point on the spectrum between "commit your dotfiles to git" and Nix that would be useful, I don't know what it is though. Containerization is kinda like this, but they're entirely imperatively defined rather than declarative and I wouldn't really want to use a docker container as my main system environment.
- o11c 2y ago
  I'm pretty sure the best middle ground that currently exists is the various "immutable, snapshot to upgrade" distros out there.
  - pxc 2y ago
    Depends on your use cases. I use Nix all the time at work but I don't use NixOS there at all. (I'd like to, but there are barriers and it's not a priority.) Distros like that don't address my use cases at all.
  - microtonal 2y ago
    Yes and no. It has some of the same advantages, but not the declarative definition of a system. For me (even though I use NixOS the desktop) there is a difference between desktops and servers. I can set up a desktop pretty quickly - install a bunch of flatpaks, checkout a bunch of dotfiles and I don't do it often. Also, popular desktops like Fedora tend to be far more polished than the NixOS desktop, which has all kinds of glitches. Just to name two current ones: (1) gdm login will fail if you are too fast and log in before WiFi is up (usually you are thrown back in gdm, sometimes the session freezes up completly); (2) fwupd firmware updates usually fail. On the other hand, on servers and remote development VMs, the setup work is annoying because I spin up/down machines far more frequently and managing them as pets gets old pretty quickly. So NixOS is much nicer, because you can have a system up and identical in 5 minutes. You could of course approximate it with something like Ansible on non-NixOS. Though, I think the differences will become smaller since Fedora-based immutable systems will switch from OSTree to bootable containers soon [1]. Of course, you can use Nix on another immutable distro than NixOS. [1] https://docs.fedoraproject.org/en-US/bootc/getting-started/ https://docs.fedoraproject.org/en-US/bootc/getting-started/
- chubot 2y ago
  I honestly wonder why there are such divergent opinions between say git and Nix? (I've never used Nix, but I use git all the time) Is Nix so much harder to use than git? git also has a clear model (the Merkle tree of file and directory nodes) but a famously unfriendly UI (git checkout does 5 or more unrelated things) Some people do not like git, but I get the feeling that most people just use it, and get on with their day. Why the difference with Nix? Maybe because building packages is inherently slower. Whereas you can quickly get yourself intro trouble with git, but you can also quickly get out of trouble (rm -rf, git clone) Maybe Nix is more stateful? Although the git index/staging area I find to be a fabulously odd piece of state, and honestly breaks all the rules when I think about CLIs Also Nix does rely on a very big global repo, whereas git doesn't It also seems that Nix's model is less clean, and perhaps doesn't match the problem domain as well ... there are disagreements on whether it is "reproducible", etc. Or maybe it's just a harder problem
  - amarshall 2y ago
    Git doesn’t really care at all about the content. Nix does. Really they’re not reasonably comparable at all.
  - pxc 2y ago
    You may not use Nix, but some pretty cool Nix tools use your work. :) Just today I was working on some integrations with a runtime dependency resolver for shell programs that uses OSH for parsing on the job. We use Nix to manage our development environments, and we use it to make some small wrappers and tools written in Bash into something portable and reproducible that we can include in development environments that run both on Linux and macOS. Historically we've just included them inline in our Nix files, but thanks to resholve, we're switching to a nicer system so that they live in separate files that are more pleasant to edit. The "source code" of the scripts lives in the repo as normal bash scripts, but when they get built into the development environment, all command invocations get automatically replaced with hard-coded references to the paths of the relevant executables in the Nix store, the shebang gets pinned to a specific bash version in the same way, and they also get run through ShellCheck. Now we have not only a really nice and quick way to define portable wrappers in our Nix code, but a sane way to manage longer scripts without any portability issues. So thanks for your cool shell and associated tools and libs! > Or maybe it's just a harder problem Fwiw, I think this is true. > Some people do not like git, but I get the feeling that most people just use it, and get on with their day. I think one of the other devs on my team probably feels this way. He thinks it's conceptually cool but is somewhat horrified by the complexity and UI. He makes simple uses of it in ways that have precedent on our team, but never really dives in. > Why the difference with Nix? Nix didn't have a celebrity author to spur adoption early on, in some ways it can be slow, and I think maybe it isn't as much better than entrenched alternative stacks than Git was as opposed to SVN. The pain of SVN was very acute for the average developer. I'm not sure that any of the pains of dependency hell, stateful configuration management, distribution of portable executables, etc., are quite as acute for the average operator or developer as that. People who feel Nix makes their professional lives easier tend to have come to it after their career has inflicted more specialized pain upon them. Using Git also doesn't generally (ever?) involve writing code in a Turing-complete language, but to make the best use of Nix, you do have to do that. The paradigm of that language is not very mainstream, either, and although it's generally suited to its domain imo, it certainly has some warts.
- tombert 2y ago
  I run NixOS on every computer I'm allowed to install it and I really don't think it's hard to use, just different. Adapting to a new workflow is hard, but I don't know that NixOS is intrinsically more difficult than any other Linux. I wouldn't want a docker container as my main environment, but I do like having NixOS managing my main environment for a few reasons. First, the declarative nature of everything makes it clear and easy to know what is actually installed on my computer. I can simply look at my configuration file and see if a program is installed, and if it's not. If I want to uninstall something, I delete the program from the configuration.nix and rebuild. This might not seem insignificant, but to me it's game changing. How many times have you uninstalled things in Ubuntu or something and had a ton of lingering transitive dependencies? How many times have you been trying to debug a problem, end up installing a million things, and then painstakingly having to track down every unnecessary dependency and transitive dependency that you ended up installing, only to miss something for months? Maybe most people here are better at this than I am, but these things happened to me all the time. Second, the declarative nature of NixOS makes snapshotting trivial. Again, this is game-changing for me, and it makes fixing stuff on my computer more fun and less scary. If I break something, all I have to do is reboot and choose the last generation, then fix it. This might not seem like a big deal, and again maybe for people smarter than me it's not, but for me it completely changed the way I deal with computers. When I first started using Ubuntu, when I would do something like break the video driver or the WiFi driver, I would end up having to nuke the system and start again, because I would get into a state where I didn't know how to fix it. I probably could fix these things now, I've been doing this stuff for awhile, but even still, it's nice to be able to not ever have to worry about getting into a state like that.
  - ChocolateGod 2y ago
    My biggest gripe with Nix (from real world experience) is that my .nix files randomly break due to changes and I have to spend my time going through Github commits to see what changed in the settings I used to fix it. That and when things do error, the error messages may as well be generated from /dev/random
    - adamtulinius 2y ago
- eru 2y ago
  > [...] but they're entirely imperatively defined rather than declarative [...] The conceptual problem with Docker isn't imperative vs declarative. It's that Docker doesn't even try to be reproducible. Executing the same Dockerfile twice doesn't necessarily build you the same container. (Imperative vs declarative is still an interesting problem to think about, it's just independent of reproducibility in the abstract.)
- darthrupert 2y ago
  I'm also one of the people who have tried Nix(OS) a couple of times and found it too much of a hassle, but nightmare is exaggerating a bit, I feel. Nix's strength and weakness is that it wants to take over everything, and if you want to do something without it, you might be in a world of pain. And after doing more or less standard unixy things for 20+ years, it's difficult to hand over control to a new thing like that.
- xorcist 2y ago
  > some point on the spectrum between "commit your dotfiles to git" and Nix That would be configuration management tools such as Salt/Puppet/Ansible/Chef. They were popular ten years ago, and gained a lot of exposure as the devops movement gained ground, but they never stopped being useful. Having your non-running state defined declarative is powerful, and if you can define a single source of truth for entire distributed systems, that suddenly makes you able to reason about state on whole systems.
  - XorNot 2y ago
    What I can't currently figure out with Nix though is how I kill off dependency explosions though. I want to reach into a big tree of derivations and force library and compiler versions to a single common one so we don't, for example, have 6 rust compilers being used.
    - chriswarbo 2y ago
      The usual approach is to give Nixpkgs some `overlays`, which override the attributes you want. This can be handy in conjunction with attributes like `.override`, `.overrideAttrs`, etc. for swapping-out things deeper in the dependency graph. The https://codeberg.org/amjoseph/infuse.nix https://codeberg.org/amjoseph/infuse.nix project looks nice as a way to simplify annoying chains of overrides; though I haven't used it personally.
amarshall 2y ago
For what it’s worth, nothing in this article is really necessarily for general usage of Nix, as the derivation format is mostly abstracted-away, like how the OCI image format is irrelevant to everyday authoring of Dockerfiles.
drowsspa 2y ago
Yeah, it's like those famous posts comparing monads to burritos or something

tikhonj 2y ago

For context, this particular article is a cool deep dive into how Nix works, but it doesn't represent what using Nix + Nixpkgs is like in practice. I've been using Nix personally and professionally for almost 10 years now (yikes has time passed quickly!) and I have never needed to operate at the level of derivations like this.

setheron 2y ago
Professionally for 10 years ? I would like to hear more. I remember Target funded some earlier Nix work like lorri. Interestingly, I found operating at this level to have been a necessity if I wanted to fix or write a derivation. Otherwise I would face random errors and be left clueless as to what the problem was. Writing a derivation builder (nix pill) was illuminating.
- tikhonj 2y ago
  Yep, I was one of the people who originally started using Nix on that particular team at Target back in 2016. I wasn't the only person interested in Nix but, if I'm recalling correctly, I was the person who first set it up and supported our Nix environment on Linux and macOS for the first year or two. After we scaled up a bit we hired some more Nix-focused people and some consultants (for stuff like lorri), so I stopped having to do much Nix stuff day-to-day. That was a really special team. Nobody "let" me use Nix, we just did it because it made sense to us, and I went out of my way to support it internally not because I felt pressured to do it, but because I was enthusiastic about making it work. The culture there reminded me of this story[1] about getting Python into production at Goldman—and it turns out that the VP who built the supply chain team at Target started his career in strats at Goldman and was heavily inspired by the same Armen from that story :) [1]: https://news.ycombinator.com/item?id=29104401 https://news.ycombinator.com/item?id=29104401 After leaving Target, I also used Nix for my own work setup at CX Score. That was a seed-stage startup and I didn't want to push Nix on anybody else, but it made my own work setup much nicer. Most recently I moved to Mercury where we use Nix at a pretty big scale (several hundred developers). We have a team that handles most of it so I haven't had to do much myself, but I've helped a few people work through Nix issues just because I could :) I'm also one of the weirdos who uses NixOS on my work laptop instead of getting a macbook. And, in parallel, I've been using NixOS on all my personal machines ever since I first discovered it. That was relatively shortly before joining Target, so, if I had to guess, late 2015 or early 2016. Coming up on 10 years shortly, even though it really does not feel like it...

setheron 2y ago

What a fun read. Thanks for the nice call out on my blog. If you're also interested I (Farid) also had a follow up where I follow up on how the hashes are calculated. (Using the error to get the hashes also bugged me) I use it to also create vanity hashes :) https://fzakaria.com/2025/03/28/what-s-in-a-nix-store-path https://fzakaria.com/2025/03/28/what-s-in-a-nix-store-path https://fzakaria.com/2025/03/27/nix-vanity-store-paths https://fzakaria.com/2025/03/27/nix-vanity-store-paths

pveierland 2y ago

Nix derivations are pretty neat! I've been building a Nix store navigator for MyNixOS v2. It can help you get a feel for how derivations are connected in Nix: https://v2.mynixos.com/nix/store/lsk1c4v03y4lmpxdcwal99nm5nwz2rj7-texlive-2024-env.drv https://v2.mynixos.com/nix/store/lsk1c4v03y4lmpxdcwal99nm5nw... PS: The controls to the upper right can be used to limit what is downloaded.

Joker_vD 2y ago

Ah, don't you just like it when projects that use hashsums, calculate them in such a way that you can't actually recalculate them on your own? And when you start digging, you find not only that it uses the same basic design of XML-DSig while barely escaping its fatal flaws, it also uses some bizarre data encodings nobody else uses for anything. And then the resulting hash is not even the truncation of the actual hash, it has an additional (again, entirely undocumented) strange post-processing step for unspecified reasons.

danlitt 2y ago

c.f. guix, where the command is simply `guix hash`.

chombier 2y ago

What I found hard with Nix is the sheer amount of things I had to get familiar with before it started to really click: - nix, the command-line tool - nix, the language - nixpkgs with the general API/idioms (overriding, overlays) - individual nixpkgs packages that sometimes deviate from common practices - flakes (which I haven't properly looked into yet) The Nix pills series [1] and nixpkgs documentation [2] do help a lot, but that is quite a lot to process. [1] https://nixos.org/guides/nix-pills https://nixos.org/guides/nix-pills [2] https://nixos.org/manual/nixpkgs/stable/ https://nixos.org/manual/nixpkgs/stable/

woile 2y ago
they are a lot of things, but it's hard to avoid otherwise. Overall I'm glad it's relatively cohesive how it all works.
angra_mainyu 2y ago
Controversial, but I found just going through the docs, attempting to write my own flake, throwing it at ChatGPT and asking it to review it and give me feedback was enough to get started. I never ask it for code, just to review code - keeps me sharp and makes the iteration cycles faster.
- pxc 2y ago
  ChatGPT is generally pretty bad at writing Nix code IME, and has a much greater tendency to hallucinate than when writing code in other languages. But it's better at advice and review than writing code. And if you want to ask it about Nixpkgs conventions or the behavior of some Nix codebase, it does much better if you dump the relevant source files in the chat for it to consume. It's important with Nix to seriously understand what you're doing, so I think the way you've described is the most appropriate way to use an LLM with Nix. Use it to point you to docs, ask it questions about some code and then read the code closely, ask it which docs to read, ask it to point you to example repos on GitHub. If you use it this way it can actually be helpful. If you have a longer time horizon, interacting with real community members is even better, and of course can also enrich the corpus for LLMs like ChatGPT. Discourse.nixos.org has a lot of smart and helpful people whose insight is more reliable than the advice of an LLM, so it's worth visiting as well.

klysm 2y ago

Is there any technical pathway for changing the underlying hash function? Similar to how git is kind of stuck on SHA1, what would it take to get this onto SHA512?

chriswarbo 2y ago
I think it could be done in a backwards-compatible way, since existing hashes fit a known pattern; e.g. (silly example) store paths using SHA512 could be distinguished using prefix like 'sha512_'. One problem would be cache misses and duplication of files. I'm not completely up to date on Git's hash situation, but it seems to me like SHA256 is usable; but only for new repos. I imagine lots of existing infrastructure assumes SHA1 hashes though, and would break for silly reasons like different lengths. Relatedly, Nix has a new feature called `git-hashing` which can use Git tree hashes to validate fixed-output derivations, rather than requiring two separate hashes; e.g. the following `fetchTreeFromGitHub` function uses a single `tree` argument for a Git tree ID, rather than needing separate `rev` and `hash` arguments: { owner, repo, tree }: (pkgs.fetchFromGitHub { inherit owner repo; rev = tree; hash = builtins.convertHash { hash = tree; hashAlgo = "sha1"; toHashFormat = "sri"; }; }).overrideAttrs (_: { outputHashMode = "git"; }) Unfortunately, this git-hashing functionality only works for SHA1; so it's a good example of the infrastructure around Git not allowing us to move away from SHA1!