8 ms·

> Nix, on the other hand, does have a concept of state. If you make a one-line change to a 200-line Nix configuration, it doesn’t have to re-do all the work fro

> Nix, on the other hand, does have a concept of state. If you make a one-line change to a 200-line Nix configuration, it doesn’t have to re-do all the work from the other 199 lines. It can evaluate the state of the system against the configuration file and recognize that it just has to apply the one-line change. And that change usually happens in a few seconds. The author seems to have some misguided ideas about Nix. Nix is not fast because it is stateful. It is fast because it is functional and reproducible, which allows for caching without compromising correctness. I don't want to split hairs, but referentially transparent caching like this is not quite what I'd call state. Yes, there is some statefulness in system activation, but this is not what makes Nix Nix -- quite the opposite.

hardware2win 3y ago
Functional?
- globular-toast 3y ago
  Just in case: https://en.wikipedia.org/wiki/Functional_programming https://en.wikipedia.org/wiki/Functional_programming
  - hardware2win 3y ago
    I mean how Nix uses FP?
    - maleldil 3y ago
      Did you read the entire comment? The core idea of functional programming is pure functions (given the same outputs, the output is always the same). Nix applies that to building systems.
      - hardware2win 3y ago
        Pure functions are just a concept that is used in FP world Yet, pure functions can "stand alone". You do not need FP to use pure functions. They are independent from FP. You can be OOP maniac and still use them. So unless you want to sound fancy and trendy then why call Nix functional instead of side-effects free?
        markeibes 3y ago
        How is "side-effects free" less fancy and trendy than functional?
        kaba0 3y ago
        Because it is basically a dynamically typed Haskell? It also does currying. Like, there is no one definition of FP, but Nix is definitely an FP language as well as the whole idea behind it.
        dwohnitmok 3y ago
        > You do not need FP to use pure functions. They are independent from FP. You can be OOP maniac and still use them. This is not true. OOP is fundamentally built around impure operations. Objects are persistent references that you send messages to or that you call methods on (depending on your OOP language of choice). A persistent reference that is stable across different invocations (as opposed to a new reference being created on each invocation) requires that those invocations be impure operations, because the same invocation performed multiple times must have different effects, otherwise there is no point to having a stable reference. That's not necessarily a bad thing, but OOP is fundamentally impure. (If you do create a new reference on every invocation, then you no longer have OOP. You merely have a namespacing system where the first argument to a function can alternatively be written with a dot).
        captaincrisp 3y ago
        > You do not need FP to use pure functions I think this is a bit more subtle, and a bit less interesting to say, than you're thinking it is and responding to. You need to be impure in OOP world but not in every function you write. I write many pure functions in my OOP work; they make the whole thing easier to reason about.
microtonal 3y ago
Just to elaborate a bit for those not familiar to Nix (slightly simplified to exclude recent support for content addressing). Nix work with derivations, a derivation is basically a data structure that specifies how a package is built. Derivations are normally not created by hand but using a function (eg. stdenv.mkDerivation). When you ask Nix to build a package, it hashes a normalized form of derivation data structure. This hash is useful in various ways, but one way it is used [1] is to look up whether the derivation is already in the Nix Store. Because if it is, there is no need to build it. So Nix looks up whether /nix/<the_derivation_hash> exists. If it exists, the build is done. If it doesn't exist and you have a binary cache configured (which by default is the binary cache provided by the NixOS project), Nix will look up the derivation hash in the binary cache. If it exists in the binary cache, Nix will download the path to the local Nix store. After that /nix/<the_derivation_hash> exists in the store and the build is done (without building anything). Only if that fails, Nix will actually build the derivation. Now, one of the cool things about Nix is that it is derivations all the way down. So, it's not that just what we traditionally think of as packages is a derivation, but people wrap up all kinds of things as derivations, including configuration, etc. Since derivations are usually generated by functions, there are all kinds of useful functions that make derivations for eg.: single configuration files, scripts, etc. In the end, building a NixOS system generation is just building a derivation. nixos-rebuild switches to a different generation by just setting a bunch of symlinks to an output path in the store containing that system generation (/nix/<system_config_derivation_hash). At any rate, when you make a one-line change to a 200-line Nix configuration, Nix does have state to keep track of what it needs to rebuild or not. Nix will just try to build the derivation (and its dependencies), but it hashes the derivations, finds that their output paths are already in the store. Some might argue that then the store is state. But it's not, at build time you are evaluating a pure function with memoization (the Nix Store). [1] There is also a package name and version in the store path, but lets keep it simple.
- henrydark 3y ago
  I agree with everything but the last statement. This all comes down to: do you consider memoization to be state. I predict people's answers to this question will come from experience with memoization. Here's mine: I kept trying to get nix to build tensorflow locally, so that I would get the avx512 benefits of the big, but gpu-less machine I had. I hadn't realized some other derivation had already downloaded tensorflow from online cache, so didn't have avx512 enabled. I kept making shells, trying tensorflow, seeing it doesn't have support. The solution was to tell nix to disregard the nix store, in order to force the local build. This experience has left me with the concrete feeling that the nix store is full-on state, and I the user must be aware of it.
crubier 3y ago
Memoization is state. As a functional programming maximalist myself, I know it hurts a little bit, but still, Caching/Memoization is statefulness. That said, you are right that as a user of the system, that statefulness is abstracted from you and you don't have to worry about it (until some subtle caching bugs forces you to dive deep in the rabbit hole)
- rgoulter 3y ago
  Reminds me of this more abstract perspective: you can draw an analogy between working the filesystem, and programming languages working with memory. https://www.tweag.io/blog/2022-07-14-taming-unix-with-nix/ https://www.tweag.io/blog/2022-07-14-taming-unix-with-nix/
- sanderjd 3y ago
  Yeah I really bumped on this. A cache is absolutely state. It is sufficient to say "a referentially transparent cache is a technique to store state while avoiding most of the drawbacks typical to storing state". This is already a big accomplishment! No need to also redefine the word "state" to make the technique seem even more magical than it is.
  - pxc 3y ago
    It's state at a different level than most configuration management software deals with. When Puppet reasons about the state of an existing system, it asks things like 'what's on the PATH right now?'. NixOS doesn't— it reconstructs the directory tree under /run/current-system/sw/bin without asking that. Same with 'modification' of files that live under /etc. What you and the GP write here is nuance worth admitting, but I wanted to highlight what people are getting at when they say Nix's approach is 'stateless'. What they mean is that Nix's approach to state is really qualitatively different, and hands state off to be managed by a lower level part of the system as much as possible.
adamgordonbell 3y ago
Explaining what Nix is seems like one of its biggest barriers to adoption. Part of the difficulty is it means different things to different people. My colleague spent a whole lot of time trying to answer this question and ended up with this: My main take away after spending some time learning about Nix is that it embraces the functional programming concept of a pure function. If I give a function a certain set of inputs, it will return the same result every time, no matter what. Nix is about building software the same way, whether it’s your own software, someone else’s software, or your entire OS: You declare all your inputs explicitly and it will be built the same way every time. What really mattered to them, regardless of how they were using Nix, was its ability to bring purely functional programming concepts to computing areas that were previously off-limits. From that single idea you get a whole ecosystem of tools. We mainly covered the Nix language, the Nix Package Manager, and NixOS, but there’s also a continuous build system called Hydra, nix shell, and a deployment and provisioning tool called NixOps. Probably, there’s even more. https://earthly.dev/blog/what-is-nix/ https://earthly.dev/blog/what-is-nix/ Another way someone else described Nix to me was "Gentoo for Haskell devs".
- teekert 3y ago
  I often find difficult to explain ideas the most interesting ones. It’s a new paradigm, naturally we need to develop “the neurons” to deal with it, to reason with it, to hold the paradigm in our minds. I find the best way to get the paradigm is to dive in. I see from people that did that they are always enthusiastic, it must be worth it.
- thumbuddy 3y ago
  The biggest barrier to adoption unfortunately is not people's inability to explain what the tool is in my opinion. It's that the tool is incredibly complicated, extremely hard to walk someone through compared to alternative projects, and honestly... In my opinion the problem it attempts to solve doesn't really exist. I'll take an "impure" os or package manager over a pure one any day if complexity is a thousand fold less and the learning curve doesn't require half a decade. Got stuff to do!
  - 3y ago
mark38848 3y ago
Nix is only fast if your time has no value :D
- SuperSandro2000 3y ago
  It is not incredible fast and while it is running you can do something else like reading hackernews but there is definitely room for improvement and smart people are already thinking about that right now.
deleted 3y ago
[deleted]
mtlynch 3y ago
Author here. Thanks for the clarification! I'm still new to Nix, so I'm trying to share useful things I'm learning without overstepping my expertise and saying wrong things. My mental model of Nix was that if I'm in system state A, which is the result of performing task X + Y, and I want to get to system state B, which is the result of task X + Y + Z, then Nix would recognize that it's already in state A, so it only has to perform task Z to get to state B. It sounds like what you're saying is that Nix's actual behavior is that if I'm in state A and tell Nix to bring me to state B, then Nix still performs tasks X and Y again, but the results are cached, so tasks X and Y feel as fast as a no-op, and the only task I perceive as taking time is task Z. Is that right?
- Filligree 3y ago
  When it comes to building derivations, yes. Which describes almost everything you can do. For live system rebuilds in NixOS, the final step involves examining the system to decide which systemd services need restarting and restarting only those; that’s part of the process called activation. But that’s the only exception, and doesn’t happen if you reboot instead.
- hamandcheese 3y ago
  That's largely correct! To use more precise language, let's discuss two distinct phases: building and activation. Nix will always "build" x, y, and z. But x and y might turn into a no-op by being cached in the nix store. System activation is when all those things are symlinked into place. If x, y, and z are, say, systemd services, then there might be some logic that checks if x and y have changed and if the services are already running and decide what to do -- this is probably the most similar to how Ansible works. But this is also a small part of the big picture. Activation is pretty fast even starting from nothing.
  - mtlynch 3y ago
    Ah, okay. That makes sense. Thanks! I've updated the post based on your feedback.
- pxc 3y ago
jerf 3y ago
Quibbling about what is and is not state isn't particularly productive. There is intrinsic and unavoidable subjectivity in the definition of state. The common pattern is to ignore out-of-memory errors, completely ignore the fact that in the real world time-of-execution is irreducibly an observable side effect, and that whether or not a given computation completes may even depend on whether or not an entirely separate OS process has or has not consumed the memory on the system. There's already non-zero pragmatism in the definition of "statefulness", and adding "whether or not the state of the disk has changed as a result of this 'pure computation'" isn't exactly an impossible stretch. It isn't a binary characteristic; when you really get into the nitty-gritty it is unavoidably a continuum. All real systems have state of some kind.
- hamandcheese 3y ago
  I'm not quibbling about what state is - the author is very much confusing Nix with something like a control-loop based approach that converges on the desired state by examining the current state. Regardless of your definition of state, that's not what Nix is.

hardware2win 3y ago

Functional?

globular-toast 3y ago
Just in case: https://en.wikipedia.org/wiki/Functional_programming https://en.wikipedia.org/wiki/Functional_programming
- hardware2win 3y ago
  I mean how Nix uses FP?
  - maleldil 3y ago
    Did you read the entire comment? The core idea of functional programming is pure functions (given the same outputs, the output is always the same). Nix applies that to building systems.
    - hardware2win 3y ago
      Pure functions are just a concept that is used in FP world Yet, pure functions can "stand alone". You do not need FP to use pure functions. They are independent from FP. You can be OOP maniac and still use them. So unless you want to sound fancy and trendy then why call Nix functional instead of side-effects free?
      - markeibes 3y ago
        How is "side-effects free" less fancy and trendy than functional?
      - kaba0 3y ago
        Because it is basically a dynamically typed Haskell? It also does currying. Like, there is no one definition of FP, but Nix is definitely an FP language as well as the whole idea behind it.
      - dwohnitmok 3y ago
        > You do not need FP to use pure functions. They are independent from FP. You can be OOP maniac and still use them. This is not true. OOP is fundamentally built around impure operations. Objects are persistent references that you send messages to or that you call methods on (depending on your OOP language of choice). A persistent reference that is stable across different invocations (as opposed to a new reference being created on each invocation) requires that those invocations be impure operations, because the same invocation performed multiple times must have different effects, otherwise there is no point to having a stable reference. That's not necessarily a bad thing, but OOP is fundamentally impure. (If you do create a new reference on every invocation, then you no longer have OOP. You merely have a namespacing system where the first argument to a function can alternatively be written with a dot).
        captaincrisp 3y ago
        > You do not need FP to use pure functions I think this is a bit more subtle, and a bit less interesting to say, than you're thinking it is and responding to. You need to be impure in OOP world but not in every function you write. I write many pure functions in my OOP work; they make the whole thing easier to reason about.

microtonal 3y ago

Just to elaborate a bit for those not familiar to Nix (slightly simplified to exclude recent support for content addressing). Nix work with derivations, a derivation is basically a data structure that specifies how a package is built. Derivations are normally not created by hand but using a function (eg. stdenv.mkDerivation). When you ask Nix to build a package, it hashes a normalized form of derivation data structure. This hash is useful in various ways, but one way it is used [1] is to look up whether the derivation is already in the Nix Store. Because if it is, there is no need to build it. So Nix looks up whether /nix/<the_derivation_hash> exists. If it exists, the build is done. If it doesn't exist and you have a binary cache configured (which by default is the binary cache provided by the NixOS project), Nix will look up the derivation hash in the binary cache. If it exists in the binary cache, Nix will download the path to the local Nix store. After that /nix/<the_derivation_hash> exists in the store and the build is done (without building anything). Only if that fails, Nix will actually build the derivation. Now, one of the cool things about Nix is that it is derivations all the way down. So, it's not that just what we traditionally think of as packages is a derivation, but people wrap up all kinds of things as derivations, including configuration, etc. Since derivations are usually generated by functions, there are all kinds of useful functions that make derivations for eg.: single configuration files, scripts, etc. In the end, building a NixOS system generation is just building a derivation. nixos-rebuild switches to a different generation by just setting a bunch of symlinks to an output path in the store containing that system generation (/nix/<system_config_derivation_hash). At any rate, when you make a one-line change to a 200-line Nix configuration, Nix does have state to keep track of what it needs to rebuild or not. Nix will just try to build the derivation (and its dependencies), but it hashes the derivations, finds that their output paths are already in the store. Some might argue that then the store is state. But it's not, at build time you are evaluating a pure function with memoization (the Nix Store). [1] There is also a package name and version in the store path, but lets keep it simple.

henrydark 3y ago
I agree with everything but the last statement. This all comes down to: do you consider memoization to be state. I predict people's answers to this question will come from experience with memoization. Here's mine: I kept trying to get nix to build tensorflow locally, so that I would get the avx512 benefits of the big, but gpu-less machine I had. I hadn't realized some other derivation had already downloaded tensorflow from online cache, so didn't have avx512 enabled. I kept making shells, trying tensorflow, seeing it doesn't have support. The solution was to tell nix to disregard the nix store, in order to force the local build. This experience has left me with the concrete feeling that the nix store is full-on state, and I the user must be aware of it.

crubier 3y ago

Memoization is state. As a functional programming maximalist myself, I know it hurts a little bit, but still, Caching/Memoization is statefulness. That said, you are right that as a user of the system, that statefulness is abstracted from you and you don't have to worry about it (until some subtle caching bugs forces you to dive deep in the rabbit hole)

rgoulter 3y ago
Reminds me of this more abstract perspective: you can draw an analogy between working the filesystem, and programming languages working with memory. https://www.tweag.io/blog/2022-07-14-taming-unix-with-nix/ https://www.tweag.io/blog/2022-07-14-taming-unix-with-nix/
sanderjd 3y ago
Yeah I really bumped on this. A cache is absolutely state. It is sufficient to say "a referentially transparent cache is a technique to store state while avoiding most of the drawbacks typical to storing state". This is already a big accomplishment! No need to also redefine the word "state" to make the technique seem even more magical than it is.
- pxc 3y ago
  It's state at a different level than most configuration management software deals with. When Puppet reasons about the state of an existing system, it asks things like 'what's on the PATH right now?'. NixOS doesn't— it reconstructs the directory tree under /run/current-system/sw/bin without asking that. Same with 'modification' of files that live under /etc. What you and the GP write here is nuance worth admitting, but I wanted to highlight what people are getting at when they say Nix's approach is 'stateless'. What they mean is that Nix's approach to state is really qualitatively different, and hands state off to be managed by a lower level part of the system as much as possible.

adamgordonbell 3y ago

Explaining what Nix is seems like one of its biggest barriers to adoption. Part of the difficulty is it means different things to different people. My colleague spent a whole lot of time trying to answer this question and ended up with this: My main take away after spending some time learning about Nix is that it embraces the functional programming concept of a pure function. If I give a function a certain set of inputs, it will return the same result every time, no matter what. Nix is about building software the same way, whether it’s your own software, someone else’s software, or your entire OS: You declare all your inputs explicitly and it will be built the same way every time. What really mattered to them, regardless of how they were using Nix, was its ability to bring purely functional programming concepts to computing areas that were previously off-limits. From that single idea you get a whole ecosystem of tools. We mainly covered the Nix language, the Nix Package Manager, and NixOS, but there’s also a continuous build system called Hydra, nix shell, and a deployment and provisioning tool called NixOps. Probably, there’s even more. https://earthly.dev/blog/what-is-nix/ https://earthly.dev/blog/what-is-nix/ Another way someone else described Nix to me was "Gentoo for Haskell devs".

teekert 3y ago
I often find difficult to explain ideas the most interesting ones. It’s a new paradigm, naturally we need to develop “the neurons” to deal with it, to reason with it, to hold the paradigm in our minds. I find the best way to get the paradigm is to dive in. I see from people that did that they are always enthusiastic, it must be worth it.
thumbuddy 3y ago
The biggest barrier to adoption unfortunately is not people's inability to explain what the tool is in my opinion. It's that the tool is incredibly complicated, extremely hard to walk someone through compared to alternative projects, and honestly... In my opinion the problem it attempts to solve doesn't really exist. I'll take an "impure" os or package manager over a pure one any day if complexity is a thousand fold less and the learning curve doesn't require half a decade. Got stuff to do!
- 3y ago

mark38848 3y ago

Nix is only fast if your time has no value :D

SuperSandro2000 3y ago
It is not incredible fast and while it is running you can do something else like reading hackernews but there is definitely room for improvement and smart people are already thinking about that right now.

deleted 3y ago

[deleted]

mtlynch 3y ago

Author here. Thanks for the clarification! I'm still new to Nix, so I'm trying to share useful things I'm learning without overstepping my expertise and saying wrong things. My mental model of Nix was that if I'm in system state A, which is the result of performing task X + Y, and I want to get to system state B, which is the result of task X + Y + Z, then Nix would recognize that it's already in state A, so it only has to perform task Z to get to state B. It sounds like what you're saying is that Nix's actual behavior is that if I'm in state A and tell Nix to bring me to state B, then Nix still performs tasks X and Y again, but the results are cached, so tasks X and Y feel as fast as a no-op, and the only task I perceive as taking time is task Z. Is that right?

Filligree 3y ago
When it comes to building derivations, yes. Which describes almost everything you can do. For live system rebuilds in NixOS, the final step involves examining the system to decide which systemd services need restarting and restarting only those; that’s part of the process called activation. But that’s the only exception, and doesn’t happen if you reboot instead.
hamandcheese 3y ago
That's largely correct! To use more precise language, let's discuss two distinct phases: building and activation. Nix will always "build" x, y, and z. But x and y might turn into a no-op by being cached in the nix store. System activation is when all those things are symlinked into place. If x, y, and z are, say, systemd services, then there might be some logic that checks if x and y have changed and if the services are already running and decide what to do -- this is probably the most similar to how Ansible works. But this is also a small part of the big picture. Activation is pretty fast even starting from nothing.
- mtlynch 3y ago
  Ah, okay. That makes sense. Thanks! I've updated the post based on your feedback.
pxc 3y ago

jerf 3y ago

Quibbling about what is and is not state isn't particularly productive. There is intrinsic and unavoidable subjectivity in the definition of state. The common pattern is to ignore out-of-memory errors, completely ignore the fact that in the real world time-of-execution is irreducibly an observable side effect, and that whether or not a given computation completes may even depend on whether or not an entirely separate OS process has or has not consumed the memory on the system. There's already non-zero pragmatism in the definition of "statefulness", and adding "whether or not the state of the disk has changed as a result of this 'pure computation'" isn't exactly an impossible stretch. It isn't a binary characteristic; when you really get into the nitty-gritty it is unavoidably a continuum. All real systems have state of some kind.

hamandcheese 3y ago
I'm not quibbling about what state is - the author is very much confusing Nix with something like a control-loop based approach that converges on the desired state by examining the current state. Regardless of your definition of state, that's not what Nix is.