16 ms·

Flakes – Proposed mechanism to package Nix expressions into composable entities

164 points by based2 7y ago

deleted 7y ago
[deleted]
codedokode 7y ago
While Nix's concepts are awesome, the implementation doesn't look so nice. Their "language" for writing Nix files [1] doesn't look attractive. Doesn't make me want to write a package description. Of course, it is better than languages with lots of brackets, but still not good enough. [1] https://github.com/edolstra/nixpkgs/blob/release-19.03/flake.nix https://github.com/edolstra/nixpkgs/blob/release-19.03/flake...
- siscia 7y ago
  But since when a language need to looks attractive? First of all, "attractive" is simply your way to say "familiar to me". Then, a language need to be just powerful enough to express it's semantic and no more. This critique is quite hollow and noisy.
  - juskrey 7y ago
    Even after writing substantial number of nix scripts and taking some weeks of break (because my job is not writing nix scripts 24/7), I struggle to remember all of that structure inheritance magic and have to reread the docs. Not a good sign, usually happens with me with unnecessarily overengineered things.
    - eridius 7y ago
      What do you mean by "structure inheritance magic"? I can't think of anything I'd term "magic" in the Nix language. Nixpkgs itself does a lot of funky things, especially around overrides and whatnot, but those are features of Nixpkgs, not of Nix the language.
    - ris 7y ago
      > Not a good sign, usually happens with me with unnecessarily overengineered things. Haha yes, I mean clearly this means it's Nix's fault. Has anyone ever considered using you as some kind of "overengineering divining rod" where they can just wave something in front of your nose and go back to the drawing board if it doesn't pass?
- jchw 7y ago
  Sounds like you wouldn’t like GuixSD either. Well you might be right, but I find Nix and NixOS to be great. Honestly the language isn’t the issue, it’s just that it isn’t nearly as well supported as more popular distros like Ubuntu, so for example my VMWare Workstation license has been sitting dormant for a while now... (Which is mostly OK. I’ve been using libvirt for everything and pleasantly surprised at what it can do.) The language takes some getting used to, and I am not really 100% sold on pure functional, but it works and my systems are much more reliable now.
- hollerith 7y ago
  At least that language does not require quotes around almost every sequence of letters like JSON does. (Yes, I know, JSON is not a language, but people do sometimes use it for things approximately as long and as complicated as a Nix expression.)
  - k__ 7y ago
    cough YAML cough
    - marcosdumay 7y ago
      Oh, man, let's not go there. It's a silly place.
      - k__ 7y ago
        Somehow, right? But why isn't there an ML with the good parts of YAML, but without the bad parts?
        clhodapp 7y ago
        YAML (and TOML and even XML when it's being used in a tag-centric way instead of a text-centric way) are not markup languages. A markup language is one that allows you to assign meaningful annotations to regions within a chunk of textual content (e.g. "these three words are underlined", "this relates to footnote 3", etc.). YAML and TOML have both changed the meanings of their acronyms to try and distance themselves from their original misnamings as "markup languages".
        tannhaeuser 7y ago
        Yep. Maven (ugh) even has switched off entity expansion, a basic feature of XML/SGML, in favour of their own ad-hoc ${property} syntax.
        marcosdumay 7y ago
        Anyway, the GP's question is a good one. Where are the configuration/serialization languages that import the only good parts of YAML? AFAIK, one can not subset YAML and get something coherent. But one can start from scratch and use its concepts. I imagine the languages are not there just because nobody got the time to do them.
- vertex-four 7y ago
  For basic package descriptions, the language is not terribly different from JSON. Here's a definition of a Python application I'm working on: { pkgs ? import <nixpkgs> {}}: with pkgs; python3Packages.buildPythonApplication rec { name = "quicktill"; src = ./.; doCheck = false; propagatedBuildInputs = with python3Packages; [ dateutil pyyaml sqlalchemy psycopg2 pycups reportlab requests_oauthlib pygobject3 ]; nativeBuildInputs = [ wrapGAppsHook ]; buildInputs = [ gobjectIntrospection gtk3 ]; } The only big things here are the fact that this is defined as a function that takes the package repository as an argument to access dependencies (the first line), and the "with" expressions that allow me to skip having to type "pkgs." and "pkgs.python3Packages." for every package I depend on.
  - jforberg 7y ago
    That looks completely inscrutable to me. Like a sequence of magic words. I don't see the connection with JSON at all.
    - vertex-four 7y ago
      I'm not really sure what you're finding inscrutable. I'll tear out the boilerplate that explains that this file is a function and returns a Python application (which is documented in the nixpkgs manual and can just be copy-pasted if you really don't want to figure out what it does): rec { name = "quicktill"; src = ./.; doCheck = false; propagatedBuildInputs = with python3Packages; [ dateutil pyyaml sqlalchemy psycopg2 pycups reportlab requests_oauthlib pygobject3 ]; nativeBuildInputs = [ wrapGAppsHook ]; buildInputs = [ gobjectIntrospection gtk3 ]; } Is that any better? It looks an awful lot like some dialect of something JSON-like to me - there's some different symbols used and array values are separated with spaces rather than commas, but aside from that it's just a record of key-value pairs. From there, the first line in the original example says "the following expression is a function that is called with this parameter", and the parameter is an object containing a key called "pkgs", which is defaulted to the result of "import <nixpkgs> {}". This is explained in more detail in a handful of lines in the language manual. Functions are called just by referencing the function and then the arguments separated with spaces, like Ruby or Haskell - so python3Packages.buildPythonApplication is a function that is called with the object above. Finally, "with foo;" just means "every key in foo can be accessed without typing foo. in the next expression". The result is basically: this file defines a function which takes an object containing the package repository as the "pkgs" key, and returns the result of calling pkgs.python3Packages.buildPythonApplication with the object defined in the rest of the file. It's a lot more derived from the syntax of functional languages than C-like ones, but that's not necessarily a bad thing. Also: this is basically the entire language. There's a couple of different string syntaxes and some stuff for merging objects together, and also a mechanism for temporarily naming things that aren't part of an object, but in general... there's not a lot more to it. Most of the complexity is in the domain itself - what's the difference between a propagated, native and regular build input, how do you manage plugins and options for a package, etc. What's your preferred packaging DSL?
- rufugee 7y ago
  I ran Nix for about six months. The language was one of the factors that really bugged me at the time. That and the hoops you had to jump through to do basic things like installing npm packages...
- vmchale 7y ago
  The nix language is a bit hinky; unfortunately nothing available even approaches it in terms of capabilities.
- patrec 7y ago
  There are various problems with nix and the pragmatics of the language (error reporting combined with lack of static type checking, inconsistent naming conventions, evaluation speed, tooling, etc.) – but the core language itself is, IMO, actually extremely well designed. I can think of very few good external DSLs (postscript is the only one that comes to mind right now) – but I'd definitely place nix among this list.
tbenst 7y ago
I’d love for a way to write nix expressions that easily lead to reproducible builds. Of course this is possible right now next, but requires substantial legwork to pin Nixpkgs version or add SHA for every tarball. Especially now as the machine learning story for NixOS is becoming strong, it would be awesome to have fully reproducible model training using such a system. This highlights a difference from docker or nvidia-docker, which do not control driver version.
- colemickens 7y ago
  Fortunately, `builtins.fetchTarball` makes this easier. I do this in my `nixcfg` repo so that I can build my entire machine config (patches and enabling Iris in Mesa, and all) on a stupidly-cheap Packet.net VM: https://github.com/colemickens/nixcfg/blob/master/default.nix https://github.com/colemickens/nixcfg/blob/master/default.ni.... If you follow the rabbit-hole: `default.nix` -> `lib.nix`, etc, you can see that I pin nixpkgs, have an update script that updates the nixpkgs revs I build against, it supports building against a custom, local `~/code/nixpkgs` if it exists, and I have my machine config abstracted out to where I can build a "GNOME instance of my machine" or by default, my machine with Sway and sway related packages installed. Much of this should be easier with flakes, as I understand it. It's not the cleanest config, the README needs love, but maybe it can be inspiration in the meantime. :) My latest trick was figuring out how to get the new Mesa Intel Iris Gallium driver enabled without rebuilding the world, and I extracted it to what I call a "mixin" that anyone can copy and just use: https://github.com/colemickens/nixcfg/blob/master/modules/mixin-intel-iris.nix https://github.com/colemickens/nixcfg/blob/master/modules/mi... (And technically I still use mozilla/nixpkgs-mozilla to pull the latest Firefox Nightly which is impure and thus not always perfectly, perfectly reproducable. I do however pin the overlays themselves also - like my nixpkgs-wayland overlay that packages HEAD versions of Sway and other Wayland related tools!)
  - tbenst 7y ago
    This is interesting. Sounds like you do this on the system level? I’m interested in doing this on a shell or package level eg I’d love to have system on bleeding edge but be able to jump back to same binary blobs associated with a particular revision of software
kreetx 7y ago
It will replace nix channels? Looks like a big change!
kalium-xyz 7y ago
Looks good, but makes me worry that target sponsored this. Are RFCs normally cooperate sponsored?
- tbenst 7y ago
  I had the opposite reaction--it's great for NixOS to be gaining corporate support!
- lilyball 7y ago
  Target is also behind lorri: https://github.com/target/lorri/ https://github.com/target/lorri/ I view corporate interest as a good sign for the continued health of the ecosystem. As long as they don't "take over" the project there's no harm. And in particular, publishing an RFC like this shows that they're doing things the right way and cooperating with the community.
cwp 7y ago
This is really interesting. Right now you can't really compose nix expressions from several different sources. There are ways of doing it, but they all involve one hack or another that limits the usefulness of the result. This is why nixpkgs is a huge monolithic repository. Flakes would allow package definitions from different sources play well with each other. This means a bunch of things: 1) Nixpkgs could be broken up into several package sets for better scalability. Eg, haskelPackages and pythonPackages could be broken out into separate repositories. 2) Package authors could provide package definitions along with the source. 3) Companies could have a flake for internal packages without having to patch or overlay nixpkgs. 4) Alternatives to nixpkgs can get started without having to supply all the packages that a user might need. 5) You could even create libraries of nix functions that supply no packages at all, but get used by other flakes to define packages. For example, the machinery for composing a python environment.
- ris 7y ago
  > 1) Nixpkgs could be broken up into several package sets for better scalability. Eg, haskelPackages and pythonPackages could be broken out into separate repositories. While this could happen, it's exactly what I hope won't happen, because in my opinion the idea that haskell or python packages are fundamentally different and can live in their own world is a fallacy. I spend quite a bit of my time fixing python packages on nixpkgs master and the number of times the fix actually involves making a tweak to a non-python library means I would forever have to be playing games synchronizing repositories. Also, gone would be the ability to make atomic changes across these boundaries. Viva la monorepo.
  - Apaec 7y ago
    > Viva la monorepo. Absolutely. Look at edolstra take on this though https://github.com/NixOS/rfcs/pull/49#discussion_r305450668 https://github.com/NixOS/rfcs/pull/49#discussion_r305450668 This is why I'm scared of this change.
    - spindle 7y ago
      Me too. I think (hope) that the NixOS community won't end up doing all the work it would take to break up the existing monorepo; instead they'll just add a few small additional repos.
Apaec 7y ago
This proposal makes me scared for my bet on going full with NixOS. To me it seems like a really complicated technical solution to a social problem(the monorepo approach doesn't scale). If more people are needed to maintain the nixpkgs repo why instead not focus on ways to increase the community size. Missing docs have been a problem for NixOS for a long time, most awesome features are lost on some proficient users tweets. Having better docs would do wonders for the community. I think if NixOS wants to copy some of Rust strengths, quality docs should be its first priority.
- ris 7y ago
  > the monorepo approach doesn't scale Citation needed.
  - Apaec 7y ago
    That's mentioned by edolstra in a comment on the proposal. https://github.com/NixOS/rfcs/pull/49#discussion_r305450668 https://github.com/NixOS/rfcs/pull/49#discussion_r305450668
    - gmfawcett 7y ago
      I know who the author of that comment is, but it is not a universally held opinion among Nix users.

164 points by based2 7y ago

deleted 7y ago

[deleted]

While Nix's concepts are awesome, the implementation doesn't look so nice. Their "language" for writing Nix files [1] doesn't look attractive. Doesn't make me want to write a package description. Of course, it is better than languages with lots of brackets, but still not good enough. [1] https://github.com/edolstra/nixpkgs/blob/release-19.03/flake.nix https://github.com/edolstra/nixpkgs/blob/release-19.03/flake...

siscia 7y ago
But since when a language need to looks attractive? First of all, "attractive" is simply your way to say "familiar to me". Then, a language need to be just powerful enough to express it's semantic and no more. This critique is quite hollow and noisy.
- juskrey 7y ago
  Even after writing substantial number of nix scripts and taking some weeks of break (because my job is not writing nix scripts 24/7), I struggle to remember all of that structure inheritance magic and have to reread the docs. Not a good sign, usually happens with me with unnecessarily overengineered things.
  - eridius 7y ago
    What do you mean by "structure inheritance magic"? I can't think of anything I'd term "magic" in the Nix language. Nixpkgs itself does a lot of funky things, especially around overrides and whatnot, but those are features of Nixpkgs, not of Nix the language.
  - ris 7y ago
    > Not a good sign, usually happens with me with unnecessarily overengineered things. Haha yes, I mean clearly this means it's Nix's fault. Has anyone ever considered using you as some kind of "overengineering divining rod" where they can just wave something in front of your nose and go back to the drawing board if it doesn't pass?
jchw 7y ago
Sounds like you wouldn’t like GuixSD either. Well you might be right, but I find Nix and NixOS to be great. Honestly the language isn’t the issue, it’s just that it isn’t nearly as well supported as more popular distros like Ubuntu, so for example my VMWare Workstation license has been sitting dormant for a while now... (Which is mostly OK. I’ve been using libvirt for everything and pleasantly surprised at what it can do.) The language takes some getting used to, and I am not really 100% sold on pure functional, but it works and my systems are much more reliable now.
hollerith 7y ago
At least that language does not require quotes around almost every sequence of letters like JSON does. (Yes, I know, JSON is not a language, but people do sometimes use it for things approximately as long and as complicated as a Nix expression.)
- k__ 7y ago
  cough YAML cough
  - marcosdumay 7y ago
    Oh, man, let's not go there. It's a silly place.
    - k__ 7y ago
      Somehow, right? But why isn't there an ML with the good parts of YAML, but without the bad parts?
      - clhodapp 7y ago
        YAML (and TOML and even XML when it's being used in a tag-centric way instead of a text-centric way) are not markup languages. A markup language is one that allows you to assign meaningful annotations to regions within a chunk of textual content (e.g. "these three words are underlined", "this relates to footnote 3", etc.). YAML and TOML have both changed the meanings of their acronyms to try and distance themselves from their original misnamings as "markup languages".
        tannhaeuser 7y ago
        Yep. Maven (ugh) even has switched off entity expansion, a basic feature of XML/SGML, in favour of their own ad-hoc ${property} syntax.
        marcosdumay 7y ago
        Anyway, the GP's question is a good one. Where are the configuration/serialization languages that import the only good parts of YAML? AFAIK, one can not subset YAML and get something coherent. But one can start from scratch and use its concepts. I imagine the languages are not there just because nobody got the time to do them.
vertex-four 7y ago
For basic package descriptions, the language is not terribly different from JSON. Here's a definition of a Python application I'm working on: { pkgs ? import <nixpkgs> {}}: with pkgs; python3Packages.buildPythonApplication rec { name = "quicktill"; src = ./.; doCheck = false; propagatedBuildInputs = with python3Packages; [ dateutil pyyaml sqlalchemy psycopg2 pycups reportlab requests_oauthlib pygobject3 ]; nativeBuildInputs = [ wrapGAppsHook ]; buildInputs = [ gobjectIntrospection gtk3 ]; } The only big things here are the fact that this is defined as a function that takes the package repository as an argument to access dependencies (the first line), and the "with" expressions that allow me to skip having to type "pkgs." and "pkgs.python3Packages." for every package I depend on.
- jforberg 7y ago
  That looks completely inscrutable to me. Like a sequence of magic words. I don't see the connection with JSON at all.
  - vertex-four 7y ago
    I'm not really sure what you're finding inscrutable. I'll tear out the boilerplate that explains that this file is a function and returns a Python application (which is documented in the nixpkgs manual and can just be copy-pasted if you really don't want to figure out what it does): rec { name = "quicktill"; src = ./.; doCheck = false; propagatedBuildInputs = with python3Packages; [ dateutil pyyaml sqlalchemy psycopg2 pycups reportlab requests_oauthlib pygobject3 ]; nativeBuildInputs = [ wrapGAppsHook ]; buildInputs = [ gobjectIntrospection gtk3 ]; } Is that any better? It looks an awful lot like some dialect of something JSON-like to me - there's some different symbols used and array values are separated with spaces rather than commas, but aside from that it's just a record of key-value pairs. From there, the first line in the original example says "the following expression is a function that is called with this parameter", and the parameter is an object containing a key called "pkgs", which is defaulted to the result of "import <nixpkgs> {}". This is explained in more detail in a handful of lines in the language manual. Functions are called just by referencing the function and then the arguments separated with spaces, like Ruby or Haskell - so python3Packages.buildPythonApplication is a function that is called with the object above. Finally, "with foo;" just means "every key in foo can be accessed without typing foo. in the next expression". The result is basically: this file defines a function which takes an object containing the package repository as the "pkgs" key, and returns the result of calling pkgs.python3Packages.buildPythonApplication with the object defined in the rest of the file. It's a lot more derived from the syntax of functional languages than C-like ones, but that's not necessarily a bad thing. Also: this is basically the entire language. There's a couple of different string syntaxes and some stuff for merging objects together, and also a mechanism for temporarily naming things that aren't part of an object, but in general... there's not a lot more to it. Most of the complexity is in the domain itself - what's the difference between a propagated, native and regular build input, how do you manage plugins and options for a package, etc. What's your preferred packaging DSL?
rufugee 7y ago
I ran Nix for about six months. The language was one of the factors that really bugged me at the time. That and the hoops you had to jump through to do basic things like installing npm packages...
vmchale 7y ago
The nix language is a bit hinky; unfortunately nothing available even approaches it in terms of capabilities.
patrec 7y ago
There are various problems with nix and the pragmatics of the language (error reporting combined with lack of static type checking, inconsistent naming conventions, evaluation speed, tooling, etc.) – but the core language itself is, IMO, actually extremely well designed. I can think of very few good external DSLs (postscript is the only one that comes to mind right now) – but I'd definitely place nix among this list.

tbenst 7y ago

I’d love for a way to write nix expressions that easily lead to reproducible builds. Of course this is possible right now next, but requires substantial legwork to pin Nixpkgs version or add SHA for every tarball. Especially now as the machine learning story for NixOS is becoming strong, it would be awesome to have fully reproducible model training using such a system. This highlights a difference from docker or nvidia-docker, which do not control driver version.

colemickens 7y ago
Fortunately, `builtins.fetchTarball` makes this easier. I do this in my `nixcfg` repo so that I can build my entire machine config (patches and enabling Iris in Mesa, and all) on a stupidly-cheap Packet.net VM: https://github.com/colemickens/nixcfg/blob/master/default.nix https://github.com/colemickens/nixcfg/blob/master/default.ni.... If you follow the rabbit-hole: `default.nix` -> `lib.nix`, etc, you can see that I pin nixpkgs, have an update script that updates the nixpkgs revs I build against, it supports building against a custom, local `~/code/nixpkgs` if it exists, and I have my machine config abstracted out to where I can build a "GNOME instance of my machine" or by default, my machine with Sway and sway related packages installed. Much of this should be easier with flakes, as I understand it. It's not the cleanest config, the README needs love, but maybe it can be inspiration in the meantime. :) My latest trick was figuring out how to get the new Mesa Intel Iris Gallium driver enabled without rebuilding the world, and I extracted it to what I call a "mixin" that anyone can copy and just use: https://github.com/colemickens/nixcfg/blob/master/modules/mixin-intel-iris.nix https://github.com/colemickens/nixcfg/blob/master/modules/mi... (And technically I still use mozilla/nixpkgs-mozilla to pull the latest Firefox Nightly which is impure and thus not always perfectly, perfectly reproducable. I do however pin the overlays themselves also - like my nixpkgs-wayland overlay that packages HEAD versions of Sway and other Wayland related tools!)
- tbenst 7y ago
  This is interesting. Sounds like you do this on the system level? I’m interested in doing this on a shell or package level eg I’d love to have system on bleeding edge but be able to jump back to same binary blobs associated with a particular revision of software

kreetx 7y ago

It will replace nix channels? Looks like a big change!

kalium-xyz 7y ago

Looks good, but makes me worry that target sponsored this. Are RFCs normally cooperate sponsored?

tbenst 7y ago
I had the opposite reaction--it's great for NixOS to be gaining corporate support!
lilyball 7y ago
Target is also behind lorri: https://github.com/target/lorri/ https://github.com/target/lorri/ I view corporate interest as a good sign for the continued health of the ecosystem. As long as they don't "take over" the project there's no harm. And in particular, publishing an RFC like this shows that they're doing things the right way and cooperating with the community.

cwp 7y ago

This is really interesting. Right now you can't really compose nix expressions from several different sources. There are ways of doing it, but they all involve one hack or another that limits the usefulness of the result. This is why nixpkgs is a huge monolithic repository. Flakes would allow package definitions from different sources play well with each other. This means a bunch of things: 1) Nixpkgs could be broken up into several package sets for better scalability. Eg, haskelPackages and pythonPackages could be broken out into separate repositories. 2) Package authors could provide package definitions along with the source. 3) Companies could have a flake for internal packages without having to patch or overlay nixpkgs. 4) Alternatives to nixpkgs can get started without having to supply all the packages that a user might need. 5) You could even create libraries of nix functions that supply no packages at all, but get used by other flakes to define packages. For example, the machinery for composing a python environment.

ris 7y ago
> 1) Nixpkgs could be broken up into several package sets for better scalability. Eg, haskelPackages and pythonPackages could be broken out into separate repositories. While this could happen, it's exactly what I hope won't happen, because in my opinion the idea that haskell or python packages are fundamentally different and can live in their own world is a fallacy. I spend quite a bit of my time fixing python packages on nixpkgs master and the number of times the fix actually involves making a tweak to a non-python library means I would forever have to be playing games synchronizing repositories. Also, gone would be the ability to make atomic changes across these boundaries. Viva la monorepo.
- Apaec 7y ago
  > Viva la monorepo. Absolutely. Look at edolstra take on this though https://github.com/NixOS/rfcs/pull/49#discussion_r305450668 https://github.com/NixOS/rfcs/pull/49#discussion_r305450668 This is why I'm scared of this change.
  - spindle 7y ago
    Me too. I think (hope) that the NixOS community won't end up doing all the work it would take to break up the existing monorepo; instead they'll just add a few small additional repos.

Apaec 7y ago

This proposal makes me scared for my bet on going full with NixOS. To me it seems like a really complicated technical solution to a social problem(the monorepo approach doesn't scale). If more people are needed to maintain the nixpkgs repo why instead not focus on ways to increase the community size. Missing docs have been a problem for NixOS for a long time, most awesome features are lost on some proficient users tweets. Having better docs would do wonders for the community. I think if NixOS wants to copy some of Rust strengths, quality docs should be its first priority.

ris 7y ago
> the monorepo approach doesn't scale Citation needed.
- Apaec 7y ago
  That's mentioned by edolstra in a comment on the proposal. https://github.com/NixOS/rfcs/pull/49#discussion_r305450668 https://github.com/NixOS/rfcs/pull/49#discussion_r305450668
  - gmfawcett 7y ago
    I know who the author of that comment is, but it is not a universally held opinion among Nix users.