5 ms·

> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies We have a good hunch.

by tazjin 4y ago

> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager. For this to work at scale we'd need to overcome a couple of large problems though (in ascending order of complexity): 1. A better technical implementation of the model (working on it [1]). 2. A mindset shift to make people understand that "binary distribution" is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong). 3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules "modules", "classes" or "gorboodles", whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don't have to NIH a package manager. This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out. [0]: https://code.tvl.fyi/about/nix/buildGo https://code.tvl.fyi/about/nix/buildGo [1]: https://cs.tvl.fyi/depot/-/tree/tvix/ https://cs.tvl.fyi/depot/-/tree/tvix/

folex 4y ago
Yes! I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.
- mijoharas 4y ago
  Not a generic package manager, but it's probably worth calling out asdf as the generic version manager[0] (maybe you're already aware of it, but it's a generic replacement for nvm, rvm, virtualenv, *vm, which supports any language based on plugins.) Again, maybe you're already aware of it, but I think it's a nice example of genericising a concern common to many languages which sounds similar to what you're asking for (albeit unfortunately in a slightly different space). [0] https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf
  - dmix 4y ago
    At a minimum it's useful having the `.tool-versions` dotfile in a project directory which lists the versions of each language, db, and package manager version (our Rails project lists versions for ruby, bundler, postgres, node, yarn, redis). Even if all devs don't use asdf, it's a useful reference point. https://asdf-vm.com/manage/configuration.html#tool-versions https://asdf-vm.com/manage/configuration.html#tool-versions Similar to an `.editorconfig` to globally define stuff like tab widths for IDEs.
- drpyser22 4y ago
  Did you compare nix to npm? Is there a good comparison out there?
- pxc 4y ago
  Check out Denxi. Might be the closest thing to what you're looking for today, and is informed by the state of the art in package management without totally imposing strict discipline on all packages. https://docs.racket-lang.org/denxi-guide/index.html https://docs.racket-lang.org/denxi-guide/index.html
pastage 4y ago
2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at? [0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in class by far. It is also completely different from the notion of compiling a program. I wonder what you consider the goal of a package system.
- tazjin 4y ago
  > are we thinking about Debian packaging in completely different ways? Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number). > Something that even Nix struggles with Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course. > I wonder what you consider the goal of a package system I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed. Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).
potamic 4y ago
What's a good source, for someone who has never used nix, to read about its conceptual model and how it is different from contemporary dependency managers?
- tazjin 4y ago
  There really isn't one that I know of. There's the original thesis[0], but it's from a different time and it focuses only on the package management (~ for a distribution) aspect. Everything else is mostly written to teach people how to use Nix, and the more recent the thing is the more it will focus on surface-level features of the C++ implementation of Nix. [0]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf
  - deleted 4y ago
    [deleted]
- rkrzr 4y ago
  We wrote a blog post on how Nix helps us solve some of the problems that we encountered with other package managers like e.g. apt and pip. https://www.channable.com/tech/nix-is-the-ultimate-devops-toolkit https://www.channable.com/tech/nix-is-the-ultimate-devops-to... (It doesn't go very much in depth on the conceptual model, but touches on the the main ideas)
unicornmama 4y ago
If the community just focused on creating a great package manager instead of an Everything Monster, and “meeting developers where they are”, then Nix might do something great.
noobermin 4y ago
I think 3 is probably true for language developers, but for users, language repos feel like they are needed because like none of the linux distros really ship everything you need, or even a reliable fraction without installing a bunch of dumb hacks to get it to work. It's much easier just to do "pip install" so that's where the demand is. And sure, may be you're right that distro packaging is the "wrong model," again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don't have the option to do the "right" thing, so they do use the mishmash of packaging/repo systems as just the cost of doing business.
- carlmr 4y ago
  Apart from not shipping what you need, you often only get a fraction of the versions you need. For development you may want to test your code against multiple versions of the system libraries. This is not easy using a distro package manager.
  - fragmede 4y ago
    Maybe our definition of "easy" doesn't agree, but for Ksplice, we had to build kernels using the exact same, often old versions of GCC and everything else (binutils, etc). We made schroots (this predated docker) and then we used the system package manager to install the requisite old version of packages. Wasn't trivial to get the whole system setup, but getting old versions of packages installed wasn't impossibly hard either.
puffoflogic 4y ago
Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.
- tazjin 4y ago
  [flagged]
  - pxc 4y ago
    Pretend I asked the nice version of this, for the benefit of other readers? I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?
    - otabdeveloper4 4y ago
      Nix is capable of that, but you need a database mapping versions to VCS refs. Nixpkgs doesn't go down that road because it would be unwieldy at their scale, but more specialized nix-based tools do.
      - pxc 4y ago
        (I actually think Nixpkgs could and should go down this road, with a couple of caveats, for immense benefit. But I'm curious about what tazjin thinks here.)
      - haradion 4y ago
        At least for (unlocked) flake sources, you could probably use a branch naming convention instead of a separate database, although that requires your repo to be set up a specific way.
- otabdeveloper4 4y ago
jnxx 4y ago
And Guix has put that in an beautiful form. There are two things which make Guix special: 1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme). 2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.

by tazjin 4y ago

folex 4y ago

Yes! I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.

mijoharas 4y ago
Not a generic package manager, but it's probably worth calling out asdf as the generic version manager[0] (maybe you're already aware of it, but it's a generic replacement for nvm, rvm, virtualenv, *vm, which supports any language based on plugins.) Again, maybe you're already aware of it, but I think it's a nice example of genericising a concern common to many languages which sounds similar to what you're asking for (albeit unfortunately in a slightly different space). [0] https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf
- dmix 4y ago
  At a minimum it's useful having the `.tool-versions` dotfile in a project directory which lists the versions of each language, db, and package manager version (our Rails project lists versions for ruby, bundler, postgres, node, yarn, redis). Even if all devs don't use asdf, it's a useful reference point. https://asdf-vm.com/manage/configuration.html#tool-versions https://asdf-vm.com/manage/configuration.html#tool-versions Similar to an `.editorconfig` to globally define stuff like tab widths for IDEs.
drpyser22 4y ago
Did you compare nix to npm? Is there a good comparison out there?
pxc 4y ago
Check out Denxi. Might be the closest thing to what you're looking for today, and is informed by the state of the art in package management without totally imposing strict discipline on all packages. https://docs.racket-lang.org/denxi-guide/index.html https://docs.racket-lang.org/denxi-guide/index.html

pastage 4y ago

2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at? [0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in class by far. It is also completely different from the notion of compiling a program. I wonder what you consider the goal of a package system.

tazjin 4y ago
> are we thinking about Debian packaging in completely different ways? Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number). > Something that even Nix struggles with Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course. > I wonder what you consider the goal of a package system I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed. Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).

potamic 4y ago

What's a good source, for someone who has never used nix, to read about its conceptual model and how it is different from contemporary dependency managers?

tazjin 4y ago
There really isn't one that I know of. There's the original thesis[0], but it's from a different time and it focuses only on the package management (~ for a distribution) aspect. Everything else is mostly written to teach people how to use Nix, and the more recent the thing is the more it will focus on surface-level features of the C++ implementation of Nix. [0]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf
- deleted 4y ago
  [deleted]
rkrzr 4y ago
We wrote a blog post on how Nix helps us solve some of the problems that we encountered with other package managers like e.g. apt and pip. https://www.channable.com/tech/nix-is-the-ultimate-devops-toolkit https://www.channable.com/tech/nix-is-the-ultimate-devops-to... (It doesn't go very much in depth on the conceptual model, but touches on the the main ideas)

unicornmama 4y ago

If the community just focused on creating a great package manager instead of an Everything Monster, and “meeting developers where they are”, then Nix might do something great.

noobermin 4y ago

I think 3 is probably true for language developers, but for users, language repos feel like they are needed because like none of the linux distros really ship everything you need, or even a reliable fraction without installing a bunch of dumb hacks to get it to work. It's much easier just to do "pip install" so that's where the demand is. And sure, may be you're right that distro packaging is the "wrong model," again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don't have the option to do the "right" thing, so they do use the mishmash of packaging/repo systems as just the cost of doing business.

carlmr 4y ago
Apart from not shipping what you need, you often only get a fraction of the versions you need. For development you may want to test your code against multiple versions of the system libraries. This is not easy using a distro package manager.
- fragmede 4y ago
  Maybe our definition of "easy" doesn't agree, but for Ksplice, we had to build kernels using the exact same, often old versions of GCC and everything else (binutils, etc). We made schroots (this predated docker) and then we used the system package manager to install the requisite old version of packages. Wasn't trivial to get the whole system setup, but getting old versions of packages installed wasn't impossibly hard either.

puffoflogic 4y ago

Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.

tazjin 4y ago
[flagged]
- pxc 4y ago
  Pretend I asked the nice version of this, for the benefit of other readers? I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?
  - otabdeveloper4 4y ago
    Nix is capable of that, but you need a database mapping versions to VCS refs. Nixpkgs doesn't go down that road because it would be unwieldy at their scale, but more specialized nix-based tools do.
    - pxc 4y ago
      (I actually think Nixpkgs could and should go down this road, with a couple of caveats, for immense benefit. But I'm curious about what tazjin thinks here.)
    - haradion 4y ago
      At least for (unlocked) flake sources, you could probably use a branch naming convention instead of a separate database, although that requires your repo to be set up a specific way.
otabdeveloper4 4y ago

jnxx 4y ago

And Guix has put that in an beautiful form. There are two things which make Guix special: 1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme). 2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.