6 ms·

This post got me thinking: why must every language reinvent the package manager, poorly? It seems profoundly wasteful. How much of the logic and infrastructure

by tikhonj 12y ago

This post got me thinking: why must every language reinvent the package manager, poorly? It seems profoundly wasteful. How much of the logic and infrastructure involved is actually specific to the language being used? I could be missing something, but it seems very little. It would be really cool if languages just started using a common package manager instead. It could also be more fully-featured, able to do things like track dependencies that aren't written in the language like C libraries. We would also avoid having a ton of custom file formats and dependency solvers. From what I've seen, Nix[1] is the perfect candidate for this. It's self-contained, can install packages locally and does a good job of simultaneously managing multiple versions of the same package: all very useful for development. It's also portable, working on Linux, OS X, FreeBSD and even Windows with CygWin. In a sense, I suppose this would be like reinventing the system package manager. But I don't think that's a problem. Unfortunately, most Linux package managers seem configured in a way that is not convenient for development (ie global packages only). It's also a bit of a pain for a language community to maintain a package for everyone's favorite package manager; if they just settled on one (like Nix), they could have a single Nix repository similar to how they have a custom package list now. [1]: https://nixos.org/nix/ https://nixos.org/nix/

eudox 12y ago
We have this discussion literally every time. Different languages have different conventions and build systems, and packaging a library for every operating system package manager is just not feasible. A single package, for a single package manager that runs everywhere and is tightly integrated with the language, is the only reasonable solution.
- dubcanada 12y ago
  I see where he is coming from though. When I look at github repos and they have more random config files in their home directory then actual source files it starts to scare me.
- pnathan 12y ago
  IMO,a multiplexing system should be done: a stack emits a given standard output. Which then gets read by a operating system package builder and ginned into the OS standard format.
  - TylerE 12y ago
    That's never going to work. How is yum or apt ever going to grok the 5+ python virtualenvs I'm working with at any given time?
- tikhonj 12y ago
  My point is not to use the system package manager, but to replace the various language-specific ones with, say, Nix. It should be flexible enough to work with the different languages' build systems and conventions, while reusing all the existing systems for managing dependencies, conflicts and so on. Very little of all that logic is language specific, and it's traditionally done poorly by language-specific package managers. Basically, I do agree on using a single package manager. I just don't think that the need to be integrated with the language is severe enough to warrant writing a new one for each language! It's much less effort for everyone involved to set integration up for an existing package manager instead. In other words: replace Pip/Cabal/Babel/whatever with Nix, maybe with some plugins for the needs of your specific language. You would even still ship it with your language (optionally), so the only change would be maintaining a plugin, if necessary, instead of a whole manager.
  - munificent 12y ago
    > to replace the various language-specific ones with, say, Nix. So now, to use packages for programming language X, I also have to install Nix, be running a Unix-based OS (sorry Windows folks!), and have whatever other dependencies and toolchain Nix requires installed. This is the core problem. A programming language needs a package manager today. Code reuse is too important for a language to succeed with out a nice user experience to get and share code and deal with transitive dependencies. That puts a package manager on the critical path for a language. Given that, it's important that the package manager itself have minimal dependencies. It would suck, for example, if a package manager for Python was written in Ruby and required Ruby to be installed! The only thing most languages' package managers are willing to depend on is the language itself and its core libraries. That means for every programming language, you get a new package manager. It's not ideal, but it seems to be the best solution we've found given those constraints.
    - pmahoney 12y ago
      I don't really disagree with your points, but "the best solution we've found given those constraints" include a lot of pain. Something like Nix comes with a lot of different pain, but I suspect many pain points could be lessened if language-specific package managers were designed with consideration for more generic tools. For example, rubygems cannot properly express dependencies on C libraries. Rubygems that include C extensions run the compilation step during "gem install" [1] (which typically assumes the presence of a C compiler, etc.). A gem like libxml-ruby assumes the presence of libxml development headers. (Nokogiri punted and includes an entire libxml source tree in its gem which leads to its own problems.) Nix has some means to convert rubygems into nix packages, and it has a list of overlay dependencies to apply to various common gems (e.g. libxml-ruby gets enhanced with that dependency on libxml). Language-specific package managers handle simple cases well, but other cases make increasingly large assumptions about the external world that complicate automation and confound repeatability. [1] There is support for binary gems, but I don't know of anyone who uses these in practice, nor do I know of a repository of pre-compiled gems.
- rdtsc 12y ago
  > A single package, for a single package manager that runs everywhere and is tightly integrated with the language, is the only reasonable solution. Well yeah if your product is uni-language. If it is not then you are installing python pips, ruby gems, java jars, make && make installs for C++ code. And some stuff is uni-language, granted. A lot of Github demo projects and libraries are uni-langue And then they might have cross-dependencies, some have transitive dependencies. And you might have to uninstall some old package and install new one, except you can't make make & make install just dumps files on the file system and doesn't know how to properly clean up after itself. Then in many cases uni-platform is common. That calls for using the OS package manager. Say if your services sit on tops of CentOS 6.5 then why run around sprinkling files all over the file systems via shell scripts or re-implement an adhoc package manager in bash when you can just use RPMs. I have seen this all too many time "RPM writers are stupid, look at this complexity, I'll do it quicker". In the end they come up with something messier, less reliable, not-quite-working-right re-implementation of RPMs.
zimbatm 12y ago
Nix doesn't provide the source package metadata and publishing infrastucture that CPAN/rubygems/pypi provide. In fact nixpkgs is a consumer of those infrastructures. On the client side, nix expressions' dependency resolution is name-based. Meaning you can say: my program depends on "ruby" and the definition of "ruby" gives you the version of ruby. As program dependencies tend to evolve quite a bit it would be a chore to update all the dependencies manually, we are used to having a sat solver to do the work for us. Both things could be bolted onto Nix I think. You just need that publishing infrastructure (github is an existing one for a start) and a dependency solver that generates nix expressions.
- pmahoney 12y ago
  > dependency solver that generates nix expressions I have some code that uses Bundler to convert its Gemfile.lock (resolved dependencies) into a set of nix expressions. I think it's a nice way to go. (Though Bundler is aimed at end users, which for my use case means an annoyingly large amount of functionality and no stable api; e.g. it insists on installing gems as part of the dependency resolution process.) Last I checked, Nix has no way to publish just a small nix expression for one project. You'd really need to include the entire stack of nix expressions for all your dependencies, all the way down to gcc (which is exactly what nixpkgs is), or somehow declare "depends on http://nixpkgs/2014-07-24.tar.xz" http://nixpkgs/2014-07-24.tar.xz".
  - zimbatm 12y ago
    That's pretty cool. I was thinking of bundling a default.nix in the root of my projects with maybe a recipes/ folder for things like ruby, libxml2, ... It's definitely possible to do that and even link to nixpkgs but then your projects dependencies might break each time you updated nixpkgs. I don't know of a nice way to solve that. My sample looked something like that: let pkgs = import <nixpkgs> {}; stdenv = pkgs.stdenv; in stdenv.mkDerivation { name = "my_project"; src = ./.; ... There's also a deps.mix for just the dependencies. It's on an internal project so I can't really divulge the details but email me if you want to talk more about it.
programminggeek 12y ago
It does a terrible job of selling itself. "The Purely Functional Package Manager" - what does that mean and why do I want that instead of ruby gems, php composer, jar files, maven, leinegen, nugget, etc? "Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible." - What if I want Windows? (serious question) I could go on, but none of what is on the landing page explains why Nix should replace ruby gems, php composer, or any other package manager. It's like saying you are mad at the 13 other standards so you create the 14th standard for people to standardize around. Now you have 14 different standards.
- sparkie 12y ago
  By "Purely functional package manager", it's implying "referential transparency" of package definitions/build processes - meaning that given the same inputs to the package definition each time, it should always produce the same result. (OK, in practice this doesn't work exactly as we want it because of various things out of our control, such as timestamps which get inserted into binaries - but there's ongoing work to overcome these deficiencies so we can create binary-exact replicas, reliably[1]). For now, Nix is only referentially transparent at the source level - it requires that packages are built from the same source code that was specified in the original package definition. The reason we want referential transparency for packages is to make them reliable - so that another person building your package gets the same result you did, and something like an environment variable, missing packages, or different native package on his system doesn't cause the software to break, or simply fail to build. If you've ever tried to build some software and configure or make has returned some cryptic error you end up googling - this is exactly what Nix solves. On "What about Windows?" It's a serious question, but the serious answer is that nobody is interested - if Windows users really wanted it, they'd build it, and so far, they're two decades behind even the most basic kind of package management offered by Debian. The best Windows has so far is nuget, which is language specific, and also pretty miserable at actually solving the package management problem - it's a convenient hack. A more interesting project on Windows is CoApp, although I'm not sure if it's still in development. The idea behind CoApp was to create a Visual Studio project file for common open source software and test them - such that Windows users will not be required to install mingw/msys/cygwin to provide some basic functions to their machines. Nobody is suggesting that Nix should replace all the package managers, but the idea behind it is probably more valuable than others going into the future where complexity will continue to increase, and traditional package managers have stagnated in solving our dependency problems. The issue of standards and "one to rule them all" isn't really what nix aims for either - it's in fact, completely incompatible with other PMs by nature - because it requires that dependencies are exact, and other PMs simply cannot provide anything of that nature - they only produce "approximate" packages. [1]:https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds
lmm 12y ago
I once spent a few days trying to package up some python with Maven. It worked, but how many real advantages are there? A package manager is not a huge amount of code compared to e.g. a language standard library (and languages that reuse a standard library often do reuse the package manager). Even within a single ecosystem there are often multiple competing package managers, because people disagree about the right way to do things and often the best way to resolve this is to let them try and see how it turns out. And people derive value from having the package manager written in the same language they're developing in; it means they can extend or integrate it without having to learn a different language. But a package manager that relies on Cygwin certainly isn't going to win. Most language-specific package managers are already doing better than that.
malandrew 12y ago
Having worked on trying to generalize npm, component and bower, I do agree that a lot of what package managers do can be generalized. However, there are ways in which package manages can be very useful if there are bindings for the language in question. More specifically, the import/require statements in a language, coupled with the information you can get from a package manager and syntax tree parsers provides enough information to map out entire programming ecosystems. If there were ever one package manager to rule them all, it would have to support bindings (possibly as a plugin) for every language for it to truly be as valuable as the language/runtimes it supports.
- MichaelDickens 12y ago
  If you're the author of a new programming language, it's easier for you to develop a plugin for an existing package manager than to develop a new package manager.
  - dom96 12y ago
    Is it though? Do existing package managers even provide a way for you to write plugins for them?
    - malandrew 12y ago
      Not that I know of. I haven't come across many that are thoroughly documented for the sake of people that come later and furthermore many come with the baggage of the decisions made early on that are difficult if not impossible to change. No one wants a "so so package manager" that supports plugins. Furthermore, writing a package manager is usually one of the first large scale projects common to most language ecosystems that really exercise what a language is capable of. Other tools for working with the language, such as parsers, interpreters and compilers are among the first projects in any ecosystem.

by tikhonj 12y ago

eudox 12y ago

We have this discussion literally every time. Different languages have different conventions and build systems, and packaging a library for every operating system package manager is just not feasible. A single package, for a single package manager that runs everywhere and is tightly integrated with the language, is the only reasonable solution.

dubcanada 12y ago
I see where he is coming from though. When I look at github repos and they have more random config files in their home directory then actual source files it starts to scare me.
pnathan 12y ago
IMO,a multiplexing system should be done: a stack emits a given standard output. Which then gets read by a operating system package builder and ginned into the OS standard format.
- TylerE 12y ago
  That's never going to work. How is yum or apt ever going to grok the 5+ python virtualenvs I'm working with at any given time?
tikhonj 12y ago
My point is not to use the system package manager, but to replace the various language-specific ones with, say, Nix. It should be flexible enough to work with the different languages' build systems and conventions, while reusing all the existing systems for managing dependencies, conflicts and so on. Very little of all that logic is language specific, and it's traditionally done poorly by language-specific package managers. Basically, I do agree on using a single package manager. I just don't think that the need to be integrated with the language is severe enough to warrant writing a new one for each language! It's much less effort for everyone involved to set integration up for an existing package manager instead. In other words: replace Pip/Cabal/Babel/whatever with Nix, maybe with some plugins for the needs of your specific language. You would even still ship it with your language (optionally), so the only change would be maintaining a plugin, if necessary, instead of a whole manager.
- munificent 12y ago
  > to replace the various language-specific ones with, say, Nix. So now, to use packages for programming language X, I also have to install Nix, be running a Unix-based OS (sorry Windows folks!), and have whatever other dependencies and toolchain Nix requires installed. This is the core problem. A programming language needs a package manager today. Code reuse is too important for a language to succeed with out a nice user experience to get and share code and deal with transitive dependencies. That puts a package manager on the critical path for a language. Given that, it's important that the package manager itself have minimal dependencies. It would suck, for example, if a package manager for Python was written in Ruby and required Ruby to be installed! The only thing most languages' package managers are willing to depend on is the language itself and its core libraries. That means for every programming language, you get a new package manager. It's not ideal, but it seems to be the best solution we've found given those constraints.
  - pmahoney 12y ago
    I don't really disagree with your points, but "the best solution we've found given those constraints" include a lot of pain. Something like Nix comes with a lot of different pain, but I suspect many pain points could be lessened if language-specific package managers were designed with consideration for more generic tools. For example, rubygems cannot properly express dependencies on C libraries. Rubygems that include C extensions run the compilation step during "gem install" [1] (which typically assumes the presence of a C compiler, etc.). A gem like libxml-ruby assumes the presence of libxml development headers. (Nokogiri punted and includes an entire libxml source tree in its gem which leads to its own problems.) Nix has some means to convert rubygems into nix packages, and it has a list of overlay dependencies to apply to various common gems (e.g. libxml-ruby gets enhanced with that dependency on libxml). Language-specific package managers handle simple cases well, but other cases make increasingly large assumptions about the external world that complicate automation and confound repeatability. [1] There is support for binary gems, but I don't know of anyone who uses these in practice, nor do I know of a repository of pre-compiled gems.
rdtsc 12y ago
> A single package, for a single package manager that runs everywhere and is tightly integrated with the language, is the only reasonable solution. Well yeah if your product is uni-language. If it is not then you are installing python pips, ruby gems, java jars, make && make installs for C++ code. And some stuff is uni-language, granted. A lot of Github demo projects and libraries are uni-langue And then they might have cross-dependencies, some have transitive dependencies. And you might have to uninstall some old package and install new one, except you can't make make & make install just dumps files on the file system and doesn't know how to properly clean up after itself. Then in many cases uni-platform is common. That calls for using the OS package manager. Say if your services sit on tops of CentOS 6.5 then why run around sprinkling files all over the file systems via shell scripts or re-implement an adhoc package manager in bash when you can just use RPMs. I have seen this all too many time "RPM writers are stupid, look at this complexity, I'll do it quicker". In the end they come up with something messier, less reliable, not-quite-working-right re-implementation of RPMs.

zimbatm 12y ago

Nix doesn't provide the source package metadata and publishing infrastucture that CPAN/rubygems/pypi provide. In fact nixpkgs is a consumer of those infrastructures. On the client side, nix expressions' dependency resolution is name-based. Meaning you can say: my program depends on "ruby" and the definition of "ruby" gives you the version of ruby. As program dependencies tend to evolve quite a bit it would be a chore to update all the dependencies manually, we are used to having a sat solver to do the work for us. Both things could be bolted onto Nix I think. You just need that publishing infrastructure (github is an existing one for a start) and a dependency solver that generates nix expressions.

pmahoney 12y ago
> dependency solver that generates nix expressions I have some code that uses Bundler to convert its Gemfile.lock (resolved dependencies) into a set of nix expressions. I think it's a nice way to go. (Though Bundler is aimed at end users, which for my use case means an annoyingly large amount of functionality and no stable api; e.g. it insists on installing gems as part of the dependency resolution process.) Last I checked, Nix has no way to publish just a small nix expression for one project. You'd really need to include the entire stack of nix expressions for all your dependencies, all the way down to gcc (which is exactly what nixpkgs is), or somehow declare "depends on http://nixpkgs/2014-07-24.tar.xz" http://nixpkgs/2014-07-24.tar.xz".
- zimbatm 12y ago
  That's pretty cool. I was thinking of bundling a default.nix in the root of my projects with maybe a recipes/ folder for things like ruby, libxml2, ... It's definitely possible to do that and even link to nixpkgs but then your projects dependencies might break each time you updated nixpkgs. I don't know of a nice way to solve that. My sample looked something like that: let pkgs = import <nixpkgs> {}; stdenv = pkgs.stdenv; in stdenv.mkDerivation { name = "my_project"; src = ./.; ... There's also a deps.mix for just the dependencies. It's on an internal project so I can't really divulge the details but email me if you want to talk more about it.

programminggeek 12y ago

It does a terrible job of selling itself. "The Purely Functional Package Manager" - what does that mean and why do I want that instead of ruby gems, php composer, jar files, maven, leinegen, nugget, etc? "Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible." - What if I want Windows? (serious question) I could go on, but none of what is on the landing page explains why Nix should replace ruby gems, php composer, or any other package manager. It's like saying you are mad at the 13 other standards so you create the 14th standard for people to standardize around. Now you have 14 different standards.

sparkie 12y ago
By "Purely functional package manager", it's implying "referential transparency" of package definitions/build processes - meaning that given the same inputs to the package definition each time, it should always produce the same result. (OK, in practice this doesn't work exactly as we want it because of various things out of our control, such as timestamps which get inserted into binaries - but there's ongoing work to overcome these deficiencies so we can create binary-exact replicas, reliably[1]). For now, Nix is only referentially transparent at the source level - it requires that packages are built from the same source code that was specified in the original package definition. The reason we want referential transparency for packages is to make them reliable - so that another person building your package gets the same result you did, and something like an environment variable, missing packages, or different native package on his system doesn't cause the software to break, or simply fail to build. If you've ever tried to build some software and configure or make has returned some cryptic error you end up googling - this is exactly what Nix solves. On "What about Windows?" It's a serious question, but the serious answer is that nobody is interested - if Windows users really wanted it, they'd build it, and so far, they're two decades behind even the most basic kind of package management offered by Debian. The best Windows has so far is nuget, which is language specific, and also pretty miserable at actually solving the package management problem - it's a convenient hack. A more interesting project on Windows is CoApp, although I'm not sure if it's still in development. The idea behind CoApp was to create a Visual Studio project file for common open source software and test them - such that Windows users will not be required to install mingw/msys/cygwin to provide some basic functions to their machines. Nobody is suggesting that Nix should replace all the package managers, but the idea behind it is probably more valuable than others going into the future where complexity will continue to increase, and traditional package managers have stagnated in solving our dependency problems. The issue of standards and "one to rule them all" isn't really what nix aims for either - it's in fact, completely incompatible with other PMs by nature - because it requires that dependencies are exact, and other PMs simply cannot provide anything of that nature - they only produce "approximate" packages. [1]:https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds

lmm 12y ago

I once spent a few days trying to package up some python with Maven. It worked, but how many real advantages are there? A package manager is not a huge amount of code compared to e.g. a language standard library (and languages that reuse a standard library often do reuse the package manager). Even within a single ecosystem there are often multiple competing package managers, because people disagree about the right way to do things and often the best way to resolve this is to let them try and see how it turns out. And people derive value from having the package manager written in the same language they're developing in; it means they can extend or integrate it without having to learn a different language. But a package manager that relies on Cygwin certainly isn't going to win. Most language-specific package managers are already doing better than that.

malandrew 12y ago

Having worked on trying to generalize npm, component and bower, I do agree that a lot of what package managers do can be generalized. However, there are ways in which package manages can be very useful if there are bindings for the language in question. More specifically, the import/require statements in a language, coupled with the information you can get from a package manager and syntax tree parsers provides enough information to map out entire programming ecosystems. If there were ever one package manager to rule them all, it would have to support bindings (possibly as a plugin) for every language for it to truly be as valuable as the language/runtimes it supports.

MichaelDickens 12y ago
If you're the author of a new programming language, it's easier for you to develop a plugin for an existing package manager than to develop a new package manager.
- dom96 12y ago
  Is it though? Do existing package managers even provide a way for you to write plugins for them?
  - malandrew 12y ago
    Not that I know of. I haven't come across many that are thoroughly documented for the sake of people that come later and furthermore many come with the baggage of the decisions made early on that are difficult if not impossible to change. No one wants a "so so package manager" that supports plugins. Furthermore, writing a package manager is usually one of the first large scale projects common to most language ecosystems that really exercise what a language is capable of. Other tools for working with the language, such as parsers, interpreters and compilers are among the first projects in any ecosystem.