4 ms·

My point is not to use the system package manager, but to replace the various language-specific ones with, say, Nix. It should be flexible enough to work with t

by tikhonj 12y ago

My point is not to use the system package manager, but to replace the various language-specific ones with, say, Nix. It should be flexible enough to work with the different languages' build systems and conventions, while reusing all the existing systems for managing dependencies, conflicts and so on. Very little of all that logic is language specific, and it's traditionally done poorly by language-specific package managers. Basically, I do agree on using a single package manager. I just don't think that the need to be integrated with the language is severe enough to warrant writing a new one for each language! It's much less effort for everyone involved to set integration up for an existing package manager instead. In other words: replace Pip/Cabal/Babel/whatever with Nix, maybe with some plugins for the needs of your specific language. You would even still ship it with your language (optionally), so the only change would be maintaining a plugin, if necessary, instead of a whole manager.

munificent 12y ago
> to replace the various language-specific ones with, say, Nix. So now, to use packages for programming language X, I also have to install Nix, be running a Unix-based OS (sorry Windows folks!), and have whatever other dependencies and toolchain Nix requires installed. This is the core problem. A programming language needs a package manager today. Code reuse is too important for a language to succeed with out a nice user experience to get and share code and deal with transitive dependencies. That puts a package manager on the critical path for a language. Given that, it's important that the package manager itself have minimal dependencies. It would suck, for example, if a package manager for Python was written in Ruby and required Ruby to be installed! The only thing most languages' package managers are willing to depend on is the language itself and its core libraries. That means for every programming language, you get a new package manager. It's not ideal, but it seems to be the best solution we've found given those constraints.
- pmahoney 12y ago
  I don't really disagree with your points, but "the best solution we've found given those constraints" include a lot of pain. Something like Nix comes with a lot of different pain, but I suspect many pain points could be lessened if language-specific package managers were designed with consideration for more generic tools. For example, rubygems cannot properly express dependencies on C libraries. Rubygems that include C extensions run the compilation step during "gem install" [1] (which typically assumes the presence of a C compiler, etc.). A gem like libxml-ruby assumes the presence of libxml development headers. (Nokogiri punted and includes an entire libxml source tree in its gem which leads to its own problems.) Nix has some means to convert rubygems into nix packages, and it has a list of overlay dependencies to apply to various common gems (e.g. libxml-ruby gets enhanced with that dependency on libxml). Language-specific package managers handle simple cases well, but other cases make increasingly large assumptions about the external world that complicate automation and confound repeatability. [1] There is support for binary gems, but I don't know of anyone who uses these in practice, nor do I know of a repository of pre-compiled gems.
- untothebreach 12y ago
  One idea I had was to leverage nix channels to use Nix as the package manager _only for language packages_. Instead of downloading the default set of packages, you subscribe to the "Wren" (;-)) channel with `$ nix-channel --add http://github.com/wren/nixpkgs` http://github.com/wren/nixpkgs` and now you have all the local packages for your language. It could basically "be" the PyPI or RubyGems for a language.
- sparkie 12y ago
  You're focusing too much on "THE Nix", and not the idea behind it. Nix is an idea which detracts from traditional package management by concluding that ranged dependency versions actually create more problems than they solve, so we should just do away with them, give every package an immutable identity, and specify the identities as dependencies - thereby making reproduction significantly more reliable and free of unwanted side effects like environment variables or user-built packages (outside of the distro's own). Nothing in this idea says you need to run a Unix based OS, only the two current implementations of the idea (Nix and Guix) do so currently. The concept can be applied to a language specific package manager (LSPM) too, and IMO would be much better than the way many of the existing LSPMs behave - which is to duplicate dependencies into project directories - almost completely eliminating the reason to have shared object dependencies over static linked dependencies in the first place - because you need to manually update the dependencies of every project when a dependency gets a bug-fix. The real problem with LSPMs is that they're functionally incomplete PMs - they require you to (manually) use a system wide package manager to install the native dependencies which you inevitably need, because it's inefficient to rewrite everything in your language rather than provide bindings to a native library. The LSPMs basically spit out their dummy when you try to install something and a native dependency is not met - often giving a cryptic error message which doesn't explain the cause of a problem, or how to resolve it. I've suggested several times what needs to be done to resolve this, and it doesn't involve throwing away the LSPMs, but augmenting them. The solution is to provide a system wide daemon which facilitates native installs required for by the language, whereby the LSPM can query and request the installation of packages over a well defined protocol, and the daemon forwards those requests to a system-level PM. This could also improve the state of GUIs for package management - allowing a unified interface to installing them rather than a dozen ad-hoc CLI interfaces. Of course, I don't have all of the details on how this would be implemented - for starters, it would add a new dependency to every package manager using it (the IPC mechanism and protocol used), although almost any language can use a socket based IPC. Each LSPM would also need to introduce some new notation for "external" packages, which it doesn't manage internally, but instead sends off a query to the daemon, waits for a response which will either fail (the native package is not installed or won't be installed), which will also improve the state of error messages, or it may succeed, and the LSPM can continue building the rest of the project. If such system is going to be built, the Nix model of immutable packages seems really the only sane approach to the problem. The identity-based approach to dependencies means that when building a package, you're always going to be building against dependencies the developers intended - because they're the same ones he/she tested against. Each time a developer builds successfully against a different dependency version he should add its identity to the package definition, confirming that it has been tested. If you decide you want to build against a different version, you can simply adjust the identities, but you should not expect it to work without some manual fixing - because people tend to break things. If you happen to have luck, and your modified dependency works - you should publish the updated package definition so other people can learn from your testing, and blame you if you published a broken package (ie, failure to build implies a bug, as it should)

by tikhonj 12y ago

munificent 12y ago

> to replace the various language-specific ones with, say, Nix. So now, to use packages for programming language X, I also have to install Nix, be running a Unix-based OS (sorry Windows folks!), and have whatever other dependencies and toolchain Nix requires installed. This is the core problem. A programming language needs a package manager today. Code reuse is too important for a language to succeed with out a nice user experience to get and share code and deal with transitive dependencies. That puts a package manager on the critical path for a language. Given that, it's important that the package manager itself have minimal dependencies. It would suck, for example, if a package manager for Python was written in Ruby and required Ruby to be installed! The only thing most languages' package managers are willing to depend on is the language itself and its core libraries. That means for every programming language, you get a new package manager. It's not ideal, but it seems to be the best solution we've found given those constraints.

pmahoney 12y ago
I don't really disagree with your points, but "the best solution we've found given those constraints" include a lot of pain. Something like Nix comes with a lot of different pain, but I suspect many pain points could be lessened if language-specific package managers were designed with consideration for more generic tools. For example, rubygems cannot properly express dependencies on C libraries. Rubygems that include C extensions run the compilation step during "gem install" [1] (which typically assumes the presence of a C compiler, etc.). A gem like libxml-ruby assumes the presence of libxml development headers. (Nokogiri punted and includes an entire libxml source tree in its gem which leads to its own problems.) Nix has some means to convert rubygems into nix packages, and it has a list of overlay dependencies to apply to various common gems (e.g. libxml-ruby gets enhanced with that dependency on libxml). Language-specific package managers handle simple cases well, but other cases make increasingly large assumptions about the external world that complicate automation and confound repeatability. [1] There is support for binary gems, but I don't know of anyone who uses these in practice, nor do I know of a repository of pre-compiled gems.
untothebreach 12y ago
One idea I had was to leverage nix channels to use Nix as the package manager _only for language packages_. Instead of downloading the default set of packages, you subscribe to the "Wren" (;-)) channel with `$ nix-channel --add http://github.com/wren/nixpkgs` http://github.com/wren/nixpkgs` and now you have all the local packages for your language. It could basically "be" the PyPI or RubyGems for a language.
sparkie 12y ago
You're focusing too much on "THE Nix", and not the idea behind it. Nix is an idea which detracts from traditional package management by concluding that ranged dependency versions actually create more problems than they solve, so we should just do away with them, give every package an immutable identity, and specify the identities as dependencies - thereby making reproduction significantly more reliable and free of unwanted side effects like environment variables or user-built packages (outside of the distro's own). Nothing in this idea says you need to run a Unix based OS, only the two current implementations of the idea (Nix and Guix) do so currently. The concept can be applied to a language specific package manager (LSPM) too, and IMO would be much better than the way many of the existing LSPMs behave - which is to duplicate dependencies into project directories - almost completely eliminating the reason to have shared object dependencies over static linked dependencies in the first place - because you need to manually update the dependencies of every project when a dependency gets a bug-fix. The real problem with LSPMs is that they're functionally incomplete PMs - they require you to (manually) use a system wide package manager to install the native dependencies which you inevitably need, because it's inefficient to rewrite everything in your language rather than provide bindings to a native library. The LSPMs basically spit out their dummy when you try to install something and a native dependency is not met - often giving a cryptic error message which doesn't explain the cause of a problem, or how to resolve it. I've suggested several times what needs to be done to resolve this, and it doesn't involve throwing away the LSPMs, but augmenting them. The solution is to provide a system wide daemon which facilitates native installs required for by the language, whereby the LSPM can query and request the installation of packages over a well defined protocol, and the daemon forwards those requests to a system-level PM. This could also improve the state of GUIs for package management - allowing a unified interface to installing them rather than a dozen ad-hoc CLI interfaces. Of course, I don't have all of the details on how this would be implemented - for starters, it would add a new dependency to every package manager using it (the IPC mechanism and protocol used), although almost any language can use a socket based IPC. Each LSPM would also need to introduce some new notation for "external" packages, which it doesn't manage internally, but instead sends off a query to the daemon, waits for a response which will either fail (the native package is not installed or won't be installed), which will also improve the state of error messages, or it may succeed, and the LSPM can continue building the rest of the project. If such system is going to be built, the Nix model of immutable packages seems really the only sane approach to the problem. The identity-based approach to dependencies means that when building a package, you're always going to be building against dependencies the developers intended - because they're the same ones he/she tested against. Each time a developer builds successfully against a different dependency version he should add its identity to the package definition, confirming that it has been tested. If you decide you want to build against a different version, you can simply adjust the identities, but you should not expect it to work without some manual fixing - because people tend to break things. If you happen to have luck, and your modified dependency works - you should publish the updated package definition so other people can learn from your testing, and blame you if you published a broken package (ie, failure to build implies a bug, as it should)