5 ms·

Why do we keep running into this problem over and over again? Why do we need one package manager per programming language? Why is software distribution so damn

by jordigh 12y ago

Why do we keep running into this problem over and over again? Why do we need one package manager per programming language? Why is software distribution so damn hard?

makmanalp 12y ago
Because general purpose package managers suck at dealing with the concerns of individual programming languages. E.g. installation paths, virtual environments, different compilation methods, etc. There is also no single cross-platform package manager. Mac has macports and homebrew, windows barely got one, linux has yum, apt and others. As a language designer, you don't want any part in this, you want your code to work. So you make your own. edit: To clarify, I'm suggesting that we're stuck in a local maximum - now that all these systems already exist and have widespread adoption and tool dependency in their own environments, there isn't much of an impetus to use Nix or something similar even though in the grander scheme of things it'd be way nicer.
- zdw 12y ago
  I think the solution is to separate the problem into two parts - The "puts things on disk" part (posix "install" or a low level tool like dpkg or rpm is most like this) - The "determine what needs to be installed" part, (aptitude, yum) I'm all for a per-language or system version of the latter, but once that's done, you should generate packages installed by the former. Heck, wrap all the commands (cp/mv/install/chmod/chown/etc.) that write stuff to permanent places on disk to actually do "add to a package", give it a basic name/version number, and have the low level tool handle adding/removing it from the system (or multiple systems, or deploy it, etc.). All the dependencies, compatibility, etc. are handled by the higher level system. This gets you the best of both worlds - system level packages, and the ability to install whatever you need. FPM (https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm) is a pretty good example of this philosophy. But, instead, we get every punk ass CPAN descendent spraying it's crap all over the filesystem, needing the whole build environment installed, touching the internet in weird ways that don't guaranteed repeatable behavior, etc. sigh
- davexunit 12y ago
  Haskell folks seem to be gravitating towards the Nix package manager, which is general-purpose. We need to stop the proliferation of a package manager per programming language and make systems package managers capable of handling the use-cases of pip, bundler, npm, cabal, etc. Nix and Guix are the only 2 general-purpose package managers that I know of that can do it.
  - tel 12y ago
    I don't think it's gravitating "toward" Nix. Many members of the community are trying to use Nix personally and maybe Cabal proposals are inspired by Nix. I don't think the idea today would be that the entire community should move to Nix, however, not by a long shot.
  - pjmlp 12y ago
    > Nix and Guix are the only 2 general-purpose package managers that I know of that can do it. For anyone constrained to GNU/Linux, right?
    - davexunit 12y ago
      > For anyone constrained to GNU/Linux, right? I don't feel constrained by a free software operating system. Besides that, a bunch of people use Nix on OS X. On Windows I suppose you're stuck with the inferior package managers.
      - dllthomas 12y ago
        "I don't feel constrained by a free software operating system." I appreciate the sentiment, but there's a substantial difference between "constrained to" and "constrained by".
      - pjmlp 12y ago
        There are many other OS out there. Why do people only think of the triad?
    - masklinn 12y ago
- cschneid 12y ago
  In my local group of haskellers, nixos is taking off as a system level package manager that does virtual envs, and multiple package versions pretty well. I haven't had a chance to play with it, but perhaps a sufficiently smart package manager can do both?
- npsimons 12y ago
  Because general purpose package managers suck at dealing with the concerns of individual programming languages. E.g. installation paths, virtual environments, different compilation methods, etc. And yet all these nifty package managers can't even seem to get things as simple and long solved as permissions straight. Even autoconf sets permissions better than, say, pip, where I have to remember to check my umask before I "sudo pip install something".
  - deleted 12y ago
    [deleted]
- 14113 12y ago
  To add to that, the concerns that operating system package managers have are very different from the ones that programming language package managers have. With an operating systems package manager, the focus is on shipping working, uber stable code, which is unlikely to break someones system. This (imho) is due to two different factors: 1) Operating systems have to work, no two ways about it, if your OS is broken, everything else is broken. 2) Users of operating systems are not necessarily experts. If their package manager ships experimental code which breaks their particular system, they don't know how to report the bug to the central OS maintainers. I think this is handled somewhat by having different repositorys with different levels of stability, a la arch linux's [core/extra/community/testing/AUR], however at the end of the day, the main repository must avoid having breaking code. With a programming language package manager, the focus is on having up to date features, and catering towards power users who, if things go wrong, can generally fix them, or at least know how to contact, and how to phrase their requests for assistance. This means that it's generally more acceptable to expect users of a programming language repository to occasionally be served broken code, as defects will be reported more rapidly, and more clearly. I find it interesting comparing the two package managers that I use most often in my day to day computing experience: Cabal, and pacman (Arch linux). Pacman offers a single version of each library per repository, and that version is as stable and tested as the repository requires. This is in keeping with the spirit of an operating system package manager, as it allows power users to install unstable packages from more experimental repositories, but tries to serve as stable code as possible to general users. Contrast that which cabal, which basically offers no stability guarantees, but allows for much more fine grained control over library versions, sandboxing etc. TL;DR In my opinion, OS and Language package managers have goals which are at odds with each other: OS package managers want to maintain stability, Language package managers want to allow for bleeding edge code. The concessions that cabal makes towards stability (versions, sandboxes etc) often cause other problems as well.
MrBuddyCasino 12y ago
Some package managers are part of build systems too, e.g. the Maven ecosystem. Now in what language should you write extensions or plugins? There will be djihad.
Doji 12y ago
The problem Cabal is trying to solve here is supposedly more difficult than that solved by Gem, Pip, and other package managers. GHC does some kind of cross-library optimization that causes dependency problems in routine situations. I agree we shouldn't have one package manager per language, but I think so far no one package manager has proved sufficiently general and robust to serve all use cases. For example, I think something like Nix would be wonderful for all language communities, but it doesn't run on windows. That immediately takes it out of the running. And as far as I'm aware, all other package mangers would have the same "Hell" problems of cabal because of the GHC optimization mentioned above.
- freyrs3 12y ago
  It is more difficult. By design Haskell shifts an enormous number of failure modes to compile-time and as such cabal gets an unnecessary amount of blame for library bugs. If you ``pip install`` a package that transitively dependencies on another library where the author has accidently introduced a backwards incompatibility in the API that occurs in 5% of use-cases, pip won't detect the change and will happily install the package and 95% of people will assume no problem exists until they hit a runtime failure. The job that pip has to do is trivial actually. In Haskell if there is an incompatibility that breaks the interface, and in the presence of "wild west" packages there will be, then it will manifest as a compile time build error instead of proceeding.
- sjy 12y ago
  Not only is the package management problem more difficult, it's not the problem that Cabal (the Common Architecture for Building Applications and Libraries) was intended to solve! cabal-install is not a package manager; it can't uninstall packages or even record which packages it has installed.
bmurphy1976 12y ago
Let's not forget they are all garbage from somebody's perspective. I know Python. Would I prefer to write my package manifests in some bastardized Ruby DSL or Python? Of course I'd prefer Python and the Ruby guys would prefer their bastardized DSL. I don't blame them. How do we bridge that gap? Python => Ruby => Python is bad enough. What about M4 or AWK or Makefiles or Haskell or god help us all some shitty custom format ala Puppet? You might as well demand everybody on Earth speak English.
- lucian1900 12y ago
  > You might as well demand everybody on Earth speak English. That's working out quite well so far.
  - lutusp 12y ago
    >> You might as well demand everybody on Earth speak English. > That's working out quite well so far. Not really. English is a minority language, spoken as a primary language by only about 5.5% of the world's population. http://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers http://en.wikipedia.org/wiki/List_of_languages_by_number_of_... One more juicy anecdote: there are more Chinese people learning English right now, than there are English speakers in the U.S..
    - jessaustin 12y ago
      Your postscript seems to confirm GP's point. b^)
      - dllthomas 12y ago
        I don't know that that's a sign of it working well...
      - lutusp 12y ago
        Not if the students' plan is only to remain competitive with native English speakers. Knowing how to speak English as a second language, for pragmatic reasons, isn't the same as being an English-speaker in the sense of the parent post.
endgame 12y ago
I think a lot of it is that people are scared of automake, so they reinvent their own build systems, with varying degrees of badness. Then they claim that theirs is "easy" but they've only done the first half of "simple things should be easy, hard things should be possible". A properly autotooled package offers an amazing level of control about where to put things.

by jordigh 12y ago

Why do we keep running into this problem over and over again? Why do we need one package manager per programming language? Why is software distribution so damn hard?

makmanalp 12y ago

Because general purpose package managers suck at dealing with the concerns of individual programming languages. E.g. installation paths, virtual environments, different compilation methods, etc. There is also no single cross-platform package manager. Mac has macports and homebrew, windows barely got one, linux has yum, apt and others. As a language designer, you don't want any part in this, you want your code to work. So you make your own. edit: To clarify, I'm suggesting that we're stuck in a local maximum - now that all these systems already exist and have widespread adoption and tool dependency in their own environments, there isn't much of an impetus to use Nix or something similar even though in the grander scheme of things it'd be way nicer.

zdw 12y ago
I think the solution is to separate the problem into two parts - The "puts things on disk" part (posix "install" or a low level tool like dpkg or rpm is most like this) - The "determine what needs to be installed" part, (aptitude, yum) I'm all for a per-language or system version of the latter, but once that's done, you should generate packages installed by the former. Heck, wrap all the commands (cp/mv/install/chmod/chown/etc.) that write stuff to permanent places on disk to actually do "add to a package", give it a basic name/version number, and have the low level tool handle adding/removing it from the system (or multiple systems, or deploy it, etc.). All the dependencies, compatibility, etc. are handled by the higher level system. This gets you the best of both worlds - system level packages, and the ability to install whatever you need. FPM (https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm) is a pretty good example of this philosophy. But, instead, we get every punk ass CPAN descendent spraying it's crap all over the filesystem, needing the whole build environment installed, touching the internet in weird ways that don't guaranteed repeatable behavior, etc. sigh
davexunit 12y ago
Haskell folks seem to be gravitating towards the Nix package manager, which is general-purpose. We need to stop the proliferation of a package manager per programming language and make systems package managers capable of handling the use-cases of pip, bundler, npm, cabal, etc. Nix and Guix are the only 2 general-purpose package managers that I know of that can do it.
- tel 12y ago
  I don't think it's gravitating "toward" Nix. Many members of the community are trying to use Nix personally and maybe Cabal proposals are inspired by Nix. I don't think the idea today would be that the entire community should move to Nix, however, not by a long shot.
- pjmlp 12y ago
  > Nix and Guix are the only 2 general-purpose package managers that I know of that can do it. For anyone constrained to GNU/Linux, right?
  - davexunit 12y ago
    > For anyone constrained to GNU/Linux, right? I don't feel constrained by a free software operating system. Besides that, a bunch of people use Nix on OS X. On Windows I suppose you're stuck with the inferior package managers.
    - dllthomas 12y ago
      "I don't feel constrained by a free software operating system." I appreciate the sentiment, but there's a substantial difference between "constrained to" and "constrained by".
    - pjmlp 12y ago
      There are many other OS out there. Why do people only think of the triad?
  - masklinn 12y ago
cschneid 12y ago
In my local group of haskellers, nixos is taking off as a system level package manager that does virtual envs, and multiple package versions pretty well. I haven't had a chance to play with it, but perhaps a sufficiently smart package manager can do both?
npsimons 12y ago
Because general purpose package managers suck at dealing with the concerns of individual programming languages. E.g. installation paths, virtual environments, different compilation methods, etc. And yet all these nifty package managers can't even seem to get things as simple and long solved as permissions straight. Even autoconf sets permissions better than, say, pip, where I have to remember to check my umask before I "sudo pip install something".
- deleted 12y ago
  [deleted]
14113 12y ago
To add to that, the concerns that operating system package managers have are very different from the ones that programming language package managers have. With an operating systems package manager, the focus is on shipping working, uber stable code, which is unlikely to break someones system. This (imho) is due to two different factors: 1) Operating systems have to work, no two ways about it, if your OS is broken, everything else is broken. 2) Users of operating systems are not necessarily experts. If their package manager ships experimental code which breaks their particular system, they don't know how to report the bug to the central OS maintainers. I think this is handled somewhat by having different repositorys with different levels of stability, a la arch linux's [core/extra/community/testing/AUR], however at the end of the day, the main repository must avoid having breaking code. With a programming language package manager, the focus is on having up to date features, and catering towards power users who, if things go wrong, can generally fix them, or at least know how to contact, and how to phrase their requests for assistance. This means that it's generally more acceptable to expect users of a programming language repository to occasionally be served broken code, as defects will be reported more rapidly, and more clearly. I find it interesting comparing the two package managers that I use most often in my day to day computing experience: Cabal, and pacman (Arch linux). Pacman offers a single version of each library per repository, and that version is as stable and tested as the repository requires. This is in keeping with the spirit of an operating system package manager, as it allows power users to install unstable packages from more experimental repositories, but tries to serve as stable code as possible to general users. Contrast that which cabal, which basically offers no stability guarantees, but allows for much more fine grained control over library versions, sandboxing etc. TL;DR In my opinion, OS and Language package managers have goals which are at odds with each other: OS package managers want to maintain stability, Language package managers want to allow for bleeding edge code. The concessions that cabal makes towards stability (versions, sandboxes etc) often cause other problems as well.

MrBuddyCasino 12y ago

Some package managers are part of build systems too, e.g. the Maven ecosystem. Now in what language should you write extensions or plugins? There will be djihad.

Doji 12y ago

The problem Cabal is trying to solve here is supposedly more difficult than that solved by Gem, Pip, and other package managers. GHC does some kind of cross-library optimization that causes dependency problems in routine situations. I agree we shouldn't have one package manager per language, but I think so far no one package manager has proved sufficiently general and robust to serve all use cases. For example, I think something like Nix would be wonderful for all language communities, but it doesn't run on windows. That immediately takes it out of the running. And as far as I'm aware, all other package mangers would have the same "Hell" problems of cabal because of the GHC optimization mentioned above.

freyrs3 12y ago
It is more difficult. By design Haskell shifts an enormous number of failure modes to compile-time and as such cabal gets an unnecessary amount of blame for library bugs. If you ``pip install`` a package that transitively dependencies on another library where the author has accidently introduced a backwards incompatibility in the API that occurs in 5% of use-cases, pip won't detect the change and will happily install the package and 95% of people will assume no problem exists until they hit a runtime failure. The job that pip has to do is trivial actually. In Haskell if there is an incompatibility that breaks the interface, and in the presence of "wild west" packages there will be, then it will manifest as a compile time build error instead of proceeding.
sjy 12y ago
Not only is the package management problem more difficult, it's not the problem that Cabal (the Common Architecture for Building Applications and Libraries) was intended to solve! cabal-install is not a package manager; it can't uninstall packages or even record which packages it has installed.

bmurphy1976 12y ago

Let's not forget they are all garbage from somebody's perspective. I know Python. Would I prefer to write my package manifests in some bastardized Ruby DSL or Python? Of course I'd prefer Python and the Ruby guys would prefer their bastardized DSL. I don't blame them. How do we bridge that gap? Python => Ruby => Python is bad enough. What about M4 or AWK or Makefiles or Haskell or god help us all some shitty custom format ala Puppet? You might as well demand everybody on Earth speak English.

lucian1900 12y ago
> You might as well demand everybody on Earth speak English. That's working out quite well so far.
- lutusp 12y ago
  >> You might as well demand everybody on Earth speak English. > That's working out quite well so far. Not really. English is a minority language, spoken as a primary language by only about 5.5% of the world's population. http://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers http://en.wikipedia.org/wiki/List_of_languages_by_number_of_... One more juicy anecdote: there are more Chinese people learning English right now, than there are English speakers in the U.S..
  - jessaustin 12y ago
    Your postscript seems to confirm GP's point. b^)
    - dllthomas 12y ago
      I don't know that that's a sign of it working well...
    - lutusp 12y ago
      Not if the students' plan is only to remain competitive with native English speakers. Knowing how to speak English as a second language, for pragmatic reasons, isn't the same as being an English-speaker in the sense of the parent post.

endgame 12y ago

I think a lot of it is that people are scared of automake, so they reinvent their own build systems, with varying degrees of badness. Then they claim that theirs is "easy" but they've only done the first half of "simple things should be easy, hard things should be possible". A properly autotooled package offers an amazing level of control about where to put things.