7 ms·

Nice, but can anything challenge the "every language gets its own unique package manager"? (And own unique build system for that matter).

andrewl-hn 12y ago
James Halliday a while ago wrote a package system for C based on npm - a node package system: https://github.com/substack/dotc#dotc https://github.com/substack/dotc#dotc The machinery behind it is surprisingly simple. You have 2 pre-processor commands: #require and #exports and you use them to build up a hierarchy of dependencies.
- e12e 12y ago
  Interesting. Not sure how I feel about the requirment to use c++ features for a c package manager though...
cleverjake 12y ago
its a pain, but realistically, the needs of each community are often best served by that community.
yxhuvud 12y ago
While it would be very nice if one of the existing language package managers stepped up their game by supporting multiple languages and cross-language dependencies, one still has to remember that these fulfill a need - namely to make it easy to share code within a coding community. This is something distro package systems have failed to achieve - their systems have too much bureucratic overhead to make a package available for all and require that a library to be packaged multiple times to be used in more than one distro.
- zimbatm 12y ago
  It's because we need a user-space package manager. System package managers require root access to be handled and tend to provide only a single stable version of each package. While they are good to provide the base system, on the application level it's desirable to use specific version of libraries.
  - Hello71 12y ago
    > It's because we need a user-space package manager. that word doesn't mean what you think it means > System package managers require root access to be handled nope, Gentoo Prefix can be installed by anyone, as long as you have somewhere that's u+w and exec. > and tend to provide only a single stable version of each package. nope, Gentoo allows any versions of every package to be mixed and matched, as long as the underlying platform and package supports it.
    - zimbatm 12y ago
      Is Gentoo Prefix installable on other platforms ? If you want to match the language-specific package managers you also have to be able to support at least the common POSIX platforms and allow per-project dependencies management.
  - Svenstaro 12y ago
    Shouldn't software optimally always use the newest version of all the libraries it depends on? It's a security issue if they don't. And as far as I know, only Linux distro package repositories even have backports, not application package repos.
    - zimbatm 12y ago
      There's value in having automatic system updates for the common base. Once you go on the road of unsupported libraries that would be bundled with your application more work is involved for upgrades. Even with semantic versioning you depend on the author to do it properly. It's best if upgrades can be controlled to avoid regressions.
    - e12e 12y ago
      > Shouldn't software optimally always use the newest version of all the libraries it depends on? It's a security issue if they don't. Not necessarily. There's great value in having stable "long time support" versions of libraries, that are not the latest version, but often have backported security fixes. New features introduce new bugs, some (most?) new bugs will be (new) security issues. [ed: For a new application, tracking upstream is often the best way -- say you assume to have a stable(ish) release of your application in 6 months, you don't want to miss out on new features that'll be available in a supported release of some library you're using. But it doesn't follow that you should always migrate to the latest release of that library.]
mitchty 12y ago
I honestly find pkg-config to be the most sane "c package manager" As long as a ./configure && make && make install installs package config files all I need to do is set PACKAGE_CONFIG_PATH and be done with it. I don't think c needs its own package manager, or at least another one.
- jevinskie 12y ago
  I really like pkg-config but it has some issues with static linking. I had to create a GCC wrapper script to properly build and link the libimobiledevice suite into 100% static executables. pkg-config config files have fields to handle transitive shared linking but not one for static linking. P.S. Apologies for the accidental downvote. I compensated by up voting another comment.
  - mitchty 12y ago
    Yeah I love pkg-config and static linking is annoying. Generally what I do to get around that is in configure.ac something like: libuvdir=`pkg-config --variable=libdir libuv` LIBS="$LIBS $libuvdir/libuv.a" That tends to work well enough for my needs. I gate it with a --build-static flag so that I can run off shared if I want to or not. I'd rather take pkg-config+autotools over any of these newfangled options. Going to take a lot to sway me over from a tried and true option across osx/linux/solaris/eh fine hpux too but just cause i'm in a good mood.
- deleted 12y ago
  [deleted]
patmcguire 12y ago
It's not too hard to make a meta-manager, is it? Wrap the ten or so people use in a script, have dependencies like "clib": { // whatever }, "gem": { // whatever } So you could do something like install GSL in the C package and then Ruby GSL in your gems. Saves the trouble if installing things to bind to, lets the things manage what they manage best. Install for the meta-manager would install the relevant package managers, then you've got it all.
- StefanKarpinski 12y ago
  I think you significantly underestimate how difficult a problem correctly installing and configuring software is. Even just specifying and resolving the dependencies and relationships between libraries and packages is quite hard.
  - chubot 12y ago
    One thing I didn't realize until recently is that it's not just "hard" engineering-wise, it's hard theoretically, i.e. NP-hard. With version constraints, package managers actually have to solve a problem equivalent to SAT solving. All the "practical" package managers rely on relatively terrifying hacks and heuristics to get around this. I have been looking through Debian metadata, and I am very surprised at how unsound it is. The metadata is very tightly coupled to apt-get ("virtual" package handling, etc.) Not to mention that all the algorithms basically only work in one direction (going forward, not backward). It appears that some academic/niche package managers are incorporating SAT solvers. "Managing the complexity of large free and open source package-based software distributions" http://scholar.google.com/scholar?cluster=16249102547364024825&hl=en&as_sdt=0,5 http://scholar.google.com/scholar?cluster=162491025473640248... More references: http://www.mancoosi.org/edos/references/ http://www.mancoosi.org/edos/references/ Anyway the root problem is that C headers and other language-specific interfaces are not always a good way to interface software components... developing more stable binary protocols would greatly ameliorate the versioning problem.
    - StefanKarpinski 12y ago
StefanKarpinski 12y ago
There is a natural break between system-level package management and language-level package management. If you install language-level packages with system-level package managers, then each language package needs to support every single system-level package manager. E.g. you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc. This is a completely ridiculous situation to put language developers in, especially since most language-level packages don't care what system they run on. Instead, the best practice seems to be to install programming languages with the system-level package manager and install the language's packages with the language-level package manager. That way you only need one system-level package per language, and each language-level package only needs to deal with that language's package manager. The system package manager is responsible for dealing with system-specific things while the language package manager is responsible for language-specific things. It's a natural division. It is conceivable that there could be a single cross-language, language-level package manager. But what language would it be written in? And how would it deal with the fact that each language uses different technologies and has different approaches for installing and configuring libraries and packages? When you think about it for a bit, I don't think it's so insensible for each language to have its own package manager.
- Hello71 12y ago
  > you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc wrong. that's the package maintainer's job. moreover, sensible package management systems have templates or libraries (eclasses in Gentoo) to properly handle all kinds of packages. all we ask for is that developers don't actively break package management by bundling everything as is done here.
  - voltagex_ 12y ago
    Ir seems like there is often a breakdown in communication between upstream and distro package maintainers. It may also be that authors don't have time to talk with ~3 (Debian, Fedora, ???) distros, each with different packaging processes and rules.
  - drewcrawford 12y ago
mmerickel 12y ago
nix (http://nixos.org/nix/ http://nixos.org/nix/) seems to be the solution with the most potential in this area, solving all dependencies at the project level and removing the need for language specific build environments.
- Nullabillity 12y ago
  Also, per-user installs sweetens the deal even further.
- pmahoney 12y ago
  I've actually been exploring nix myself recently. It has some interesting features. I'm finding it quite nice to maintain some C-based packages across both Linux and Mac OS X (where it is somewhat of a competitor to Homebrew). I've been trying to get it to manage some Ruby applications, complete with gems, private github gems, etc. This has been difficult, in part because some areas of nix are a bit rough, and in part because Rubygems has incomplete dependency information (i.e. there's no standard way to know that, say typhoeus gem depends on C library libcurl, and I'm still unclear on how/if dynamic loading can be made to load the nix packaged libcurl). Something like nix is what I had in mind with my original question. I'd like to see new languages at least design their custom package managers with these more generic systems in mind. Rubygems has some quirks, and Bundler just includes everything and the kitchen sink (sourcing gems from local files, github, various other features) that complicate matters. Nix is good for keeping things more or less isolated and tracking complete dependencies, though an alternate way to accomplish similar is something like Docker and then just scripting all the native package managers...

andrewl-hn 12y ago

James Halliday a while ago wrote a package system for C based on npm - a node package system: https://github.com/substack/dotc#dotc https://github.com/substack/dotc#dotc The machinery behind it is surprisingly simple. You have 2 pre-processor commands: #require and #exports and you use them to build up a hierarchy of dependencies.

e12e 12y ago
Interesting. Not sure how I feel about the requirment to use c++ features for a c package manager though...

cleverjake 12y ago

its a pain, but realistically, the needs of each community are often best served by that community.

yxhuvud 12y ago

While it would be very nice if one of the existing language package managers stepped up their game by supporting multiple languages and cross-language dependencies, one still has to remember that these fulfill a need - namely to make it easy to share code within a coding community. This is something distro package systems have failed to achieve - their systems have too much bureucratic overhead to make a package available for all and require that a library to be packaged multiple times to be used in more than one distro.

zimbatm 12y ago
It's because we need a user-space package manager. System package managers require root access to be handled and tend to provide only a single stable version of each package. While they are good to provide the base system, on the application level it's desirable to use specific version of libraries.
- Hello71 12y ago
  > It's because we need a user-space package manager. that word doesn't mean what you think it means > System package managers require root access to be handled nope, Gentoo Prefix can be installed by anyone, as long as you have somewhere that's u+w and exec. > and tend to provide only a single stable version of each package. nope, Gentoo allows any versions of every package to be mixed and matched, as long as the underlying platform and package supports it.
  - zimbatm 12y ago
    Is Gentoo Prefix installable on other platforms ? If you want to match the language-specific package managers you also have to be able to support at least the common POSIX platforms and allow per-project dependencies management.
- Svenstaro 12y ago
  Shouldn't software optimally always use the newest version of all the libraries it depends on? It's a security issue if they don't. And as far as I know, only Linux distro package repositories even have backports, not application package repos.
  - zimbatm 12y ago
    There's value in having automatic system updates for the common base. Once you go on the road of unsupported libraries that would be bundled with your application more work is involved for upgrades. Even with semantic versioning you depend on the author to do it properly. It's best if upgrades can be controlled to avoid regressions.
  - e12e 12y ago
    > Shouldn't software optimally always use the newest version of all the libraries it depends on? It's a security issue if they don't. Not necessarily. There's great value in having stable "long time support" versions of libraries, that are not the latest version, but often have backported security fixes. New features introduce new bugs, some (most?) new bugs will be (new) security issues. [ed: For a new application, tracking upstream is often the best way -- say you assume to have a stable(ish) release of your application in 6 months, you don't want to miss out on new features that'll be available in a supported release of some library you're using. But it doesn't follow that you should always migrate to the latest release of that library.]

mitchty 12y ago

I honestly find pkg-config to be the most sane "c package manager" As long as a ./configure && make && make install installs package config files all I need to do is set PACKAGE_CONFIG_PATH and be done with it. I don't think c needs its own package manager, or at least another one.

jevinskie 12y ago
I really like pkg-config but it has some issues with static linking. I had to create a GCC wrapper script to properly build and link the libimobiledevice suite into 100% static executables. pkg-config config files have fields to handle transitive shared linking but not one for static linking. P.S. Apologies for the accidental downvote. I compensated by up voting another comment.
- mitchty 12y ago
  Yeah I love pkg-config and static linking is annoying. Generally what I do to get around that is in configure.ac something like: libuvdir=`pkg-config --variable=libdir libuv` LIBS="$LIBS $libuvdir/libuv.a" That tends to work well enough for my needs. I gate it with a --build-static flag so that I can run off shared if I want to or not. I'd rather take pkg-config+autotools over any of these newfangled options. Going to take a lot to sway me over from a tried and true option across osx/linux/solaris/eh fine hpux too but just cause i'm in a good mood.
deleted 12y ago
[deleted]

patmcguire 12y ago

It's not too hard to make a meta-manager, is it? Wrap the ten or so people use in a script, have dependencies like "clib": { // whatever }, "gem": { // whatever } So you could do something like install GSL in the C package and then Ruby GSL in your gems. Saves the trouble if installing things to bind to, lets the things manage what they manage best. Install for the meta-manager would install the relevant package managers, then you've got it all.

StefanKarpinski 12y ago
I think you significantly underestimate how difficult a problem correctly installing and configuring software is. Even just specifying and resolving the dependencies and relationships between libraries and packages is quite hard.
- chubot 12y ago
  One thing I didn't realize until recently is that it's not just "hard" engineering-wise, it's hard theoretically, i.e. NP-hard. With version constraints, package managers actually have to solve a problem equivalent to SAT solving. All the "practical" package managers rely on relatively terrifying hacks and heuristics to get around this. I have been looking through Debian metadata, and I am very surprised at how unsound it is. The metadata is very tightly coupled to apt-get ("virtual" package handling, etc.) Not to mention that all the algorithms basically only work in one direction (going forward, not backward). It appears that some academic/niche package managers are incorporating SAT solvers. "Managing the complexity of large free and open source package-based software distributions" http://scholar.google.com/scholar?cluster=16249102547364024825&hl=en&as_sdt=0,5 http://scholar.google.com/scholar?cluster=162491025473640248... More references: http://www.mancoosi.org/edos/references/ http://www.mancoosi.org/edos/references/ Anyway the root problem is that C headers and other language-specific interfaces are not always a good way to interface software components... developing more stable binary protocols would greatly ameliorate the versioning problem.
  - StefanKarpinski 12y ago

StefanKarpinski 12y ago

There is a natural break between system-level package management and language-level package management. If you install language-level packages with system-level package managers, then each language package needs to support every single system-level package manager. E.g. you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc. This is a completely ridiculous situation to put language developers in, especially since most language-level packages don't care what system they run on. Instead, the best practice seems to be to install programming languages with the system-level package manager and install the language's packages with the language-level package manager. That way you only need one system-level package per language, and each language-level package only needs to deal with that language's package manager. The system package manager is responsible for dealing with system-specific things while the language package manager is responsible for language-specific things. It's a natural division. It is conceivable that there could be a single cross-language, language-level package manager. But what language would it be written in? And how would it deal with the fact that each language uses different technologies and has different approaches for installing and configuring libraries and packages? When you think about it for a bit, I don't think it's so insensible for each language to have its own package manager.

Hello71 12y ago
> you write a Ruby package and now you have to support apt-get, yum, Gentoo, Arch, brew, MacPorts, etc wrong. that's the package maintainer's job. moreover, sensible package management systems have templates or libraries (eclasses in Gentoo) to properly handle all kinds of packages. all we ask for is that developers don't actively break package management by bundling everything as is done here.
- voltagex_ 12y ago
  Ir seems like there is often a breakdown in communication between upstream and distro package maintainers. It may also be that authors don't have time to talk with ~3 (Debian, Fedora, ???) distros, each with different packaging processes and rules.
- drewcrawford 12y ago

mmerickel 12y ago

nix (http://nixos.org/nix/ http://nixos.org/nix/) seems to be the solution with the most potential in this area, solving all dependencies at the project level and removing the need for language specific build environments.

Nullabillity 12y ago
Also, per-user installs sweetens the deal even further.
pmahoney 12y ago
I've actually been exploring nix myself recently. It has some interesting features. I'm finding it quite nice to maintain some C-based packages across both Linux and Mac OS X (where it is somewhat of a competitor to Homebrew). I've been trying to get it to manage some Ruby applications, complete with gems, private github gems, etc. This has been difficult, in part because some areas of nix are a bit rough, and in part because Rubygems has incomplete dependency information (i.e. there's no standard way to know that, say typhoeus gem depends on C library libcurl, and I'm still unclear on how/if dynamic loading can be made to load the nix packaged libcurl). Something like nix is what I had in mind with my original question. I'd like to see new languages at least design their custom package managers with these more generic systems in mind. Rubygems has some quirks, and Bundler just includes everything and the kitchen sink (sourcing gems from local files, github, various other features) that complicate matters. Nix is good for keeping things more or less isolated and tracking complete dependencies, though an alternate way to accomplish similar is something like Docker and then just scripting all the native package managers...