4 ms·

> OS/system package manager (SPM): this is not why we are here today Arguably, this might be already a wrong approach. Whatever you say, but this is a problem

by krick 11y ago

> OS/system package manager (SPM): this is not why we are here today Arguably, this might be already a wrong approach. Whatever you say, but this is a problem (and is causing problems) that I can do both `apt-get install python-mutagen` and `pip install mutagen`. But what I cannot do for sure at this point: get rid of one and only one of these tools. So before speaking about how can I fuck up writing some language-specific PM it might be fair to admit that pretty much nothing actually is language or project-specific when talking about computer systems, and consider why somebody might write another such tool when we're done with our task.

Chris_Newton 11y ago
Yes, I’ve argued before that in a modern software environment both package management and environment separation/virtualisation should be OS responsibilities. As you say, it’s silly that we have OS/distro level functionality, language-specific tools, and sometimes generic but not OS-integrated tools all doing essentially the same job. If we also had some simple, standardised, cross-platform convention for indicating which packages were available and had which dependencies on which platforms, so we could have common repositories that could advertise the specific versions but also the specific supported platforms of each package they hosted, and so any package management tool could speak the same protocol to those repositories, then many things would get better in terms of portability as well. This is an area where the language-specific tools do sometimes have advantages today, though those advantages typically turn to dependency hell almost immediately if you have a package containing a native code element that it decides to build with whatever C(++) compiler it can find on the target system.
sdboyer 11y ago
i agree, this overlap is a problem. and i agree that there's a not-obviously-wrong argument to be made that ignoring SPMs is the wrong approach. to this end, someone pointed me to rubinius' wiki yesterday, which had a nice breakdown of the PM hierarchy: https://github.com/rubinius/mkrb/blob/master/README.md#the-manager-hierarchy https://github.com/rubinius/mkrb/blob/master/README.md#the-m... OTOH: part of the reason that i included the notion of "compiler, phase zero" is because i think that's one thing that, scope-wise, differentiates SPMs from PDMs. an SPM's output is executables. i think this difference is significant. maybe i'm wrong. (If anything, I think there's more overlap between an SPM and an LPM).
aikah 11y ago
The difference is obvious. pip works on all os/platforms. apt-get certainly doesn't. If I write a language and ask users to use their favorite package manager to fetch dependencies, I'm not going to build a big community because packages and their dependencies will not be shared across platforms, or each user will have to add some manifest for every package manager, or different repositories will have to be maintained for each platform ... os package managers and language package managers ARE orthogonal.
- krick 11y ago
  Except they aren't. Not even remotely. The package for some scripting language like python or PHP might have binaries it's depending on, or might have a code written in another language. Pretty much any project-wise language-specific dependency manager can include something that is not actually a library or a bunch of source code, but an utility exposed globally (like unit-testing tool or framework code generator). There are dozens of projects which started out as a domain-specific library (and thus subject for being operated via PDM or LPM only) and soon end up being used as an utility, useful for end-user (because difference between a library and a tool here was a 50 lines of code, exposing some existing methods and which we would like to have anyway for testing). Such tool then is installed via several package managers (and additionally from the source code, because github is far ahead of your PM repository, yeah…) and can cause version incompatibility problems or something even more obscure. And that's just from the top of my head: there can be hundreds of cases which I did not remember right now or even think of, ever. And neither have you. In fact, pretending that these are different things and that "developer" is something drastically different from "user" is precisely the reason, why all (or, well, most) package managers in existance are so fucked up. The hard truth we must accept before we talk about package managers is that there's no hard lines in this domain. It's a complex problem and must be dealt with accordingly, without exceptional reductionism.
tgamblin 11y ago
Spack has, to some degree, a compromise between the language and system package management sides of things. Languages in Spack can specify how their extensions should be activated/deactivated within the installation, and modules like numpy can ‘extend’ the language packages. So, you can plug and play with different python modules. I haven’t looked at how Go does language extensions, but it would be interesting to see if this can be mapped to Go, Lua, etc. [2] http://software.llnl.gov/spack/basic_usage.html#extensions-python-support http://software.llnl.gov/spack/basic_usage.html#extensions-p... [3] http://software.llnl.gov/spack/packaging_guide.html#extensions http://software.llnl.gov/spack/packaging_guide.html#extensio...
yxhuvud 11y ago
While I agree with that it is the wrong approach, the proliferation of language package managers is showing a different case - that the system package manager have taken on too much responsibility and that a better solution rather would be to add support to integrate the language PMs. There really is nothing that would stop apt-get install to use pip under the hood.

by krick 11y ago

Chris_Newton 11y ago

Yes, I’ve argued before that in a modern software environment both package management and environment separation/virtualisation should be OS responsibilities. As you say, it’s silly that we have OS/distro level functionality, language-specific tools, and sometimes generic but not OS-integrated tools all doing essentially the same job. If we also had some simple, standardised, cross-platform convention for indicating which packages were available and had which dependencies on which platforms, so we could have common repositories that could advertise the specific versions but also the specific supported platforms of each package they hosted, and so any package management tool could speak the same protocol to those repositories, then many things would get better in terms of portability as well. This is an area where the language-specific tools do sometimes have advantages today, though those advantages typically turn to dependency hell almost immediately if you have a package containing a native code element that it decides to build with whatever C(++) compiler it can find on the target system.

sdboyer 11y ago

i agree, this overlap is a problem. and i agree that there's a not-obviously-wrong argument to be made that ignoring SPMs is the wrong approach. to this end, someone pointed me to rubinius' wiki yesterday, which had a nice breakdown of the PM hierarchy: https://github.com/rubinius/mkrb/blob/master/README.md#the-manager-hierarchy https://github.com/rubinius/mkrb/blob/master/README.md#the-m... OTOH: part of the reason that i included the notion of "compiler, phase zero" is because i think that's one thing that, scope-wise, differentiates SPMs from PDMs. an SPM's output is executables. i think this difference is significant. maybe i'm wrong. (If anything, I think there's more overlap between an SPM and an LPM).

aikah 11y ago

The difference is obvious. pip works on all os/platforms. apt-get certainly doesn't. If I write a language and ask users to use their favorite package manager to fetch dependencies, I'm not going to build a big community because packages and their dependencies will not be shared across platforms, or each user will have to add some manifest for every package manager, or different repositories will have to be maintained for each platform ... os package managers and language package managers ARE orthogonal.

krick 11y ago
Except they aren't. Not even remotely. The package for some scripting language like python or PHP might have binaries it's depending on, or might have a code written in another language. Pretty much any project-wise language-specific dependency manager can include something that is not actually a library or a bunch of source code, but an utility exposed globally (like unit-testing tool or framework code generator). There are dozens of projects which started out as a domain-specific library (and thus subject for being operated via PDM or LPM only) and soon end up being used as an utility, useful for end-user (because difference between a library and a tool here was a 50 lines of code, exposing some existing methods and which we would like to have anyway for testing). Such tool then is installed via several package managers (and additionally from the source code, because github is far ahead of your PM repository, yeah…) and can cause version incompatibility problems or something even more obscure. And that's just from the top of my head: there can be hundreds of cases which I did not remember right now or even think of, ever. And neither have you. In fact, pretending that these are different things and that "developer" is something drastically different from "user" is precisely the reason, why all (or, well, most) package managers in existance are so fucked up. The hard truth we must accept before we talk about package managers is that there's no hard lines in this domain. It's a complex problem and must be dealt with accordingly, without exceptional reductionism.

tgamblin 11y ago

Spack has, to some degree, a compromise between the language and system package management sides of things. Languages in Spack can specify how their extensions should be activated/deactivated within the installation, and modules like numpy can ‘extend’ the language packages. So, you can plug and play with different python modules. I haven’t looked at how Go does language extensions, but it would be interesting to see if this can be mapped to Go, Lua, etc. [2] http://software.llnl.gov/spack/basic_usage.html#extensions-python-support http://software.llnl.gov/spack/basic_usage.html#extensions-p... [3] http://software.llnl.gov/spack/packaging_guide.html#extensions http://software.llnl.gov/spack/packaging_guide.html#extensio...

yxhuvud 11y ago

While I agree with that it is the wrong approach, the proliferation of language package managers is showing a different case - that the system package manager have taken on too much responsibility and that a better solution rather would be to add support to integrate the language PMs. There really is nothing that would stop apt-get install to use pip under the hood.