5 ms·

I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming

I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. Nobody's done it because any potential solutions are much too brittle. On Debian derivatives, it's not too big a problem, but go outside of that, and even high-level tooling is a mess (the tooling on top of RPM varies from distro to distro, and even version to version). On macOS, if you're using a ports-like system, you might be dealing with fink, MacPorts, brew, pkgsrc, &c. Any competently-managed project, OTOH, will list the dependencies for a number of major distros and OSs so that the person installing it will have an idea what to look for. I'm all ears for a reasonable solution to this, if you have one.

dkersten 10y ago
I really don't know of anything like pip that can handle that case in any language I've never ever had to think about it with the Java ecosystem. I just get a jar and stuff magically works.
- talideon 10y ago
  If everything's native[1], whether you're talking about Python, Java, or any other language, you're golden. The issue arises when you're calling out to something written in, say, C. You'll have the same pain in java if you start using JNI to call across the JVM boundary. It just crops up more in Python because C extensions get used a lot in Python. Addendum: [1] By this, I mean within the boundaries of the runtime environment and not using things like Python's C extension API or JNI. 'Native' is a poor choice of words.
  - crdoconnor 10y ago
    It isn't really a question of native or non-native. It's more a question of "under the control of the package manager" vs. "not under the control". You have a similar kind of pain when you have pip and apt trying to manage the system python installation.
    - talideon 10y ago
      If all you're doing is using pure Python code, there's no issue: you can install what you like, either with the --user flag or into a virtual environment. This situation is nothing like running pip as root.
      - crdoconnor 10y ago
        No, there are issues. If you install packages that have dependencies on libraries which apt-installed packages depend upon then you can break them. This is all about running code in an uncontrolled environment.
        talideon 10y ago
        This is what virtual environments are for, and your solution still uses virtual environments. So no, there aren't. Now, I have a project I'd like to try with per-process mounts of dependencies using unionfs. It'd be a lot of work, and a lot could go wrong, but would solve a lot of problems.
        crdoconnor 10y ago
        >This is what virtual environments are for Precisely. Virtualenvs solve part of the problem by isolating the python code and its dependencies - but not all of the code. Apparently extending this behavior would cause security issues of an unknown nature...
  - dkersten 10y ago
    My point was that I've never had to care if a jar contained C libraries or not. Some do, many don't, but I've never had to care. Maybe I've just been lucky. I suppose it could have been a fat-jar thing: if the jar contains its non-java dependencies for my platform, then everything just works. If the native libs are separate, then sure, its a similar issue as described elsewhere here.
yxhuvud 10y ago
The solutions are only brittle if the different distribution systems doesn't talk with each other. Someone should gather a bunch of different language level package managers and a bunch of distro level package managers in a room and then not let them out until they have figured out a standardised API to declare, provide and require dependencies.
- talideon 10y ago
  And who is that somebody going to be? Also, what you're proposing boils down to asking them to all standardise on RPM. Have you asked yourself why all these different package management tools exist in the first place?
crdoconnor 10y ago
>I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. I actually started probably the only project to actually make a stab at solving this: https://github.com/unixpackage/unixpackage https://github.com/unixpackage/unixpackage However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. Plus I know it would never make it into pip because "it's not pip's job". >I'm all ears for a reasonable solution to this, if you have one. The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. This will balloon the size of virtualenvs and increase installation times but it'll be worth it. Not only will it eliminate those horrible installation errors making everybody happy (not just beginners), it will eliminate a nasty source of brittleness (subtly different behavior caused by different versions of these packages on different platforms).
- talideon 10y ago
  > However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. That, in a nutshell, is the problem. > Plus I know it would never make it into pip because "it's not pip's job". There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip. > The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. The 'horrible hacky code' is autotools 90% of the time, and while I'd like to see if die in a fire and for pkgconf to replace it as far as library detection goes, it's not going away any time soon. Also, there is next to no chance of what you propose happening: you're essentially proposing a parallel system package management system, and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.

dkersten 10y ago

I really don't know of anything like pip that can handle that case in any language I've never ever had to think about it with the Java ecosystem. I just get a jar and stuff magically works.

talideon 10y ago
If everything's native[1], whether you're talking about Python, Java, or any other language, you're golden. The issue arises when you're calling out to something written in, say, C. You'll have the same pain in java if you start using JNI to call across the JVM boundary. It just crops up more in Python because C extensions get used a lot in Python. Addendum: [1] By this, I mean within the boundaries of the runtime environment and not using things like Python's C extension API or JNI. 'Native' is a poor choice of words.
- crdoconnor 10y ago
  It isn't really a question of native or non-native. It's more a question of "under the control of the package manager" vs. "not under the control". You have a similar kind of pain when you have pip and apt trying to manage the system python installation.
  - talideon 10y ago
    If all you're doing is using pure Python code, there's no issue: you can install what you like, either with the --user flag or into a virtual environment. This situation is nothing like running pip as root.
    - crdoconnor 10y ago
      No, there are issues. If you install packages that have dependencies on libraries which apt-installed packages depend upon then you can break them. This is all about running code in an uncontrolled environment.
      - talideon 10y ago
        This is what virtual environments are for, and your solution still uses virtual environments. So no, there aren't. Now, I have a project I'd like to try with per-process mounts of dependencies using unionfs. It'd be a lot of work, and a lot could go wrong, but would solve a lot of problems.
        crdoconnor 10y ago
        >This is what virtual environments are for Precisely. Virtualenvs solve part of the problem by isolating the python code and its dependencies - but not all of the code. Apparently extending this behavior would cause security issues of an unknown nature...
- dkersten 10y ago
  My point was that I've never had to care if a jar contained C libraries or not. Some do, many don't, but I've never had to care. Maybe I've just been lucky. I suppose it could have been a fat-jar thing: if the jar contains its non-java dependencies for my platform, then everything just works. If the native libs are separate, then sure, its a similar issue as described elsewhere here.

yxhuvud 10y ago

The solutions are only brittle if the different distribution systems doesn't talk with each other. Someone should gather a bunch of different language level package managers and a bunch of distro level package managers in a room and then not let them out until they have figured out a standardised API to declare, provide and require dependencies.

talideon 10y ago
And who is that somebody going to be? Also, what you're proposing boils down to asking them to all standardise on RPM. Have you asked yourself why all these different package management tools exist in the first place?

crdoconnor 10y ago

>I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. I actually started probably the only project to actually make a stab at solving this: https://github.com/unixpackage/unixpackage https://github.com/unixpackage/unixpackage However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. Plus I know it would never make it into pip because "it's not pip's job". >I'm all ears for a reasonable solution to this, if you have one. The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. This will balloon the size of virtualenvs and increase installation times but it'll be worth it. Not only will it eliminate those horrible installation errors making everybody happy (not just beginners), it will eliminate a nasty source of brittleness (subtly different behavior caused by different versions of these packages on different platforms).

talideon 10y ago
> However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. That, in a nutshell, is the problem. > Plus I know it would never make it into pip because "it's not pip's job". There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip. > The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. The 'horrible hacky code' is autotools 90% of the time, and while I'd like to see if die in a fire and for pkgconf to replace it as far as library detection goes, it's not going away any time soon. Also, there is next to no chance of what you propose happening: you're essentially proposing a parallel system package management system, and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.