5 ms·

1) is really something that the maintainers of pip should deal with but unfortunately they don't see it as their problem.

talideon 10y ago
I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. Nobody's done it because any potential solutions are much too brittle. On Debian derivatives, it's not too big a problem, but go outside of that, and even high-level tooling is a mess (the tooling on top of RPM varies from distro to distro, and even version to version). On macOS, if you're using a ports-like system, you might be dealing with fink, MacPorts, brew, pkgsrc, &c. Any competently-managed project, OTOH, will list the dependencies for a number of major distros and OSs so that the person installing it will have an idea what to look for. I'm all ears for a reasonable solution to this, if you have one.
- dkersten 10y ago
  I really don't know of anything like pip that can handle that case in any language I've never ever had to think about it with the Java ecosystem. I just get a jar and stuff magically works.
  - talideon 10y ago
    If everything's native[1], whether you're talking about Python, Java, or any other language, you're golden. The issue arises when you're calling out to something written in, say, C. You'll have the same pain in java if you start using JNI to call across the JVM boundary. It just crops up more in Python because C extensions get used a lot in Python. Addendum: [1] By this, I mean within the boundaries of the runtime environment and not using things like Python's C extension API or JNI. 'Native' is a poor choice of words.
    - crdoconnor 10y ago
      It isn't really a question of native or non-native. It's more a question of "under the control of the package manager" vs. "not under the control". You have a similar kind of pain when you have pip and apt trying to manage the system python installation.
      - talideon 10y ago
        If all you're doing is using pure Python code, there's no issue: you can install what you like, either with the --user flag or into a virtual environment. This situation is nothing like running pip as root.
        crdoconnor 10y ago
        No, there are issues. If you install packages that have dependencies on libraries which apt-installed packages depend upon then you can break them. This is all about running code in an uncontrolled environment.
        talideon 10y ago
        This is what virtual environments are for, and your solution still uses virtual environments. So no, there aren't. Now, I have a project I'd like to try with per-process mounts of dependencies using unionfs. It'd be a lot of work, and a lot could go wrong, but would solve a lot of problems.
        crdoconnor 10y ago
        >This is what virtual environments are for Precisely. Virtualenvs solve part of the problem by isolating the python code and its dependencies - but not all of the code. Apparently extending this behavior would cause security issues of an unknown nature...
    - dkersten 10y ago
      My point was that I've never had to care if a jar contained C libraries or not. Some do, many don't, but I've never had to care. Maybe I've just been lucky. I suppose it could have been a fat-jar thing: if the jar contains its non-java dependencies for my platform, then everything just works. If the native libs are separate, then sure, its a similar issue as described elsewhere here.
- yxhuvud 10y ago
  The solutions are only brittle if the different distribution systems doesn't talk with each other. Someone should gather a bunch of different language level package managers and a bunch of distro level package managers in a room and then not let them out until they have figured out a standardised API to declare, provide and require dependencies.
  - talideon 10y ago
    And who is that somebody going to be? Also, what you're proposing boils down to asking them to all standardise on RPM. Have you asked yourself why all these different package management tools exist in the first place?
- crdoconnor 10y ago
  >I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. I actually started probably the only project to actually make a stab at solving this: https://github.com/unixpackage/unixpackage https://github.com/unixpackage/unixpackage However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. Plus I know it would never make it into pip because "it's not pip's job". >I'm all ears for a reasonable solution to this, if you have one. The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. This will balloon the size of virtualenvs and increase installation times but it'll be worth it. Not only will it eliminate those horrible installation errors making everybody happy (not just beginners), it will eliminate a nasty source of brittleness (subtly different behavior caused by different versions of these packages on different platforms).
  - talideon 10y ago
    > However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. That, in a nutshell, is the problem. > Plus I know it would never make it into pip because "it's not pip's job". There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip. > The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. The 'horrible hacky code' is autotools 90% of the time, and while I'd like to see if die in a fire and for pkgconf to replace it as far as library detection goes, it's not going away any time soon. Also, there is next to no chance of what you propose happening: you're essentially proposing a parallel system package management system, and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.
ubernostrum 10y ago
pip's job isn't to specify dependencies, it's to resolve and install them. Specifying the dependencies is the job of setup.py, and that's not at all an unreasonable stance to take.
- crdoconnor 10y ago
  Really? Does it resolve and install libpq-dev or postgresql-libs when you try to install psycopg2? Or even tell you that you need to install them? The problem isn't that this can't be solved in a consistent, clear way. The problem is that nobody views this as their problem to solve.
  - ubernostrum 10y ago
    It also doesn't install Postgres itself, doesn't install a TCP/IP stack or any type of socket abstraction for communicating with a Postgres server, doesn't display an interactive SQL tutorial to ensure I'll know what to do once I get it running, doesn't install an operating system capable of running the psql client, and doesn't go out and buy me a computer or sign up a cloud hosting account with a provisioned database. Would you fault it for those, too? Or would you find a point where you declare "not my problem", and then open yourself up to precisely this same criticism, just with the line drawn in a different place than where it currently is? I'm OK with the fact that the Python package installer installs Python packages. I'm OK with the fact that glancing at the documentation for psycopg immediately tells me up-front in its install instructions the things I'll need. It also helpfully pre-empts the article here, suggesting a final debugging step for a non-functioning install: Complain on your blog or on Twitter that psycopg2 is the worst package ever and about the quality time you have wasted figuring out the correct ARCHFLAGS. Especially useful from the Starbucks near you.
    - crdoconnor 10y ago
      >It also doesn't install Postgres itself Not a dependency. >doesn't install a TCP/IP stack or any type of socket abstraction for communicating with a Postgres server Because that's never caused a problem when installing psycopg2, ever. >doesn't display an interactive SQL tutorial Not a dependency. >doesn't install an operating system capable How exactly would you be running pip without an OS? >Would you fault it for those, too? I think you misunderstand what a dependency actually is.

talideon 10y ago

I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. Nobody's done it because any potential solutions are much too brittle. On Debian derivatives, it's not too big a problem, but go outside of that, and even high-level tooling is a mess (the tooling on top of RPM varies from distro to distro, and even version to version). On macOS, if you're using a ports-like system, you might be dealing with fink, MacPorts, brew, pkgsrc, &c. Any competently-managed project, OTOH, will list the dependencies for a number of major distros and OSs so that the person installing it will have an idea what to look for. I'm all ears for a reasonable solution to this, if you have one.

dkersten 10y ago
I really don't know of anything like pip that can handle that case in any language I've never ever had to think about it with the Java ecosystem. I just get a jar and stuff magically works.
- talideon 10y ago
  If everything's native[1], whether you're talking about Python, Java, or any other language, you're golden. The issue arises when you're calling out to something written in, say, C. You'll have the same pain in java if you start using JNI to call across the JVM boundary. It just crops up more in Python because C extensions get used a lot in Python. Addendum: [1] By this, I mean within the boundaries of the runtime environment and not using things like Python's C extension API or JNI. 'Native' is a poor choice of words.
  - crdoconnor 10y ago
    It isn't really a question of native or non-native. It's more a question of "under the control of the package manager" vs. "not under the control". You have a similar kind of pain when you have pip and apt trying to manage the system python installation.
    - talideon 10y ago
      If all you're doing is using pure Python code, there's no issue: you can install what you like, either with the --user flag or into a virtual environment. This situation is nothing like running pip as root.
      - crdoconnor 10y ago
        No, there are issues. If you install packages that have dependencies on libraries which apt-installed packages depend upon then you can break them. This is all about running code in an uncontrolled environment.
        talideon 10y ago
        This is what virtual environments are for, and your solution still uses virtual environments. So no, there aren't. Now, I have a project I'd like to try with per-process mounts of dependencies using unionfs. It'd be a lot of work, and a lot could go wrong, but would solve a lot of problems.
        crdoconnor 10y ago
        >This is what virtual environments are for Precisely. Virtualenvs solve part of the problem by isolating the python code and its dependencies - but not all of the code. Apparently extending this behavior would cause security issues of an unknown nature...
  - dkersten 10y ago
    My point was that I've never had to care if a jar contained C libraries or not. Some do, many don't, but I've never had to care. Maybe I've just been lucky. I suppose it could have been a fat-jar thing: if the jar contains its non-java dependencies for my platform, then everything just works. If the native libs are separate, then sure, its a similar issue as described elsewhere here.
yxhuvud 10y ago
The solutions are only brittle if the different distribution systems doesn't talk with each other. Someone should gather a bunch of different language level package managers and a bunch of distro level package managers in a room and then not let them out until they have figured out a standardised API to declare, provide and require dependencies.
- talideon 10y ago
  And who is that somebody going to be? Also, what you're proposing boils down to asking them to all standardise on RPM. Have you asked yourself why all these different package management tools exist in the first place?
crdoconnor 10y ago
>I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager. I actually started probably the only project to actually make a stab at solving this: https://github.com/unixpackage/unixpackage https://github.com/unixpackage/unixpackage However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. Plus I know it would never make it into pip because "it's not pip's job". >I'm all ears for a reasonable solution to this, if you have one. The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. This will balloon the size of virtualenvs and increase installation times but it'll be worth it. Not only will it eliminate those horrible installation errors making everybody happy (not just beginners), it will eliminate a nasty source of brittleness (subtly different behavior caused by different versions of these packages on different platforms).
- talideon 10y ago
  > However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. That, in a nutshell, is the problem. > Plus I know it would never make it into pip because "it's not pip's job". There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip. > The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself. The 'horrible hacky code' is autotools 90% of the time, and while I'd like to see if die in a fire and for pkgconf to replace it as far as library detection goes, it's not going away any time soon. Also, there is next to no chance of what you propose happening: you're essentially proposing a parallel system package management system, and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.

ubernostrum 10y ago

pip's job isn't to specify dependencies, it's to resolve and install them. Specifying the dependencies is the job of setup.py, and that's not at all an unreasonable stance to take.

crdoconnor 10y ago
Really? Does it resolve and install libpq-dev or postgresql-libs when you try to install psycopg2? Or even tell you that you need to install them? The problem isn't that this can't be solved in a consistent, clear way. The problem is that nobody views this as their problem to solve.
- ubernostrum 10y ago
  It also doesn't install Postgres itself, doesn't install a TCP/IP stack or any type of socket abstraction for communicating with a Postgres server, doesn't display an interactive SQL tutorial to ensure I'll know what to do once I get it running, doesn't install an operating system capable of running the psql client, and doesn't go out and buy me a computer or sign up a cloud hosting account with a provisioned database. Would you fault it for those, too? Or would you find a point where you declare "not my problem", and then open yourself up to precisely this same criticism, just with the line drawn in a different place than where it currently is? I'm OK with the fact that the Python package installer installs Python packages. I'm OK with the fact that glancing at the documentation for psycopg immediately tells me up-front in its install instructions the things I'll need. It also helpfully pre-empts the article here, suggesting a final debugging step for a non-functioning install: Complain on your blog or on Twitter that psycopg2 is the worst package ever and about the quality time you have wasted figuring out the correct ARCHFLAGS. Especially useful from the Starbucks near you.
  - crdoconnor 10y ago
    >It also doesn't install Postgres itself Not a dependency. >doesn't install a TCP/IP stack or any type of socket abstraction for communicating with a Postgres server Because that's never caused a problem when installing psycopg2, ever. >doesn't display an interactive SQL tutorial Not a dependency. >doesn't install an operating system capable How exactly would you be running pip without an OS? >Would you fault it for those, too? I think you misunderstand what a dependency actually is.