4 ms·

Java is a self hosted language and has very little dependency on C/C++ or something similar. Python is glue language for C/C++/Fortran/Rust/etc. projects. Most

Java is a self hosted language and has very little dependency on C/C++ or something similar. Python is glue language for C/C++/Fortran/Rust/etc. projects. Most of the valuable packages in Python in fact C++ projects (like Tensorflow, Pandas, etc.). When you are talking about Python dependency management you are talking about the Cartesian product of the packages you are depending on, their dependency in terms of compilers and header files. It is not hard to run into missing .h files while trying to install these dependencies. This situation was improved significantly a while back with the WHL files. Anyways this whole article does not make too much sense, I am using Python for 10 years professionally and never run into the concept of nested venvs let alone somebody trying to use that in production.

cdavid 6y ago
Compiled dependencies is indeed one of the main reasons. Another one is lack of enforced, static metadata because python started before most other languages.
nicoburns 6y ago
> It is not hard to run into missing .h files while trying to install these dependencies. They would be if python provided the .h files in the packages rather than expecting them to already exist on the system.
- rini17 6y ago
  That's just dangerous band-aid. If there is a mismatch between bundled .h and installed system library, you'll get (in better case) linker error, or runtime problems.
  - nicoburns 6y ago
    Well I was assuming the actual library would be bundled too: not just the headers!
    - wiredfool 6y ago
      At that point, you should just be installing a binary wheel. (Nb I’m a maintainer of a python project that’s difficult to build, and the number of issues we get that start: can’t install because foo.h can’t be found is very high, even though there’s other explanations closer to the end of the failed build)
      - nicoburns 6y ago
        Does that not happen by default when you try to install a package? And if so, why not?
        wiredfool 6y ago
        There are several reasons that people don’t get the binary wheels, but a lot of the time it’s either pinned dependencies where they’re requesting an old version while running 3.9, that window between a new python release coming out and our quarterly release schedule, or an unsupported arch for binaries like Alpine, the m1 macs, random android.
        nicoburns 6y ago
        Ah I see, that makes sense. I believe Node modules typically fall back to building from source in those cases. Which can be a little slow, but generally works reliably.
        wiredfool 6y ago
        I can see that, but we rely on a bunch of system libraries (libjpeg/zlib, + optional libtiff/freetype/etc, and python headers as well). Pretty much by definition, if we don't provide binaries, it's not going to be that simple. I'd say maybe 25-50% of developer linux workstations have the appropriate packages and will compile out of the box, and if they don't, it's almost always a one line command to install them with a package manager. We've got instructions or docker images for testing for most major linux distros. MacOS is probably in the 5-10% range for successfully compiling out of the box, some of the headers ship with MacOS, but there are a couple steps to go through. M1 is still a WIP in the python packaging community. Windows... 0% compile out of the box. You have to set it up, download dependencies and so on. There are at any one time a handful of people who can compile for the windows platform, and a smaller handful of people who can debug problems. Now, to be fair, I think this project is probably in the most complicated 1% of python projects from a packaging point of view. There are probably more complicated ones, and the only two complications I'm aware of that we don't have to deal with are 1) The chicken and egg of if we break an update we lose the ability to update all the other packages including our own (pip) and 2) GPU drivers.
- varjag 6y ago
  That's a terrible idea, all you'd end up with is a bunch of headers mismatched to actual system libraries.

cdavid 6y ago

Compiled dependencies is indeed one of the main reasons. Another one is lack of enforced, static metadata because python started before most other languages.

nicoburns 6y ago

> It is not hard to run into missing .h files while trying to install these dependencies. They would be if python provided the .h files in the packages rather than expecting them to already exist on the system.

rini17 6y ago
That's just dangerous band-aid. If there is a mismatch between bundled .h and installed system library, you'll get (in better case) linker error, or runtime problems.
- nicoburns 6y ago
  Well I was assuming the actual library would be bundled too: not just the headers!
  - wiredfool 6y ago
    At that point, you should just be installing a binary wheel. (Nb I’m a maintainer of a python project that’s difficult to build, and the number of issues we get that start: can’t install because foo.h can’t be found is very high, even though there’s other explanations closer to the end of the failed build)
    - nicoburns 6y ago
      Does that not happen by default when you try to install a package? And if so, why not?
      - wiredfool 6y ago
        There are several reasons that people don’t get the binary wheels, but a lot of the time it’s either pinned dependencies where they’re requesting an old version while running 3.9, that window between a new python release coming out and our quarterly release schedule, or an unsupported arch for binaries like Alpine, the m1 macs, random android.
        nicoburns 6y ago
        Ah I see, that makes sense. I believe Node modules typically fall back to building from source in those cases. Which can be a little slow, but generally works reliably.
        wiredfool 6y ago
        I can see that, but we rely on a bunch of system libraries (libjpeg/zlib, + optional libtiff/freetype/etc, and python headers as well). Pretty much by definition, if we don't provide binaries, it's not going to be that simple. I'd say maybe 25-50% of developer linux workstations have the appropriate packages and will compile out of the box, and if they don't, it's almost always a one line command to install them with a package manager. We've got instructions or docker images for testing for most major linux distros. MacOS is probably in the 5-10% range for successfully compiling out of the box, some of the headers ship with MacOS, but there are a couple steps to go through. M1 is still a WIP in the python packaging community. Windows... 0% compile out of the box. You have to set it up, download dependencies and so on. There are at any one time a handful of people who can compile for the windows platform, and a smaller handful of people who can debug problems. Now, to be fair, I think this project is probably in the most complicated 1% of python projects from a packaging point of view. There are probably more complicated ones, and the only two complications I'm aware of that we don't have to deal with are 1) The chicken and egg of if we break an update we lose the ability to update all the other packages including our own (pip) and 2) GPU drivers.
varjag 6y ago
That's a terrible idea, all you'd end up with is a bunch of headers mismatched to actual system libraries.