6 ms·

Could someone ELI7 why python is struggling so hard with dependencies and dependency management when in the Java world this is a solved problem and just works w

Could someone ELI7 why python is struggling so hard with dependencies and dependency management when in the Java world this is a solved problem and just works with maven, which is working so damn well it last got updated in November 2019, and the biggest fight going is whether you should use gradle as client instead of maven to access the same dependency ecosystem? I had to set up Python projects for some machine learning classes in college and it was a complete mess.

mqus 6y ago
I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python. C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy). Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.
- nicoburns 6y ago
  Python should just do what JavaScript does and install all dependencies to a project-local folder.
  - StreamBright 6y ago
    Not sure if JavaScript has anything that any other language or development tool should copy.
    - wilsonrocks 6y ago
      It is a mess... But you can often get a project running an awful lot quicker than in python
      - notdonspaulding 6y ago
        True, but if that's the metric we're basing our language choice off of then we should all go back to PHP.
        PalzyPunk 6y ago
        Yeah no kidding. I prefer it as a prototyping language. Unfortunately, I work in networking, and it's become the raison d'etre for our "DevOps". At least we're not using Ansible anymore... :|
    - nicoburns 6y ago
      JavaScript's package management is significantly better than Pythons: - There's one tool to use, and it comes pre-installed - You don't have to deal with virtualenvs or packages from different projects conflicting. - Native code dependencies "just work".
      - dboreham 6y ago
        That one tool must be yarnnpmgypbabelwebpack?
        nicoburns 6y ago
        Babel and Webpack aren't package management tools. I guess there is now yarn in addition to NPM. But NPM works just fine. And furthermore they interoperate.
        beckingz 6y ago
        which gets defined in Yet Another YAML Language
  - HelloNurse 6y ago
    You are assuming there is a "project", but Python code with massive dependencies can (and often should) be a bare script in an arbitrary folder that runs by invoking the Python interpreter on it, without "installing" anything except missing dependencies.
    - hctaw 6y ago
      That's not a bare script then, it's a project with dependencies that need to be installed. This misunderstanding is how you fuck up a developer environment.
  - IgorPartola 6y ago
    Almost nothing should do what JavaScript does, not even JavaScript. Of all the interpreted languages I’ve used, JS has the single worst ecosystem for running production code. If your experience is primarily with JS, do yourself a favor and try to grok other languages and ecosystems.
  - StavrosK 6y ago
    It does: https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/ Though, as the sibling comments say, it's debatable whether it should.
  - BiteCode_dev 6y ago
    That's literally what the article is about.
  - hctaw 6y ago
    I'm surprised this is getting downvoted. This is the only sane way to manage dependencies. They must be relative to the thing you care about. There are tradeoffs for sure, like the ridiculous sizes of those directories on development machines. But a giant SSD is cheaper than my time resolving dependency issues on user machines.
    - paulie_a 6y ago
      They essentially are. Not in the actual directory but in the virtual env and not cluttering up your project directory. This is superior to JavaScript in my opinion No language should replicate anything JavaScript does when it comes to package management
- pjfin123 6y ago
  This video really helped me understand Python packaging of binaries: https://youtu.be/02aAZ8u3wEQ https://youtu.be/02aAZ8u3wEQ
- BerislavLopac 6y ago
  Or to be precise, it is a solved issue -- several times over. After years of meandering, I think it is safe to say that deployment of Python libraries is pretty much resolved in one, general way: we have the wheel/PyPI/pip toolchain, and that's about it. On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from the underlying system in a universal way. There are different methods, from virtual environments (both standard and nonstandard ones like conda), to version managers like pyenv and asdf, to "jailers" like pipx and pyinstaller, to containers; but none of them has yet clearly won.
  - isignal 6y ago
    All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. End users cannot set up virtual envs and pip install stuff into them (which sometimes needs external tools like make). The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.
    - curryst 6y ago
      I've used pyinstaller. My experience was that pure Python was easy; things that needed to use C shared objects increased in complexity exponentially based on the number of C dependencies. If it's pure Python, you can basically run "pyinstaller <spec file>" and be done. I tried to make PyQt5 work in it, and it was extremely painful. I don't think I ever actually made it work, I think I just gave up. That being said, PyQt5 is probably the most complicated thing you'd ever package with pyinstaller, so it may not be a great test. > All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.
StreamBright 6y ago
Java is a self hosted language and has very little dependency on C/C++ or something similar. Python is glue language for C/C++/Fortran/Rust/etc. projects. Most of the valuable packages in Python in fact C++ projects (like Tensorflow, Pandas, etc.). When you are talking about Python dependency management you are talking about the Cartesian product of the packages you are depending on, their dependency in terms of compilers and header files. It is not hard to run into missing .h files while trying to install these dependencies. This situation was improved significantly a while back with the WHL files. Anyways this whole article does not make too much sense, I am using Python for 10 years professionally and never run into the concept of nested venvs let alone somebody trying to use that in production.
- cdavid 6y ago
  Compiled dependencies is indeed one of the main reasons. Another one is lack of enforced, static metadata because python started before most other languages.
- nicoburns 6y ago
  > It is not hard to run into missing .h files while trying to install these dependencies. They would be if python provided the .h files in the packages rather than expecting them to already exist on the system.
  - rini17 6y ago
    That's just dangerous band-aid. If there is a mismatch between bundled .h and installed system library, you'll get (in better case) linker error, or runtime problems.
    - nicoburns 6y ago
      Well I was assuming the actual library would be bundled too: not just the headers!
      - wiredfool 6y ago
PartiallyTyped 6y ago
I am probably in the minority, but my experience is the complete opposite. I had more trouble trying to set up a gradle project in university than I ever had with Python.
- loudtieblahblah 6y ago
  Gradle is a nightmare.
- tehbeard 6y ago
  Probably not much of a minority, as gradle's design seems much more friendly to footguns than maven.
suifbwish 6y ago
My secret is to build a new computer every year and keep images of the old ones with all my packages and libraries. Trying to run a tensorflow GitHub library from 2013? Easy just use the computer with that image
maliker 6y ago
My ELI7 understanding is that python installs required modules (libraries) at the system level by default, and doesn't have good support for installing modules into local directories like node/C/java/etc. does, which leads to conflicts. Users have to find a set of modules that support all of the scripts on their system. The current workaround is to isolate each script's requirements, but there are a ton of competing tools to do this (virtualenv, pipenv, poetry, etc.), so it's hard to choose the "best" isolation solution. One reason people like PEP 582 is that, by being something built into the interpreter, it will reduce a lot of the confusion over which isolation tool to use. Part of the problem, which local installs won't completely solve, is that the python interpreter is still changing fast enough that modules are having trouble keeping up. E.g. tensorflow is still working on python3.9 support [1], and now we'll see python3.10 in the next couple weeks. Even with local installs we'll have to be careful to use the right interpreter. Another piece of the problem, which is also kind of a good thing, is the huge size of the python module ecosystem. I've got python scripts with over a hundred requirements. It saves me from writing a lot of code, but resolving the requirements is a challenge. The pip package manager has recently added a dependency resolver, which has helped a little make sure everything can work together. The only surefire ways I can think of to make this huge module ecosystem interoperate cleanly are (1) have every module support every other version of every other module and interpreter (insane amount of work, won't happen) or (2) have simpler interop interfaces like the unix shell convention of only passing strings between programs, but that would take away a lot of the ease-of-use of passing rich objects around between modules. [1] https://github.com/tensorflow/tensorflow/issues/44485 https://github.com/tensorflow/tensorflow/issues/44485
BiteCode_dev 6y ago
First, so many tutorials give you terrible information. People don't know about "-m", the "py" command, are told to use sudo and not --user. The current situation is actually not that bad. With my students, the topic is quickly explained and mastered. But it's not a common occurrence. However, it is true that it's still not as easy as it could be. Some reasons. - Python is older than java. - Python is shipped as part of MacOS and Linux, but using the system Python can cause issues, so it's a trap. - Python deals with a lot of compiled extensions. Scipy embeds C, fortran and assembly. It's a hard problem. - The Python project has *tremendously* less resources than java. - Politics. Lots of it. - Debian et Red Had packagers made a mess by splitting the python setup in many small packages. - Python 2/3 transition took attention span from the problem during a decade. - You often install many python versions on the same machine, which makes things harder. - Python is used by a huge number of non programmers. Much more than java. This shows in the reported difficulties. - There are way too many ways to install python. So many combinations of failure modes. - Microsoft screw up with their store not packaging "py" or the core dev by not packaging it on Unix. - Anaconda brings as many problems as it solves, and is popular because of enterprise limitations.
- Spiritus 6y ago
  The “py” command?
  - kcon 6y ago
    I think that is referring to the Python launcher utility for Windows: https://docs.python.org/3/using/windows.html#python-launcher-for-windows https://docs.python.org/3/using/windows.html#python-launcher...
ingas 6y ago
> I had to set up Python projects for some machine learning classes in college and it was a complete mess. "Machine learning" here is crucial I suppose. Sklearn, keras are big beasts. Would you install something like django/flask site than I don't think you would had any problems.
peterthehacker 6y ago
> I had to set up Python projects for some machine learning classes in college and it was a complete mess. Did you use conda (anaconda/miniconda) or pip?
collyw 6y ago
We should have stuck with Perl. I never came across the same problems with CPAN.
- 0xbadcafebee 6y ago
  CPAN I think only recently got virtualenv-like support; before it was much worse and a pain. Where CPAN modules shine are they're more stable, backwards compatible, and extensible. Modules whose last release was 15 years ago work fine today, and new modules are extended on them.
Dryondristica 6y ago
People always criticise Python for dependency management, but I've never found it to be that bad. virtualenv with requirements.txt has been completely fine in my experience with medium-sized projects. If necessary you can pin versions with a separate version-locked requirements.txt, and populate that from a base requirements file.
- mejutoco 6y ago
  Came here to say exactly this. This has been my experience as well
paulie_a 6y ago
I cannot fathom where python is struggling is with that. Pip and a requirements file that can be pinned with version numbers It's super easy, barely an inconvenience Gradle is a hot mess and JavaScript is the just a complete shit show
N1H1L 6y ago
Tell me about it. I had cudatoolkit installed, and then followed every single PyTorch installation instructions, and then still it insisted that I should get the CPU version, not the GPU one. I ultimately fixed this with going to the conda files list, and choosing exactly what I needed. Not straightforward at all.

mqus 6y ago

I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python. C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy). Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.

nicoburns 6y ago
Python should just do what JavaScript does and install all dependencies to a project-local folder.
- StreamBright 6y ago
  Not sure if JavaScript has anything that any other language or development tool should copy.
  - wilsonrocks 6y ago
    It is a mess... But you can often get a project running an awful lot quicker than in python
    - notdonspaulding 6y ago
      True, but if that's the metric we're basing our language choice off of then we should all go back to PHP.
      - PalzyPunk 6y ago
        Yeah no kidding. I prefer it as a prototyping language. Unfortunately, I work in networking, and it's become the raison d'etre for our "DevOps". At least we're not using Ansible anymore... :|
  - nicoburns 6y ago
    JavaScript's package management is significantly better than Pythons: - There's one tool to use, and it comes pre-installed - You don't have to deal with virtualenvs or packages from different projects conflicting. - Native code dependencies "just work".
    - dboreham 6y ago
      That one tool must be yarnnpmgypbabelwebpack?
      - nicoburns 6y ago
        Babel and Webpack aren't package management tools. I guess there is now yarn in addition to NPM. But NPM works just fine. And furthermore they interoperate.
      - beckingz 6y ago
        which gets defined in Yet Another YAML Language
- HelloNurse 6y ago
  You are assuming there is a "project", but Python code with massive dependencies can (and often should) be a bare script in an arbitrary folder that runs by invoking the Python interpreter on it, without "installing" anything except missing dependencies.
  - hctaw 6y ago
    That's not a bare script then, it's a project with dependencies that need to be installed. This misunderstanding is how you fuck up a developer environment.
- IgorPartola 6y ago
  Almost nothing should do what JavaScript does, not even JavaScript. Of all the interpreted languages I’ve used, JS has the single worst ecosystem for running production code. If your experience is primarily with JS, do yourself a favor and try to grok other languages and ecosystems.
- StavrosK 6y ago
  It does: https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/ Though, as the sibling comments say, it's debatable whether it should.
- BiteCode_dev 6y ago
  That's literally what the article is about.
- hctaw 6y ago
  I'm surprised this is getting downvoted. This is the only sane way to manage dependencies. They must be relative to the thing you care about. There are tradeoffs for sure, like the ridiculous sizes of those directories on development machines. But a giant SSD is cheaper than my time resolving dependency issues on user machines.
  - paulie_a 6y ago
    They essentially are. Not in the actual directory but in the virtual env and not cluttering up your project directory. This is superior to JavaScript in my opinion No language should replicate anything JavaScript does when it comes to package management
pjfin123 6y ago
This video really helped me understand Python packaging of binaries: https://youtu.be/02aAZ8u3wEQ https://youtu.be/02aAZ8u3wEQ
BerislavLopac 6y ago
Or to be precise, it is a solved issue -- several times over. After years of meandering, I think it is safe to say that deployment of Python libraries is pretty much resolved in one, general way: we have the wheel/PyPI/pip toolchain, and that's about it. On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from the underlying system in a universal way. There are different methods, from virtual environments (both standard and nonstandard ones like conda), to version managers like pyenv and asdf, to "jailers" like pipx and pyinstaller, to containers; but none of them has yet clearly won.
- isignal 6y ago
  All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. End users cannot set up virtual envs and pip install stuff into them (which sometimes needs external tools like make). The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.
  - curryst 6y ago
    I've used pyinstaller. My experience was that pure Python was easy; things that needed to use C shared objects increased in complexity exponentially based on the number of C dependencies. If it's pure Python, you can basically run "pyinstaller <spec file>" and be done. I tried to make PyQt5 work in it, and it was extremely painful. I don't think I ever actually made it work, I think I just gave up. That being said, PyQt5 is probably the most complicated thing you'd ever package with pyinstaller, so it may not be a great test. > All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.

StreamBright 6y ago

Java is a self hosted language and has very little dependency on C/C++ or something similar. Python is glue language for C/C++/Fortran/Rust/etc. projects. Most of the valuable packages in Python in fact C++ projects (like Tensorflow, Pandas, etc.). When you are talking about Python dependency management you are talking about the Cartesian product of the packages you are depending on, their dependency in terms of compilers and header files. It is not hard to run into missing .h files while trying to install these dependencies. This situation was improved significantly a while back with the WHL files. Anyways this whole article does not make too much sense, I am using Python for 10 years professionally and never run into the concept of nested venvs let alone somebody trying to use that in production.

cdavid 6y ago
Compiled dependencies is indeed one of the main reasons. Another one is lack of enforced, static metadata because python started before most other languages.
nicoburns 6y ago
> It is not hard to run into missing .h files while trying to install these dependencies. They would be if python provided the .h files in the packages rather than expecting them to already exist on the system.
- rini17 6y ago
  That's just dangerous band-aid. If there is a mismatch between bundled .h and installed system library, you'll get (in better case) linker error, or runtime problems.
  - nicoburns 6y ago
    Well I was assuming the actual library would be bundled too: not just the headers!
    - wiredfool 6y ago

PartiallyTyped 6y ago

I am probably in the minority, but my experience is the complete opposite. I had more trouble trying to set up a gradle project in university than I ever had with Python.

loudtieblahblah 6y ago
Gradle is a nightmare.
tehbeard 6y ago
Probably not much of a minority, as gradle's design seems much more friendly to footguns than maven.

suifbwish 6y ago

My secret is to build a new computer every year and keep images of the old ones with all my packages and libraries. Trying to run a tensorflow GitHub library from 2013? Easy just use the computer with that image

maliker 6y ago

My ELI7 understanding is that python installs required modules (libraries) at the system level by default, and doesn't have good support for installing modules into local directories like node/C/java/etc. does, which leads to conflicts. Users have to find a set of modules that support all of the scripts on their system. The current workaround is to isolate each script's requirements, but there are a ton of competing tools to do this (virtualenv, pipenv, poetry, etc.), so it's hard to choose the "best" isolation solution. One reason people like PEP 582 is that, by being something built into the interpreter, it will reduce a lot of the confusion over which isolation tool to use. Part of the problem, which local installs won't completely solve, is that the python interpreter is still changing fast enough that modules are having trouble keeping up. E.g. tensorflow is still working on python3.9 support [1], and now we'll see python3.10 in the next couple weeks. Even with local installs we'll have to be careful to use the right interpreter. Another piece of the problem, which is also kind of a good thing, is the huge size of the python module ecosystem. I've got python scripts with over a hundred requirements. It saves me from writing a lot of code, but resolving the requirements is a challenge. The pip package manager has recently added a dependency resolver, which has helped a little make sure everything can work together. The only surefire ways I can think of to make this huge module ecosystem interoperate cleanly are (1) have every module support every other version of every other module and interpreter (insane amount of work, won't happen) or (2) have simpler interop interfaces like the unix shell convention of only passing strings between programs, but that would take away a lot of the ease-of-use of passing rich objects around between modules. [1] https://github.com/tensorflow/tensorflow/issues/44485 https://github.com/tensorflow/tensorflow/issues/44485

BiteCode_dev 6y ago

First, so many tutorials give you terrible information. People don't know about "-m", the "py" command, are told to use sudo and not --user. The current situation is actually not that bad. With my students, the topic is quickly explained and mastered. But it's not a common occurrence. However, it is true that it's still not as easy as it could be. Some reasons. - Python is older than java. - Python is shipped as part of MacOS and Linux, but using the system Python can cause issues, so it's a trap. - Python deals with a lot of compiled extensions. Scipy embeds C, fortran and assembly. It's a hard problem. - The Python project has *tremendously* less resources than java. - Politics. Lots of it. - Debian et Red Had packagers made a mess by splitting the python setup in many small packages. - Python 2/3 transition took attention span from the problem during a decade. - You often install many python versions on the same machine, which makes things harder. - Python is used by a huge number of non programmers. Much more than java. This shows in the reported difficulties. - There are way too many ways to install python. So many combinations of failure modes. - Microsoft screw up with their store not packaging "py" or the core dev by not packaging it on Unix. - Anaconda brings as many problems as it solves, and is popular because of enterprise limitations.

Spiritus 6y ago
The “py” command?
- kcon 6y ago
  I think that is referring to the Python launcher utility for Windows: https://docs.python.org/3/using/windows.html#python-launcher-for-windows https://docs.python.org/3/using/windows.html#python-launcher...

ingas 6y ago

> I had to set up Python projects for some machine learning classes in college and it was a complete mess. "Machine learning" here is crucial I suppose. Sklearn, keras are big beasts. Would you install something like django/flask site than I don't think you would had any problems.

peterthehacker 6y ago

> I had to set up Python projects for some machine learning classes in college and it was a complete mess. Did you use conda (anaconda/miniconda) or pip?

collyw 6y ago

We should have stuck with Perl. I never came across the same problems with CPAN.

0xbadcafebee 6y ago
CPAN I think only recently got virtualenv-like support; before it was much worse and a pain. Where CPAN modules shine are they're more stable, backwards compatible, and extensible. Modules whose last release was 15 years ago work fine today, and new modules are extended on them.

Dryondristica 6y ago

People always criticise Python for dependency management, but I've never found it to be that bad. virtualenv with requirements.txt has been completely fine in my experience with medium-sized projects. If necessary you can pin versions with a separate version-locked requirements.txt, and populate that from a base requirements file.

mejutoco 6y ago
Came here to say exactly this. This has been my experience as well

paulie_a 6y ago

I cannot fathom where python is struggling is with that. Pip and a requirements file that can be pinned with version numbers It's super easy, barely an inconvenience Gradle is a hot mess and JavaScript is the just a complete shit show

N1H1L 6y ago

Tell me about it. I had cudatoolkit installed, and then followed every single PyTorch installation instructions, and then still it insisted that I should get the CPU version, not the GPU one. I ultimately fixed this with going to the conda files list, and choosing exactly what I needed. Not straightforward at all.