4 ms·

It's crazy to me that in 2025 we still haven't figured out python dependency management

by seamossfet 2y ago

kevin_thibedeau 2y ago
Debian had it figured out, then they had to go and ruin it all by hopping on the venv bandwagon.
biorach 2y ago
In fairness, this is a response to a need which became common only relatively recently
fb03 2y ago
We figured, but people can't stand still and just Do Shit (TM). Poetry works fine and solves dependencies and peer dependencies. But I guess the javascript-style churn of 'every week a new tool which is better than the formers' has arrived too in Python land.
- vdvsvwvwvwvwv 2y ago
  Poetry in theory works fine but I found it a pain to use on Windows. As in flakey. A project works, then a dependency can't be found, so I reboot and then it can. The tools to manage and diagnose why it is not installing to the right env are non existant. I am probably not holding it right... but should I need to deep dive rabbit hole into installing dependencies. It is a non issue in NPM, .NET's Nuget, Haskell Stack, Ruby, etc.
VertanaNinjai 2y ago
What do you mean? Just venv your venv.
- pxc 2y ago
  It would be nice if Python could actually support this. But 'venv detection' is conventionally (everywhere) done by a stupid, never-designed involving inspecting argv0. At the same time, most Python packaging tools make various assumptions like "if I'm in a venv, I can write into it", or "there will only ever be one level of venvery, so I'm not in a venv iff I'm looking at the system Python". Nesting/layering venvs has never been supported by any Python dependency management tool except for experiments that were rejected by more senior people in the community, the maintainers of major dependency management tools, etc. Same kind of thing goes for allowing multiple versions of a dependency in the dependency tree at runtime. There's little appetite for fixing Python's dependency management issues in a principled or general way, because that would require breaking things or adding limitations. You can't have dependencies partially managed by arbitrary code that runs only at install time and also build good packaging tools, for instance.
zahlman 2y ago
People want to be able to have a free-wheeling NodeJS-like ecosystem (in a language that shares the whole "`import`s are tracked down at runtime rather than being statically linked" thing, which in turn means you can only practically have one version of the same dependency in a given environment) while also having pieces that are, well, as complex as SciPy, Tensorflow etc. are. And they want to integrate all the missing functionality without anyone ever getting their workflow broken, because everyone still remembers the 2->3 transition. They want this even though the traditional `setup.py`-only way of doing things was abhorrently designed (i.e. you have to run arbitrary code at install time to get any metadata at all - including metadata about build dependencies, the absence of which might cause that code to fail). Of course we haven't figured it out yet.
EatFlamingDeath 2y ago
Dependency management and imports (without a proper namespace solution) make me angry beyond reason. I love and hate Python.
messe 2y ago
Don't worry, there's still a couple of months left until then.
dagw 2y ago
The problem is that people are asking for more and more out of these tools. The tools we had last year are great at solving all the problems we had 20 years ago. But as long as we keep coming up with new problems, we're going to keep needing new tools.
est 2y ago
contrary to popular belief, Python itself doesn't have a dependency problem. The native ABI PyObject Cuda .so .dll shit had wayy to many serious problems. Other lang also had the same problem, think something like cgo or JNI
tengbretson 2y ago
What are you talking about? Everyone just uses OmegaStar.
BiteCode_dev 2y ago
I love the fact that almost every single answer to your comment is a completely different take.
the_biot 2y ago
I didn't realize how bad it was until I recently looked into doing a Python Qt application. There's two competing Python language bindings for Qt: PyQt and PySide. The latter appears more free and active, so that seemed the way to go. They have a compatibility matrix. It's mostly red. https://wiki.qt.io/Qt_for_Python#Python_compatibility_matrix https://wiki.qt.io/Qt_for_Python#Python_compatibility_matrix How do you even make your stuff dependent on/broken with specific Python versions? I mean how in hell? The fact that venv is so widely used in Python was always an indication that not all was well dependency-wise, but it doesn't seem like it's optional anymore.
- twic 2y ago
  My guess would be that because it's an integration with a sprawling native library, there is a lot of code which touches Python's own C API. That is not entirely stable, and it has a lot more source stability than binary stability: https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html As time passes, it makes sense to support only recent versions of both Qt and Python, hence that matrix.
- d0mine 2y ago
  All I see is that newer library versions drop support for old Python versions (seems natural, unless you have unlimited resources). For example, If you need Python 2 support, use corresponding old versions.

It's crazy to me that in 2025 we still haven't figured out python dependency management

kevin_thibedeau 2y ago

Debian had it figured out, then they had to go and ruin it all by hopping on the venv bandwagon.

biorach 2y ago

In fairness, this is a response to a need which became common only relatively recently

fb03 2y ago

We figured, but people can't stand still and just Do Shit (TM). Poetry works fine and solves dependencies and peer dependencies. But I guess the javascript-style churn of 'every week a new tool which is better than the formers' has arrived too in Python land.

vdvsvwvwvwvwv 2y ago
Poetry in theory works fine but I found it a pain to use on Windows. As in flakey. A project works, then a dependency can't be found, so I reboot and then it can. The tools to manage and diagnose why it is not installing to the right env are non existant. I am probably not holding it right... but should I need to deep dive rabbit hole into installing dependencies. It is a non issue in NPM, .NET's Nuget, Haskell Stack, Ruby, etc.

VertanaNinjai 2y ago

What do you mean? Just venv your venv.

pxc 2y ago
It would be nice if Python could actually support this. But 'venv detection' is conventionally (everywhere) done by a stupid, never-designed involving inspecting argv0. At the same time, most Python packaging tools make various assumptions like "if I'm in a venv, I can write into it", or "there will only ever be one level of venvery, so I'm not in a venv iff I'm looking at the system Python". Nesting/layering venvs has never been supported by any Python dependency management tool except for experiments that were rejected by more senior people in the community, the maintainers of major dependency management tools, etc. Same kind of thing goes for allowing multiple versions of a dependency in the dependency tree at runtime. There's little appetite for fixing Python's dependency management issues in a principled or general way, because that would require breaking things or adding limitations. You can't have dependencies partially managed by arbitrary code that runs only at install time and also build good packaging tools, for instance.

zahlman 2y ago

People want to be able to have a free-wheeling NodeJS-like ecosystem (in a language that shares the whole "`import`s are tracked down at runtime rather than being statically linked" thing, which in turn means you can only practically have one version of the same dependency in a given environment) while also having pieces that are, well, as complex as SciPy, Tensorflow etc. are. And they want to integrate all the missing functionality without anyone ever getting their workflow broken, because everyone still remembers the 2->3 transition. They want this even though the traditional `setup.py`-only way of doing things was abhorrently designed (i.e. you have to run arbitrary code at install time to get any metadata at all - including metadata about build dependencies, the absence of which might cause that code to fail). Of course we haven't figured it out yet.

EatFlamingDeath 2y ago

Dependency management and imports (without a proper namespace solution) make me angry beyond reason. I love and hate Python.

messe 2y ago

Don't worry, there's still a couple of months left until then.

dagw 2y ago

The problem is that people are asking for more and more out of these tools. The tools we had last year are great at solving all the problems we had 20 years ago. But as long as we keep coming up with new problems, we're going to keep needing new tools.

est 2y ago

contrary to popular belief, Python itself doesn't have a dependency problem. The native ABI PyObject Cuda .so .dll shit had wayy to many serious problems. Other lang also had the same problem, think something like cgo or JNI

tengbretson 2y ago

What are you talking about? Everyone just uses OmegaStar.

BiteCode_dev 2y ago

I love the fact that almost every single answer to your comment is a completely different take.

the_biot 2y ago

I didn't realize how bad it was until I recently looked into doing a Python Qt application. There's two competing Python language bindings for Qt: PyQt and PySide. The latter appears more free and active, so that seemed the way to go. They have a compatibility matrix. It's mostly red. https://wiki.qt.io/Qt_for_Python#Python_compatibility_matrix https://wiki.qt.io/Qt_for_Python#Python_compatibility_matrix How do you even make your stuff dependent on/broken with specific Python versions? I mean how in hell? The fact that venv is so widely used in Python was always an indication that not all was well dependency-wise, but it doesn't seem like it's optional anymore.

twic 2y ago
My guess would be that because it's an integration with a sprawling native library, there is a lot of code which touches Python's own C API. That is not entirely stable, and it has a lot more source stability than binary stability: https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html As time passes, it makes sense to support only recent versions of both Qt and Python, hence that matrix.
d0mine 2y ago
All I see is that newer library versions drop support for old Python versions (seems natural, unless you have unlimited resources). For example, If you need Python 2 support, use corresponding old versions.