13 ms·

Problems with Python dependency management

86 points by sanbor 2y ago

buggy6257 2y ago
This really just reads like someone who ignored literally every good practice for using python and then pikachu_shocked.jpg when he set his world on fire Just having a virtual environment and requirements.txt alone would solve 90% of this article. Also with python 3.12 you literally CANT install python packages at the system level. Giant full page warning saying “use a venv you idiot” I expected something along these lines and was still disappointed by TFA
- zahlman 2y ago
  In my experience, one of the biggest factors driving people to have the author's experience is... backwards compatibility. The old ways of doing things have existed for much longer than the new ways, and become well established. Everyone just accepts the idea of copying Pip into every new virtual environment, even though it's a) totally unnecessary (even before the `--python` option was introduced two years ago, you could sort of get by with options like `--target` and `--python-version` and `--prefix` and `--platform` and `--abi`) and b) utterly insane (slow, wasteful, makes it more confusing when your PATH gets messed up, leads to misconceptions...). And that's before considering the things people try to do that aren't officially blessed use cases - like putting `sudo` and `--break-system-packages` on the same command line without a second thought, or putting code in `setup.py` that actually tries to copy the Python files to specific locations for the user, or trying to run Pip explicitly via its non-existent "API" by calling undocumented stuff instead of just specifying dependencies (including "extras" lists) properly. (The Pip documentation explicitly recommends invoking Pip via `subprocess` instead; but you're still probably doing something wrong if this isn't part of your own custom alternative to Poetry etc., and it won't help you if Pip isn't available in the environment - which it doesn't have to be, except to support that kind of insane use case). Another part is that people just don't want to learn. Yes, you get a 'Giant full page warning saying “use a venv you idiot”'. Yes, the distro gets to customize that warning and tell you exactly what to do. Users will still give over a million hits to the corresponding Stack Overflow question (https://stackoverflow.com/questions/75608323 https://stackoverflow.com/questions/75608323), which will collect dozens of terrible answers, many of them suggesting complete circumvention of the package-management lock. It was over a year before anything significant was done about the top answer there (disclosure: I contributed quite a bit after that point; I have a personal policy of not writing new answers for Stack Overflow, but having a proper answer at the top of this question was far too important for me to ignore), which only happened because the matter was brought to the attention of the Python Discourse forum community (https://discuss.python.org/t/_/56900 https://discuss.python.org/t/_/56900).
- liontwist 2y ago
  npm ecosystem refugees trying out a new platform
- jeremyjh 2y ago
  The author's complaint is that python is supposed to be a good language for people new to programming to pick up. But the default tooling manages dependencies in a way that is unsound, and this has been known for more than a decade. Yet the defaults are still terrible, and regardless of how many articles there are on "best practices" a lot of people get burned.
  - deleted 2y ago
    [deleted]
  - drunkenmagician 2y ago
    Yes! This exactly.
  - tomnipotent 2y ago
    I don't see any language in the blog post about "people new to programming to pick up". In fifteen years of using Python, the only people I see getting burned are, conveniently, the folks writing blogs on the subject. No one I've worked with or hired seems to be running into these issues. It's not to say that people don't run into issues, but the problems seem exaggerated every time this subject comes up.
  - zahlman 2y ago
    People who are new to programming have a long way to go before even the concept of "managing dependencies" could possibly be made coherent for them. And the "unsoundness" described (i.e. not having lockfile-driven workflows by default) really just doesn't matter a huge percentage of the time. I've been writing Python for 20 years and what I write nowadays will still just work on multiple Python versions across a wide range of versions for my dependencies - if it even has any dependencies at all. But nowadays people seem to put the cart before the horse, and try to teach about programming language ecosystems before they've properly taught about programming. People new to programming need to worry about programming first. If there are any concepts they need to learn before syntax and debugging, it's how to use a command line (because it'll be harder to drive tools otherwise; IDEs introduce greater complexity) and how to use version control (so they can make mistakes fearlessly). Educators, my plea: if you teach required basic skills to programmers before you actually teach programming, then those skills are infinitely more important than modern "dependency management". And for heavens' sake, you can absolutely think of a few months' worth of satisfying lesson plans that don't require wrapping one's head around full-scale data-science APIs, or heaven forbid machine-learning libraries. If you need any more evidence of the proper priorities, just look at Stack Overflow. It gets flooded with zero-effort questions dumping some arcane error message from the bowels of Tensorflow, forwarded from some Numpy 2d arrays used as matrices having the wrong shape - and it'll get posted by someone who has no concept of debugging, no idea of any of the underlying ML theory, and very possibly no idea what matrix multiplication is or why it's useful. What good is it to teach "dependency management" to a student who's miles away from understanding the actual dependencies being managed? For that matter, sometimes they'll take a screenshot of the terminal instead of copying and pasting an error message (never mind proper formatting). Sometimes they even use a cell phone to take a picture of the computer monitor. You're just not going to teach "dependency management" successfully to someone who isn't properly comfortable with using a computer.
- Starlevel004 2y ago
  > and requirements.txt The proliferation of requirements.txt files is a massive reason for why Python dependency management sucks.
  - zahlman 2y ago
    You'll be interested in relevant upcoming packaging standards PEPs: https://peps.python.org/pep-0751/ https://peps.python.org/pep-0751/ for lock files (still in discussion - have your say at https://discuss.python.org/t/_/69721 https://discuss.python.org/t/_/69721), and (recently accepted, but I can't immediately point at existing tool support) https://peps.python.org/pep-0735/ https://peps.python.org/pep-0735/ for listing groups of dependencies in pyproject.toml.
- sunshowers 2y ago
  The design of requirements.txt is a bit outdated -- it commits the first sin of developer tooling, which is to mix manually edited and automatically generated (via pip freeze) files. Newer systems use separate lockfiles for that reason, and uv brings this state of the art to Python.
  - otteromkram 2y ago
    That's why I have a requirements folder with separate files (eg - dev.txt, prod.txt) for various installation needs. If you want to include test dependencies in development, just add it into the file like you're installing a regular requirements file: -r test.txt And, to double-down, if you read the pip documentation (the second sin of software development?), you can use things other than pip freeze. Like, python -m pip list --not-required That option flag is pretty nice because it excludes packages that aren't dependencies (aka - the primary packages that you need). If you do that you don't to worry about dependency management as much.
  - janice1999 2y ago
    I think 'pip freeze' was introduced later and requirements.txt was not designed with such a use in mind. pipenv and other tools have lockfile equivalents for a while.
  - askonomm 2y ago
    And that makes this, what, the 100th package management solution for Python? A big reason why Python package management sucks is because there are as many package managers as there are frameworks in JavaScript. Nobody ever knows what is the standard, and by the time that information propagates, it is no longer the standard.
    - sunshowers 2y ago
      I believe and hope that uv settles this discussion.
    - mmcnl 2y ago
      Is it really such a big deal that there are multiple "good enough" solutions available? I understand it is suboptimal, especially for beginners, but dependency management in Python is a solved problem for a while now. And also there are not that many solutions. Previously Poetry was the best, now uv is looking to take its place. Not 100 different solutions imo.
- TZubiri 2y ago
  "You don't know what caused the breakage, and you don't know how to go back to a working environment. " The author would be well served by using the first person and not including us in his uncertainty.
- bmitc 2y ago
  It still doesn't make Python's dependency management not to be horrible. Every other modern language has a single tool for these built-in. Python doesn't, with half of the tools coming with the core and half of the tools coming from third-parties and many issues, conflicts, and incompatibilities. Even Poetry, which makes things much, much easier, makes decisions that are incompatible or makes managing dependencies more difficult in some cases.
ethbr1 2y ago
In before 'Python dependency management isn't hard. You just use {newest Python flavor-of-the-year way} of doing {same thing that is standardized in other languages}.' Which, fair. Python is and will always be a bazaar. >> PDM (Edit 14/12/2024) When I shared this article online, I was asked why I did not mention PDM. The honest reason was because I had not heard of it. Ha! On brand.
- lanstin 2y ago
  "There should be one-- and preferably only one --obvious way to do it." It was originally not a bazaar but "batteries included" where the thing you wanted to do had an obvious best way of doing it. An extremely difficult property to maintain over the decades.
- thesuperbigfrog 2y ago
  So with Python dependency management 'there is more than one way to do it': pip, pipenv, poetry, conda, setuptools, hatch, micropipenv, PDM, pip-tools, egg, uv, ActiveState platform, homebrew, your operating system's package manager, and many others . . . Relevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/
  - zahlman 2y ago
    Of the list you offer, only Poetry, Hatch, PDM and Uv actually do "Python dependency management" - in the sense of offering an update command to recalculate locked versions of dependencies, a wrapper to install or upgrade those dependencies, and a wrapper to keep track of which environment they'll be installed into. pipenv, micropipenv and pip-tools are utilities for creating records of dependencies, but don't actually "manage" those dependencies in the above sense. Your list also includes an installer (Pip), a build backend (Setuptools - although it has older deprecated use as something vaguely resembling a workflow tool similar to modern dependency managers), a long-deprecated file format (egg) which PyPI hasn't even accepted for a year and a half (https://packaging.python.org/en/latest/discussions/package-formats/ https://packaging.python.org/en/latest/discussions/package-f...), two alternative sources for Python itself (ActiveState and homebrew - and I doubt anyone has a good reason to use ActiveState any more), and two package management solutions that are orthogonal to the Python ecosystem (Conda - which was created to support Python, but its environments aren't particularly Python-centric - and Linux system package managers). Any system can be made to look complex by conflating its parts with other vaguely related but ultimately irrelevant objects.
shlomo_z 2y ago
Some sent this in a group I am in 2 weeks ago. I will copy my response here: "Articles like that over-dramatize things a lot. As a self-taught Python developer, I mostly learned from looking at code. When I started, I didn't know anything about packaging. I just installed stuff globally with pip and moved on. I didn't really build projects, just scripts. Once in a while I used a venv when I was following a guide using them. To give a bit of perspective, for a while I didn't really know how to make a class, only a function. Fast forward a few months, I learned a bit more about projects and also started contributing and/or forking open source projects (again, never took classes, I just explore). I used poetry a little, it works nicely. Now, I use uv for anything new, and it works beautifully. It's really simple. uv init, uv add any deps, and uv run to run the project or just activate the venv. And I never run into dependency issues. It's like really simple. And being able to manage python versions really simply is a really nice bonus. My cicd doesn't even use a special docker container or anything. Literally installs uv (using curl | sh), uv sync, uv run. Finished. And very fast too. So yeah, Python dependencies aren't automatically vendored. And yes, Python tooling has historically been bad. But now we have amazing tooling and it's really really easy to learn. uv is wonderful, poetry is also great, and anyone complaining it's too hard is either over dramatizing it or doesn't have 5 minutes to read a readme. So yeah, people should stop over dramatizing something that really isn't dramatic."
- JanisErdmanis 2y ago
  I recently learned that uv is written in more than 100k lines of Rust code. Cudos to the developers! This is however also indicative of python dependency managment complexity.
- sgarland 2y ago
  100% agreed. Every time I’ve run into dependency hell, it’s been because someone at best has a requirements.txt with unpinned packages. Poetry was a wonderful breath of fresh air. uv is the same, but 100x faster.
rednafi 2y ago
Don’t we see this kind of rant pop up all the time? So much misery could be avoided by simply using a virtual environment. Plus, uv handles it all in a more elegant manner.
- ryan-duve 2y ago
  Do you recommend something different than the article?
stavros 2y ago
These days, I am extremely happy to be using uv with single scripts. Put this at the start of your script (which has to end in .py): (EDIT: Sorry, HN doesn't like code, see the start of https://github.com/skorokithakis/calumny/blob/master/calumny.py https://github.com/skorokithakis/calumny/blob/master/calumny... for an example) The script will run with uv and automatically create a venv and install all dependencies in it. It's fantastic. The other alternative, if you want to be extra sure, is to create a pex. It'll even bundle the Python interpreter in the executable (or download it if it's not available on the target machine), and will run anywhere with no other dependency (maybe libc? I forget).
- CamperBob2 2y ago
  That looks pretty awesome. What are the drawbacks to using uv? Does it get along with existing pip and conda installations?
  - stavros 2y ago
    I don't use conda, so I don't know, but it works fine with pip, it just makes its own virtualenv somewhere. It doesn't touch anything else.
  - shlomo_z 2y ago
    It is too fast and doesn't give you time to think. :) Just Kidding! It's amazing. It gets along with existing installations very well.
  - sunshowers 2y ago
    One minor drawback I've noticed is that direnv doesn't come with uv support out of the box: https://github.com/direnv/direnv/issues/1250 https://github.com/direnv/direnv/issues/1250
    - infamia 2y ago
      I'm in the middle of replacing all of my usage of direnv with mise [1]. It does everything direnv/asdf can (for my use cases) plus a lot more. Mise can install a lot of dev software/frameworks [2], run tasks, file watcher all in a compatible manner for *nix OSes. mise is ridiculously good and even minimizes my need for docker locally. [1] https://mise.jdx.dev/ https://mise.jdx.dev/ [2] https://mise.jdx.dev/registry.html https://mise.jdx.dev/registry.html
yosito 2y ago
Wait until you hear about JavaScript dependency management!
roenxi 2y ago
Bit of a Chesterton's fence situation. One of the reasons that Python is popular at all is because the dependency management is terrible and easy to get wrong. The point is it enables relative novices to get started without engaging with important questions. Things like what environmental assumptions are being made, what dependency versions are supported, what a sustainable build pipeline looks like. They can just be vague and their project will work in the here and now - enough to do whatever they want to do. There is a trade off here and the equilibrium is probably deliberate. The sort of person who tries to get dependencies right up is a professional programmer and although a lot of them use Python the language is designed for a much broader audience. Java is an example of much better dependency management by default and in the main it is only professionals with a view to the long term using Java. Setting up a Java project needs a tutorial and ideally specialist software.
- xg15 2y ago
  I don't think the OP's complaint was that python's dependency management wasn't complicated enough or doesn't involve enough cryptic XML yet. The point was that the novice will find themselves massively frustrated the moment they try to run their script on a different computer. Indeed, the novice shouldn't have to make all kinds of intricate choices about the build system to get something running. The language designers should have provided a good set of default choices here. The problem with python is that the default choices aren't actually the good ones.
- c0redump 2y ago
  Can you elaborate on the difficulties of setting up Java projects? I have never worked in Java at an enterprise level, but have tinkered a bit. IntelliJ makes setting up a maven-based project pretty much one-click. But I’m guessing that the complexity that you’re referring to wouldn’t be apparent to someone like me who is only using it for small hobby projects.
  - roenxi 2y ago
    You have to have identified that you need IntelliJ and know what a maven-based project is (indeed, know what a 'project' is, the concept is a bit technical). And there is the split between the JDK & JVM. This is all to be learned before actually approaching the challenge of adding a dependency to the project. Compare that to a Python beginner where they install Python & use the text editor that exists on their machine & it can be a general purpose text editor rather than something specifically developed to write Java code. There might be one `pip install` along the way, but no requirement to understand a folder layout, project concept or even create a file for dependency management. There is even a reasonable chance that Python is pre-installed on their machine.
martynassubo 2y ago
While uv wasn't very mature, I wrote a similar piece, where in the end I tried suggesting pyenv/pipx/poetry: https://open.substack.com/pub/martynassubonis/p/python-project-management-primer https://open.substack.com/pub/martynassubonis/p/python-proje... However, these days, I would just probably go with uv and hope that astral doesn't pull any VC shenanigans: https://open.substack.com/pub/martynassubonis/p/python-project-management-primer-a55 https://open.substack.com/pub/martynassubonis/p/python-proje... It's not Rust/Go toolchain, but we work with what we got here...
WillAdams 2y ago
Well, there is an XKCD on this: https://xkcd.com/1987/ https://xkcd.com/1987/ FWIW, I've rolled, crashed, and burned on at least one attempt to use Python tooling due to forgotten about installs/environments which required uninstalling/deleting everything and start anew. It would be nice if there was a consensus and an agreed-upon solution and process which would work as a new user might expect.
m4thfr34k 2y ago
Requirements file and everything is installed in a container where the code runs from. Lots of problems solved.
hk1337 2y ago
I think the biggest issue is that nobody can settle on what to use and how to use it. pip, pdm, poetry, and maybe something else I don’t know about. Some new flashy dependency manager comes along and that’s the new hotness. I'm probably just as guilty of that.
sunshowers 2y ago
I'm quite happy with uv these days! It's taken most of the pain out of Python management, and I can just think in terms of `uv run`, etc. (Astral sponsors me on GitHub, but I'd hold the same belief even if they didn't.)
Alir3z4 2y ago
Almost 2 decades of working with python. I create a venv. Pip install and keep my direct deps in requirements.txt That's it. Never understood all these python dependency management problems dramas. Recently, I started using pyproject.toml as well which makes the whole thing more compact. I make lots of python packages too. Either I go setup.py or sometimes I like to use flit for no specific reason. I haven't ever felt the need for something like uv. I'm good with pip.
- rcxdude 2y ago
  This is more or less my experience, but I think in part it took a while for pip to actually get into a usable position, hence some of the proliferation of other options.
- mrbungie 2y ago
  This. And also try always to fix the version of the requirements, and that's it. Never had a problem making reproducible builds doing so.
  - Fethbita 2y ago
    I had issues with exactly this method. One of my dependencies was pulled off to a paid model so my project no longer worked.
    - mrbungie 2y ago
      Anyone remember the leftpad fiasco in the node ecosystem? That could happen in any dependency system that allows owners to unpublish dependencies and that's one risk users must weigh when adding them.
  - Alir3z4 2y ago
    Yeah, I assume pinning the version is something everyone does? Or probably many just don't and will have those "python deps management is a mess drama". TBH, I've seen tutorials or even some companies simply do `pip freeze > requirements.txt` :shrug: which is a mess.
ltbarcly3 2y ago
Nah, it's fine. Use poetry or uv and get on with it.
dmart 2y ago
Venvs are so clunky and probably the biggest stumbling block for beginners. There was a proposal for a directory-based node_modules analogue which was unfortunately rejected. I think that would have been the single biggest improvement to the Python onboarding experience ever.
- silverwind 2y ago
  Indeed, python dug its own grave by not supporting in-directory venv. One can emulate it with tools like poetry and uv but that incurs a performance penalty that every script has to go through `poetry run` and `uv run` which is often a few hundret ms and unsuitable for performant CLIs.
- zahlman 2y ago
  > There was a proposal for a directory-based node_modules analogue which was unfortunately rejected. There were many problems with the proposal. The corresponding discussion (https://discuss.python.org/t/_/963 https://discuss.python.org/t/_/963) is worth looking through, despite the length. Installers like Pip could help by offering to install `--in-new-environment` for the first install, and Brett Cannon (a core developer) has done some work on a universal (i.e. not just Windows) "launcher" (https://github.com/brettcannon/python-launcher https://github.com/brettcannon/python-launcher) which can automatically detect and use the project's venv if you create it in the right place (i.e., what you'd have to do with __pypackages__ anyway).
daemonologist 2y ago
For me the vast majority of the pain occurs when _updating_ dependencies. It can be an all-day chore to find and force versions of everything which will work together, and occasionally you'll have an irreconcilable conflict in subdependencies. I have in the past had to hack in a second version under a different name and update all imports in the parent dependencies to match. I know other languages have various solutions for this (to basically have package namespaces local to a branch of the dependency tree), but I don't know how much better that experience is.
- tcfhgj 2y ago
  10 hours later: Invoking SAT with clause count: 9661561 Invoking SAT with clause count: 5164645
- otteromkram 2y ago
  Have you tried using pip? Or, reading the documentation? It has more features than just "install." You could try running python -m pip check To check dependencies. Or, python -m pip inspect To get a JSON output of your current virtual environment. Or, update stuff automatically: python -m pip install --upgrade Or, skip worrying about dependencies: python -m pip install --no-deps Or, do a dry run installation with full report on what would be installed: pip install --ignore-installed --dry-run --quiet --report And, there's a lot more than that. Pip is pretty powerful and I'm surprised everyone dislikes it so much. Hope this helps. Cheers!
  - zahlman 2y ago
    In principle, yes. In practice, there are a lot of issues with that workflow. For example, `pip install --ignore-installed --dry-run --quiet --report` will build sdists (and run arbitrary code from `setup.py` or other places specified by a build backend) - just so that it can confirm that the downloaded sdist would produce a wheel with the right name and version. Even `pip download` will do the same. I'm not kidding. There are multiple outstanding issues on the tracker that are all ultimately about this problem, which has persisted through multiple versions of the UI and all the evolving packaging standards, going back almost the entire history of Pip. See for example https://github.com/pypa/pip/issues/1884 https://github.com/pypa/pip/issues/1884 ; I have a long list of related reports written down somewhere. A security researcher was once infamously bitten by this (https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html https://moyix.blogspot.com/2022/09/someones-been-messing-wit...).
booleandilemma 2y ago
Having recently used python and poetry - I agree 100%!
TZubiri 2y ago
'Python is mostly a "glue" language' 100% disagree, yes it may be used for that, but it's a fully fledged language, people use it for data science or for backend business logic.
- lanstin 2y ago
  The data science uses are all using it for glueing a bit of data massage/data gathering into their high performance math/ML libraries. Django users are using it natively, and cherryPy/etc. Python full-stack maybe, but they'll have DB drivers and JSON parsers probably from C.
- tcfhgj 2y ago
  In data science mostly as glue code by design using frameworks which do the heavy work in native machine code.
  - TZubiri 2y ago
    Sure, dependency wise. But the main business logic is in python, how is that python being the glue, more like the brain.
abeppu 2y ago
> Good dependency management means ... > it is possible to safely evolve and update your environment when new versions of dependencies become available. The post, despite its length, doesn't really spend time on this aspect, and I think it's one of the weaker areas. Suppose my library depends on both A and B, and both of A and B depend on C, and in the current version of my code, there's some set of versions that matches all declared dependencies. But A releases a new version which has new features I want to use, and also depends on a newer version of C than B supports. I'm blocked from upgrading A with the normal tooling, unless I either ditch or modify B. This can be a problem in other ecosystems too, but better tools support working around it. In the java world (and I may be out of date here), the maven-shade tool would allow you to effectively use multiple versions of C, e.g. by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need. I think this is also most of the time in principle possible in python (though perhaps some relevant static analysis of seeing what is actually used in more difficult), but the ecosystem doesn't really support or encourage it.
- zahlman 2y ago
  >I'm blocked from upgrading A with the normal tooling, unless I either ditch or modify B.... I think this is also most of the time in principle possible in python, but the ecosystem doesn't really support or encourage it. No, the main problems with this are at the language level. Languages like Java can do it because the import is resolved statically. Python's `import` statement is incredibly dynamic, offering a variety of rarely-used hooks (start at https://docs.python.org/3/library/sys.html#sys.meta_path https://docs.python.org/3/library/sys.html#sys.meta_path and follow the links, if you dare) that ultimately (by default) put imported modules into a single global dictionary, keyed by name. If you want to work around that, you can, but you'll definitely have to roll up your sleeves a bit. Even if you just want to choose between two different versions at runtime, the base `import` syntax doesn't have a way to specify a version. If the package version isn't explicitly namespaced (which causes more pain for everyone who doesn't care about the exact version) then you need to make special arrangements so that the one you want will be found first (normally, by manipulating `sys.path`). But if you want to use both during the same run, you'll encounter far more problems. The global dict strategy is very deliberate. Of course it offers a performance benefit (caching) but it's also required for correctness in many normal cases. It makes sure that different parts of the code importing the same module see the same module object, and therefore the same module-level globals. That is: in the Python world, it's common that the design of C involves mutating its global state. If you "let B use its older version" then it would have to be an entirely separate object with its own state, and mutating that would result in changes not seen by A. Whether or not that's desirable would depend on your individual use case. A lot of the time, you would want the change to be shared. But then, even if you could propagate the changes automatically, you'd have to deal with the fact that the two versions of C don't necessarily represent their state identically. > by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need. In principle, the B author can make the dependency on C optional ("extra"), for those users who do need that part of B. Or even for B to optionally depend on B-prime, which encapsulates the C-using parts. This is actually not very difficult (and the above-described dynamic nature of Python is very helpful here), but in practice it isn't done very much (perhaps Python developers are wary of creating another left-pad situation). But I do wish e.g. Numpy were more like that. But the "fat-jar" analog is also possible in Python, as long as it isn't a problem that you aren't sharing state between C-old and C-new. It's called "vendoring"; Pip does a lot of it; and the tooling that Pip uses to enable it is also made available (https://pypi.org/project/vendoring/ https://pypi.org/project/vendoring/) by one of the main Pip devs (Pradyun Gedam).
llIIllIIllIIl 2y ago
Python deps are alright unless you’re updating celery.
lofaszvanitt 2y ago
And the drama keeps happening because the article is 70K long...
stcg 2y ago
One of the biggest usability problems with Python dependencies is that the name you import might be different from the name that you use to install the package. So if you find some script on the web that has an `import foo` at the top, you cannot just `pip install foo`. Instead, you'll have to do some research into which package was originally used. Maybe it's named `pyfoo` or `foolib`. Compare that to for example Java, which does not have that problem, thanks to Reverse Domain Name Notation. That is a much better system.
- zahlman 2y ago
  "install name == import name" cannot work in Python, because when you `pip install foo`, you may get more than one top-level package. Or you may get a single-file module. Or you may, validly per the spec, get no Python code whatsoever. (For example, you could publish large datasets separately from your data science library, as separate wheels which could be listed as optional dependencies.) The lack of good namespacing practice is a problem. Part of the reason for it, in my estimation, is that developers have cargo-culted around a mistaken understanding of `__init__.py`.
jredwards 2y ago
This post feels like it was written 10 years ago. There are plenty of excellent dependency management systems for Python these days. Honestly, the biggest problem is that there are too many ways to set it up.
ac130kz 2y ago
uv (and requirements.txt as a lock) + Debian Docker container (distroless with added packages for runtime). There are no problems with this setup.
sebazzz 2y ago
> You decide to re-install your operating system and vow to go back to doing everything in Excel. Which itself can run Python now. Only in the Microsoft Cloud, not only to rake in that sweet subscription money, but probably also to avoid these headaches.
aszen 2y ago
Everyone seems to praising uv in the comments, seems like no one read this article, it has its drawbacks, it can't install system level libraries and dependencies which many packages depend on.
- thebigspacefuck 2y ago
  From the docs “For convenience, uv pip install --system will install into the system Python environment.” Are you referring to something else?
  - aszen 2y ago
    No I'm referring to system level libs, that you need to install via your operating system package manager like `apt` for python packages that depend on them.
cr125rider 2y ago
Docker and Devcontainers. You can make a mess in your own little box and not mess with my system. We use conda/momba to install some more complex dependencies that are piles of shared libraries and a messy web of sub dependencies.
VeejayRampay 2y ago
really sucks that we're stuck with python when languages of the same realm (JS and ruby for example) are so much better (as languages and with regard to the associated tooling)
sgarland 2y ago
One easy trick they don’t want you to know about: stop importing dependencies. You probably don’t actually need numpy. You probably don’t need requests. The stdlib is incredibly broad. This doesn’t always work, of course (especially for large projects), but for smaller ones, it absolutely does.
- nothrowaways 2y ago
  Or copy entire numpy and requests repos with your code, they don't want you to know this either.
egberts1 2y ago
Biggest turn off of all is the lack of centralized (if any) search engine for Python packages at CLI-level `pip` command.

86 points by sanbor 2y ago

This really just reads like someone who ignored literally every good practice for using python and then pikachu_shocked.jpg when he set his world on fire Just having a virtual environment and requirements.txt alone would solve 90% of this article. Also with python 3.12 you literally CANT install python packages at the system level. Giant full page warning saying “use a venv you idiot” I expected something along these lines and was still disappointed by TFA

zahlman 2y ago
In my experience, one of the biggest factors driving people to have the author's experience is... backwards compatibility. The old ways of doing things have existed for much longer than the new ways, and become well established. Everyone just accepts the idea of copying Pip into every new virtual environment, even though it's a) totally unnecessary (even before the `--python` option was introduced two years ago, you could sort of get by with options like `--target` and `--python-version` and `--prefix` and `--platform` and `--abi`) and b) utterly insane (slow, wasteful, makes it more confusing when your PATH gets messed up, leads to misconceptions...). And that's before considering the things people try to do that aren't officially blessed use cases - like putting `sudo` and `--break-system-packages` on the same command line without a second thought, or putting code in `setup.py` that actually tries to copy the Python files to specific locations for the user, or trying to run Pip explicitly via its non-existent "API" by calling undocumented stuff instead of just specifying dependencies (including "extras" lists) properly. (The Pip documentation explicitly recommends invoking Pip via `subprocess` instead; but you're still probably doing something wrong if this isn't part of your own custom alternative to Poetry etc., and it won't help you if Pip isn't available in the environment - which it doesn't have to be, except to support that kind of insane use case). Another part is that people just don't want to learn. Yes, you get a 'Giant full page warning saying “use a venv you idiot”'. Yes, the distro gets to customize that warning and tell you exactly what to do. Users will still give over a million hits to the corresponding Stack Overflow question (https://stackoverflow.com/questions/75608323 https://stackoverflow.com/questions/75608323), which will collect dozens of terrible answers, many of them suggesting complete circumvention of the package-management lock. It was over a year before anything significant was done about the top answer there (disclosure: I contributed quite a bit after that point; I have a personal policy of not writing new answers for Stack Overflow, but having a proper answer at the top of this question was far too important for me to ignore), which only happened because the matter was brought to the attention of the Python Discourse forum community (https://discuss.python.org/t/_/56900 https://discuss.python.org/t/_/56900).
liontwist 2y ago
npm ecosystem refugees trying out a new platform
jeremyjh 2y ago
The author's complaint is that python is supposed to be a good language for people new to programming to pick up. But the default tooling manages dependencies in a way that is unsound, and this has been known for more than a decade. Yet the defaults are still terrible, and regardless of how many articles there are on "best practices" a lot of people get burned.
- deleted 2y ago
  [deleted]
- drunkenmagician 2y ago
  Yes! This exactly.
- tomnipotent 2y ago
  I don't see any language in the blog post about "people new to programming to pick up". In fifteen years of using Python, the only people I see getting burned are, conveniently, the folks writing blogs on the subject. No one I've worked with or hired seems to be running into these issues. It's not to say that people don't run into issues, but the problems seem exaggerated every time this subject comes up.
- zahlman 2y ago
  People who are new to programming have a long way to go before even the concept of "managing dependencies" could possibly be made coherent for them. And the "unsoundness" described (i.e. not having lockfile-driven workflows by default) really just doesn't matter a huge percentage of the time. I've been writing Python for 20 years and what I write nowadays will still just work on multiple Python versions across a wide range of versions for my dependencies - if it even has any dependencies at all. But nowadays people seem to put the cart before the horse, and try to teach about programming language ecosystems before they've properly taught about programming. People new to programming need to worry about programming first. If there are any concepts they need to learn before syntax and debugging, it's how to use a command line (because it'll be harder to drive tools otherwise; IDEs introduce greater complexity) and how to use version control (so they can make mistakes fearlessly). Educators, my plea: if you teach required basic skills to programmers before you actually teach programming, then those skills are infinitely more important than modern "dependency management". And for heavens' sake, you can absolutely think of a few months' worth of satisfying lesson plans that don't require wrapping one's head around full-scale data-science APIs, or heaven forbid machine-learning libraries. If you need any more evidence of the proper priorities, just look at Stack Overflow. It gets flooded with zero-effort questions dumping some arcane error message from the bowels of Tensorflow, forwarded from some Numpy 2d arrays used as matrices having the wrong shape - and it'll get posted by someone who has no concept of debugging, no idea of any of the underlying ML theory, and very possibly no idea what matrix multiplication is or why it's useful. What good is it to teach "dependency management" to a student who's miles away from understanding the actual dependencies being managed? For that matter, sometimes they'll take a screenshot of the terminal instead of copying and pasting an error message (never mind proper formatting). Sometimes they even use a cell phone to take a picture of the computer monitor. You're just not going to teach "dependency management" successfully to someone who isn't properly comfortable with using a computer.
Starlevel004 2y ago
> and requirements.txt The proliferation of requirements.txt files is a massive reason for why Python dependency management sucks.
- zahlman 2y ago
  You'll be interested in relevant upcoming packaging standards PEPs: https://peps.python.org/pep-0751/ https://peps.python.org/pep-0751/ for lock files (still in discussion - have your say at https://discuss.python.org/t/_/69721 https://discuss.python.org/t/_/69721), and (recently accepted, but I can't immediately point at existing tool support) https://peps.python.org/pep-0735/ https://peps.python.org/pep-0735/ for listing groups of dependencies in pyproject.toml.
sunshowers 2y ago
The design of requirements.txt is a bit outdated -- it commits the first sin of developer tooling, which is to mix manually edited and automatically generated (via pip freeze) files. Newer systems use separate lockfiles for that reason, and uv brings this state of the art to Python.
- otteromkram 2y ago
  That's why I have a requirements folder with separate files (eg - dev.txt, prod.txt) for various installation needs. If you want to include test dependencies in development, just add it into the file like you're installing a regular requirements file: -r test.txt And, to double-down, if you read the pip documentation (the second sin of software development?), you can use things other than pip freeze. Like, python -m pip list --not-required That option flag is pretty nice because it excludes packages that aren't dependencies (aka - the primary packages that you need). If you do that you don't to worry about dependency management as much.
- janice1999 2y ago
  I think 'pip freeze' was introduced later and requirements.txt was not designed with such a use in mind. pipenv and other tools have lockfile equivalents for a while.
- askonomm 2y ago
  And that makes this, what, the 100th package management solution for Python? A big reason why Python package management sucks is because there are as many package managers as there are frameworks in JavaScript. Nobody ever knows what is the standard, and by the time that information propagates, it is no longer the standard.
  - sunshowers 2y ago
    I believe and hope that uv settles this discussion.
  - mmcnl 2y ago
    Is it really such a big deal that there are multiple "good enough" solutions available? I understand it is suboptimal, especially for beginners, but dependency management in Python is a solved problem for a while now. And also there are not that many solutions. Previously Poetry was the best, now uv is looking to take its place. Not 100 different solutions imo.
TZubiri 2y ago
"You don't know what caused the breakage, and you don't know how to go back to a working environment. " The author would be well served by using the first person and not including us in his uncertainty.
bmitc 2y ago
It still doesn't make Python's dependency management not to be horrible. Every other modern language has a single tool for these built-in. Python doesn't, with half of the tools coming with the core and half of the tools coming from third-parties and many issues, conflicts, and incompatibilities. Even Poetry, which makes things much, much easier, makes decisions that are incompatible or makes managing dependencies more difficult in some cases.

ethbr1 2y ago

In before 'Python dependency management isn't hard. You just use {newest Python flavor-of-the-year way} of doing {same thing that is standardized in other languages}.' Which, fair. Python is and will always be a bazaar. >> PDM (Edit 14/12/2024) When I shared this article online, I was asked why I did not mention PDM. The honest reason was because I had not heard of it. Ha! On brand.

lanstin 2y ago
"There should be one-- and preferably only one --obvious way to do it." It was originally not a bazaar but "batteries included" where the thing you wanted to do had an obvious best way of doing it. An extremely difficult property to maintain over the decades.
thesuperbigfrog 2y ago
So with Python dependency management 'there is more than one way to do it': pip, pipenv, poetry, conda, setuptools, hatch, micropipenv, PDM, pip-tools, egg, uv, ActiveState platform, homebrew, your operating system's package manager, and many others . . . Relevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/
- zahlman 2y ago
  Of the list you offer, only Poetry, Hatch, PDM and Uv actually do "Python dependency management" - in the sense of offering an update command to recalculate locked versions of dependencies, a wrapper to install or upgrade those dependencies, and a wrapper to keep track of which environment they'll be installed into. pipenv, micropipenv and pip-tools are utilities for creating records of dependencies, but don't actually "manage" those dependencies in the above sense. Your list also includes an installer (Pip), a build backend (Setuptools - although it has older deprecated use as something vaguely resembling a workflow tool similar to modern dependency managers), a long-deprecated file format (egg) which PyPI hasn't even accepted for a year and a half (https://packaging.python.org/en/latest/discussions/package-formats/ https://packaging.python.org/en/latest/discussions/package-f...), two alternative sources for Python itself (ActiveState and homebrew - and I doubt anyone has a good reason to use ActiveState any more), and two package management solutions that are orthogonal to the Python ecosystem (Conda - which was created to support Python, but its environments aren't particularly Python-centric - and Linux system package managers). Any system can be made to look complex by conflating its parts with other vaguely related but ultimately irrelevant objects.

shlomo_z 2y ago

Some sent this in a group I am in 2 weeks ago. I will copy my response here: "Articles like that over-dramatize things a lot. As a self-taught Python developer, I mostly learned from looking at code. When I started, I didn't know anything about packaging. I just installed stuff globally with pip and moved on. I didn't really build projects, just scripts. Once in a while I used a venv when I was following a guide using them. To give a bit of perspective, for a while I didn't really know how to make a class, only a function. Fast forward a few months, I learned a bit more about projects and also started contributing and/or forking open source projects (again, never took classes, I just explore). I used poetry a little, it works nicely. Now, I use uv for anything new, and it works beautifully. It's really simple. uv init, uv add any deps, and uv run to run the project or just activate the venv. And I never run into dependency issues. It's like really simple. And being able to manage python versions really simply is a really nice bonus. My cicd doesn't even use a special docker container or anything. Literally installs uv (using curl | sh), uv sync, uv run. Finished. And very fast too. So yeah, Python dependencies aren't automatically vendored. And yes, Python tooling has historically been bad. But now we have amazing tooling and it's really really easy to learn. uv is wonderful, poetry is also great, and anyone complaining it's too hard is either over dramatizing it or doesn't have 5 minutes to read a readme. So yeah, people should stop over dramatizing something that really isn't dramatic."

JanisErdmanis 2y ago
I recently learned that uv is written in more than 100k lines of Rust code. Cudos to the developers! This is however also indicative of python dependency managment complexity.
sgarland 2y ago
100% agreed. Every time I’ve run into dependency hell, it’s been because someone at best has a requirements.txt with unpinned packages. Poetry was a wonderful breath of fresh air. uv is the same, but 100x faster.

rednafi 2y ago

Don’t we see this kind of rant pop up all the time? So much misery could be avoided by simply using a virtual environment. Plus, uv handles it all in a more elegant manner.

ryan-duve 2y ago
Do you recommend something different than the article?

stavros 2y ago

These days, I am extremely happy to be using uv with single scripts. Put this at the start of your script (which has to end in .py): (EDIT: Sorry, HN doesn't like code, see the start of https://github.com/skorokithakis/calumny/blob/master/calumny.py https://github.com/skorokithakis/calumny/blob/master/calumny... for an example) The script will run with uv and automatically create a venv and install all dependencies in it. It's fantastic. The other alternative, if you want to be extra sure, is to create a pex. It'll even bundle the Python interpreter in the executable (or download it if it's not available on the target machine), and will run anywhere with no other dependency (maybe libc? I forget).

CamperBob2 2y ago
That looks pretty awesome. What are the drawbacks to using uv? Does it get along with existing pip and conda installations?
- stavros 2y ago
  I don't use conda, so I don't know, but it works fine with pip, it just makes its own virtualenv somewhere. It doesn't touch anything else.
- shlomo_z 2y ago
  It is too fast and doesn't give you time to think. :) Just Kidding! It's amazing. It gets along with existing installations very well.
- sunshowers 2y ago
  One minor drawback I've noticed is that direnv doesn't come with uv support out of the box: https://github.com/direnv/direnv/issues/1250 https://github.com/direnv/direnv/issues/1250
  - infamia 2y ago
    I'm in the middle of replacing all of my usage of direnv with mise [1]. It does everything direnv/asdf can (for my use cases) plus a lot more. Mise can install a lot of dev software/frameworks [2], run tasks, file watcher all in a compatible manner for *nix OSes. mise is ridiculously good and even minimizes my need for docker locally. [1] https://mise.jdx.dev/ https://mise.jdx.dev/ [2] https://mise.jdx.dev/registry.html https://mise.jdx.dev/registry.html

yosito 2y ago

Wait until you hear about JavaScript dependency management!

roenxi 2y ago

Bit of a Chesterton's fence situation. One of the reasons that Python is popular at all is because the dependency management is terrible and easy to get wrong. The point is it enables relative novices to get started without engaging with important questions. Things like what environmental assumptions are being made, what dependency versions are supported, what a sustainable build pipeline looks like. They can just be vague and their project will work in the here and now - enough to do whatever they want to do. There is a trade off here and the equilibrium is probably deliberate. The sort of person who tries to get dependencies right up is a professional programmer and although a lot of them use Python the language is designed for a much broader audience. Java is an example of much better dependency management by default and in the main it is only professionals with a view to the long term using Java. Setting up a Java project needs a tutorial and ideally specialist software.

xg15 2y ago
I don't think the OP's complaint was that python's dependency management wasn't complicated enough or doesn't involve enough cryptic XML yet. The point was that the novice will find themselves massively frustrated the moment they try to run their script on a different computer. Indeed, the novice shouldn't have to make all kinds of intricate choices about the build system to get something running. The language designers should have provided a good set of default choices here. The problem with python is that the default choices aren't actually the good ones.
c0redump 2y ago
Can you elaborate on the difficulties of setting up Java projects? I have never worked in Java at an enterprise level, but have tinkered a bit. IntelliJ makes setting up a maven-based project pretty much one-click. But I’m guessing that the complexity that you’re referring to wouldn’t be apparent to someone like me who is only using it for small hobby projects.
- roenxi 2y ago
  You have to have identified that you need IntelliJ and know what a maven-based project is (indeed, know what a 'project' is, the concept is a bit technical). And there is the split between the JDK & JVM. This is all to be learned before actually approaching the challenge of adding a dependency to the project. Compare that to a Python beginner where they install Python & use the text editor that exists on their machine & it can be a general purpose text editor rather than something specifically developed to write Java code. There might be one `pip install` along the way, but no requirement to understand a folder layout, project concept or even create a file for dependency management. There is even a reasonable chance that Python is pre-installed on their machine.

martynassubo 2y ago

While uv wasn't very mature, I wrote a similar piece, where in the end I tried suggesting pyenv/pipx/poetry: https://open.substack.com/pub/martynassubonis/p/python-project-management-primer https://open.substack.com/pub/martynassubonis/p/python-proje... However, these days, I would just probably go with uv and hope that astral doesn't pull any VC shenanigans: https://open.substack.com/pub/martynassubonis/p/python-project-management-primer-a55 https://open.substack.com/pub/martynassubonis/p/python-proje... It's not Rust/Go toolchain, but we work with what we got here...

WillAdams 2y ago

Well, there is an XKCD on this: https://xkcd.com/1987/ https://xkcd.com/1987/ FWIW, I've rolled, crashed, and burned on at least one attempt to use Python tooling due to forgotten about installs/environments which required uninstalling/deleting everything and start anew. It would be nice if there was a consensus and an agreed-upon solution and process which would work as a new user might expect.

m4thfr34k 2y ago

Requirements file and everything is installed in a container where the code runs from. Lots of problems solved.

hk1337 2y ago

I think the biggest issue is that nobody can settle on what to use and how to use it. pip, pdm, poetry, and maybe something else I don’t know about. Some new flashy dependency manager comes along and that’s the new hotness. I'm probably just as guilty of that.

sunshowers 2y ago

I'm quite happy with uv these days! It's taken most of the pain out of Python management, and I can just think in terms of `uv run`, etc. (Astral sponsors me on GitHub, but I'd hold the same belief even if they didn't.)

Alir3z4 2y ago

Almost 2 decades of working with python. I create a venv. Pip install and keep my direct deps in requirements.txt That's it. Never understood all these python dependency management problems dramas. Recently, I started using pyproject.toml as well which makes the whole thing more compact. I make lots of python packages too. Either I go setup.py or sometimes I like to use flit for no specific reason. I haven't ever felt the need for something like uv. I'm good with pip.

rcxdude 2y ago
This is more or less my experience, but I think in part it took a while for pip to actually get into a usable position, hence some of the proliferation of other options.
mrbungie 2y ago
This. And also try always to fix the version of the requirements, and that's it. Never had a problem making reproducible builds doing so.
- Fethbita 2y ago
  I had issues with exactly this method. One of my dependencies was pulled off to a paid model so my project no longer worked.
  - mrbungie 2y ago
    Anyone remember the leftpad fiasco in the node ecosystem? That could happen in any dependency system that allows owners to unpublish dependencies and that's one risk users must weigh when adding them.
- Alir3z4 2y ago
  Yeah, I assume pinning the version is something everyone does? Or probably many just don't and will have those "python deps management is a mess drama". TBH, I've seen tutorials or even some companies simply do `pip freeze > requirements.txt` :shrug: which is a mess.

ltbarcly3 2y ago

Nah, it's fine. Use poetry or uv and get on with it.

dmart 2y ago

Venvs are so clunky and probably the biggest stumbling block for beginners. There was a proposal for a directory-based node_modules analogue which was unfortunately rejected. I think that would have been the single biggest improvement to the Python onboarding experience ever.

silverwind 2y ago
Indeed, python dug its own grave by not supporting in-directory venv. One can emulate it with tools like poetry and uv but that incurs a performance penalty that every script has to go through `poetry run` and `uv run` which is often a few hundret ms and unsuitable for performant CLIs.
zahlman 2y ago
> There was a proposal for a directory-based node_modules analogue which was unfortunately rejected. There were many problems with the proposal. The corresponding discussion (https://discuss.python.org/t/_/963 https://discuss.python.org/t/_/963) is worth looking through, despite the length. Installers like Pip could help by offering to install `--in-new-environment` for the first install, and Brett Cannon (a core developer) has done some work on a universal (i.e. not just Windows) "launcher" (https://github.com/brettcannon/python-launcher https://github.com/brettcannon/python-launcher) which can automatically detect and use the project's venv if you create it in the right place (i.e., what you'd have to do with __pypackages__ anyway).

daemonologist 2y ago

For me the vast majority of the pain occurs when _updating_ dependencies. It can be an all-day chore to find and force versions of everything which will work together, and occasionally you'll have an irreconcilable conflict in subdependencies. I have in the past had to hack in a second version under a different name and update all imports in the parent dependencies to match. I know other languages have various solutions for this (to basically have package namespaces local to a branch of the dependency tree), but I don't know how much better that experience is.

tcfhgj 2y ago
10 hours later: Invoking SAT with clause count: 9661561 Invoking SAT with clause count: 5164645
otteromkram 2y ago
Have you tried using pip? Or, reading the documentation? It has more features than just "install." You could try running python -m pip check To check dependencies. Or, python -m pip inspect To get a JSON output of your current virtual environment. Or, update stuff automatically: python -m pip install --upgrade Or, skip worrying about dependencies: python -m pip install --no-deps Or, do a dry run installation with full report on what would be installed: pip install --ignore-installed --dry-run --quiet --report And, there's a lot more than that. Pip is pretty powerful and I'm surprised everyone dislikes it so much. Hope this helps. Cheers!
- zahlman 2y ago
  In principle, yes. In practice, there are a lot of issues with that workflow. For example, `pip install --ignore-installed --dry-run --quiet --report` will build sdists (and run arbitrary code from `setup.py` or other places specified by a build backend) - just so that it can confirm that the downloaded sdist would produce a wheel with the right name and version. Even `pip download` will do the same. I'm not kidding. There are multiple outstanding issues on the tracker that are all ultimately about this problem, which has persisted through multiple versions of the UI and all the evolving packaging standards, going back almost the entire history of Pip. See for example https://github.com/pypa/pip/issues/1884 https://github.com/pypa/pip/issues/1884 ; I have a long list of related reports written down somewhere. A security researcher was once infamously bitten by this (https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html https://moyix.blogspot.com/2022/09/someones-been-messing-wit...).

booleandilemma 2y ago

Having recently used python and poetry - I agree 100%!

TZubiri 2y ago

'Python is mostly a "glue" language' 100% disagree, yes it may be used for that, but it's a fully fledged language, people use it for data science or for backend business logic.

lanstin 2y ago
The data science uses are all using it for glueing a bit of data massage/data gathering into their high performance math/ML libraries. Django users are using it natively, and cherryPy/etc. Python full-stack maybe, but they'll have DB drivers and JSON parsers probably from C.
tcfhgj 2y ago
In data science mostly as glue code by design using frameworks which do the heavy work in native machine code.
- TZubiri 2y ago
  Sure, dependency wise. But the main business logic is in python, how is that python being the glue, more like the brain.

abeppu 2y ago

> Good dependency management means ... > it is possible to safely evolve and update your environment when new versions of dependencies become available. The post, despite its length, doesn't really spend time on this aspect, and I think it's one of the weaker areas. Suppose my library depends on both A and B, and both of A and B depend on C, and in the current version of my code, there's some set of versions that matches all declared dependencies. But A releases a new version which has new features I want to use, and also depends on a newer version of C than B supports. I'm blocked from upgrading A with the normal tooling, unless I either ditch or modify B. This can be a problem in other ecosystems too, but better tools support working around it. In the java world (and I may be out of date here), the maven-shade tool would allow you to effectively use multiple versions of C, e.g. by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need. I think this is also most of the time in principle possible in python (though perhaps some relevant static analysis of seeing what is actually used in more difficult), but the ecosystem doesn't really support or encourage it.

zahlman 2y ago
>I'm blocked from upgrading A with the normal tooling, unless I either ditch or modify B.... I think this is also most of the time in principle possible in python, but the ecosystem doesn't really support or encourage it. No, the main problems with this are at the language level. Languages like Java can do it because the import is resolved statically. Python's `import` statement is incredibly dynamic, offering a variety of rarely-used hooks (start at https://docs.python.org/3/library/sys.html#sys.meta_path https://docs.python.org/3/library/sys.html#sys.meta_path and follow the links, if you dare) that ultimately (by default) put imported modules into a single global dictionary, keyed by name. If you want to work around that, you can, but you'll definitely have to roll up your sleeves a bit. Even if you just want to choose between two different versions at runtime, the base `import` syntax doesn't have a way to specify a version. If the package version isn't explicitly namespaced (which causes more pain for everyone who doesn't care about the exact version) then you need to make special arrangements so that the one you want will be found first (normally, by manipulating `sys.path`). But if you want to use both during the same run, you'll encounter far more problems. The global dict strategy is very deliberate. Of course it offers a performance benefit (caching) but it's also required for correctness in many normal cases. It makes sure that different parts of the code importing the same module see the same module object, and therefore the same module-level globals. That is: in the Python world, it's common that the design of C involves mutating its global state. If you "let B use its older version" then it would have to be an entirely separate object with its own state, and mutating that would result in changes not seen by A. Whether or not that's desirable would depend on your individual use case. A lot of the time, you would want the change to be shared. But then, even if you could propagate the changes automatically, you'd have to deal with the fact that the two versions of C don't necessarily represent their state identically. > by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need. In principle, the B author can make the dependency on C optional ("extra"), for those users who do need that part of B. Or even for B to optionally depend on B-prime, which encapsulates the C-using parts. This is actually not very difficult (and the above-described dynamic nature of Python is very helpful here), but in practice it isn't done very much (perhaps Python developers are wary of creating another left-pad situation). But I do wish e.g. Numpy were more like that. But the "fat-jar" analog is also possible in Python, as long as it isn't a problem that you aren't sharing state between C-old and C-new. It's called "vendoring"; Pip does a lot of it; and the tooling that Pip uses to enable it is also made available (https://pypi.org/project/vendoring/ https://pypi.org/project/vendoring/) by one of the main Pip devs (Pradyun Gedam).

llIIllIIllIIl 2y ago

Python deps are alright unless you’re updating celery.

lofaszvanitt 2y ago

And the drama keeps happening because the article is 70K long...

stcg 2y ago

One of the biggest usability problems with Python dependencies is that the name you import might be different from the name that you use to install the package. So if you find some script on the web that has an `import foo` at the top, you cannot just `pip install foo`. Instead, you'll have to do some research into which package was originally used. Maybe it's named `pyfoo` or `foolib`. Compare that to for example Java, which does not have that problem, thanks to Reverse Domain Name Notation. That is a much better system.

zahlman 2y ago
"install name == import name" cannot work in Python, because when you `pip install foo`, you may get more than one top-level package. Or you may get a single-file module. Or you may, validly per the spec, get no Python code whatsoever. (For example, you could publish large datasets separately from your data science library, as separate wheels which could be listed as optional dependencies.) The lack of good namespacing practice is a problem. Part of the reason for it, in my estimation, is that developers have cargo-culted around a mistaken understanding of `__init__.py`.

jredwards 2y ago

This post feels like it was written 10 years ago. There are plenty of excellent dependency management systems for Python these days. Honestly, the biggest problem is that there are too many ways to set it up.

ac130kz 2y ago

uv (and requirements.txt as a lock) + Debian Docker container (distroless with added packages for runtime). There are no problems with this setup.

sebazzz 2y ago

> You decide to re-install your operating system and vow to go back to doing everything in Excel. Which itself can run Python now. Only in the Microsoft Cloud, not only to rake in that sweet subscription money, but probably also to avoid these headaches.

aszen 2y ago

Everyone seems to praising uv in the comments, seems like no one read this article, it has its drawbacks, it can't install system level libraries and dependencies which many packages depend on.

thebigspacefuck 2y ago
From the docs “For convenience, uv pip install --system will install into the system Python environment.” Are you referring to something else?
- aszen 2y ago
  No I'm referring to system level libs, that you need to install via your operating system package manager like `apt` for python packages that depend on them.

cr125rider 2y ago

Docker and Devcontainers. You can make a mess in your own little box and not mess with my system. We use conda/momba to install some more complex dependencies that are piles of shared libraries and a messy web of sub dependencies.

VeejayRampay 2y ago

really sucks that we're stuck with python when languages of the same realm (JS and ruby for example) are so much better (as languages and with regard to the associated tooling)

sgarland 2y ago

One easy trick they don’t want you to know about: stop importing dependencies. You probably don’t actually need numpy. You probably don’t need requests. The stdlib is incredibly broad. This doesn’t always work, of course (especially for large projects), but for smaller ones, it absolutely does.

nothrowaways 2y ago
Or copy entire numpy and requests repos with your code, they don't want you to know this either.

egberts1 2y ago

Biggest turn off of all is the lack of centralized (if any) search engine for Python packages at CLI-level `pip` command.