6 ms·
Cooperative Package Management for Python
- korijn 5y agoI'm thrilled to see people adressing this (decades old?) footgun. I hope there will be more collaboration in the future. The split universes system and language specific package managers operate in are a cause for much more issues.
- chriswarbo 5y agoEww. I get the rationale, but Python's packaging/import logic is already ridiculously convoluted. The underlying problem is using global mutable state (e.g. /usr/lib/.../site-packages). Giving each application its own package directory is better; there are Python-specific solutions to that (like virtualenv mentioned in the article), but I prefer Nix since it's language-agnostic (it can handle non-Python dependencies too). I'm also not a fan of all the Rube Goldberg machines being cobbled together to appease Docker's fundamentally broken way of working: > Distros that produce official images for single-application containers (e.g., Docker container images) should remove the EXTERNALLY‑MANAGED file, preferably in a way that makes it not come back if a user of that image installs package updates inside their image (think RUN apt‑get dist‑upgrade). On dpkg-based systems, using dpkg‑divert ‑‑local to persistently rename the file would work. On other systems, there may need to be some configuration flag available to a post-install script to re-remove the EXTERNALLY‑MANAGED file. Here a "single-application container" is assumed to contain an entire OS rather than, you know, a single application. That was supposed to make dependency management easier, since we can tailor the OS to just that one application; but based on this article it sounds like even that was a lie. Should we revisit the assumption that Docker makes dependency management easier? No, the OS maintainers are now expected to change their distros, to add another layer to the Rube Goldberg machine. I also find it pretty alarming that 'apt-get dist-upgrade' is given as an example of something we might want to do to an "official image". What's the point sanctioning a snapshot as "official" if we're just going to immediately, and non-determinisitcally, overwrite arbitrary parts of it based on external-server-state du jour?
- BiteCode_dev 5y agoWhile I agree about Nix being a cleaner solution, it will never be adopted, because... it's a cleaner solution. Nix is pretty much incompatible with everything, because our entire legacy of software stack has state, and we need to accommodate for it. So we keep adding those hacks because we favor compat above purity. Even nix has to, after all, there plenty of bash calls in NixOS config scripts as soon as you do something non trivial.
- chriswarbo 5y agoSure, but I'm not saying everyone should adopt Nix; I'm saying the solution to problems caused by global mutable state is to avoid adding more global mutable state. If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, and (b) why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.
- BiteCode_dev 5y ago> If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, debian packagers won't use pip for obvious reasons. dev need pip because it's portable, and doesn't need packagers validation. It would have been better if something like nix would have been adopted 30 years ago by both community, but it hasn't been so we have several packages managers. It's even worst now with poetry, conda, snap, flatpack and docker. > why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies. pip now uses wheels, and basically does that. Whl files are zip, and pip just unpack it at a know location. The problem is, this location is currently shared with the OS if you don't use --user or a venv. This article address that problems. My other comment also talk about some other solution. It's very hacky, because, well legacy and all that.
- BiteCode_dev 5y agoIt's a good safeguard, and it's going in the direction of the other initiatives to make python package management default behavior saner. PEP 852 is the another one to follow up: https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/ It basically uses the concept of node_modules, making python interpreters load any local directory names `__pypackages__` . There are 2 differences though: • Unlike JS, python can only have one version of each lib for a given setup. • Since having several versions of python often matters, you may have several __pypackages__/X.Y sub dirs to cater to each of them. It also forces you to use "-m" to call commands written in Python, which is the best practice anyway. I hope it will push jupyter to fix "-m" on windows for them because that's a blocker for beginners. If you are not already using "-m", start now. It solves a lot of different problems with running python cli programs. It's an old flag, but too few people know about it. E.G: instead of running "black" or "pylint", do "python -m black" or "python -m pylint". Or course you may want to chose a specific version of python, so "python3.8 -m black" for unix, or "py -3.8 -m black" on windows. To test out the __pypackages__ concept, give a try to the pdm project: https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm At last, some other tools that I wish people knew more about that solves packaging issues: • pyflow (https://github.com/David-OConnor/pyflow https://github.com/David-OConnor/pyflow): it's a package manager like poetry, but it also installs whatever python you want like pyenv. Except it provides the binary, no need to compile anything. It's a young project with plenty of bugs, but I wish it succeeds because it's really a great concept. Give it a try, it needs users and we would all benefit from it becoming popular as it's a very sane way of setuping a python dev env. • shiv (https://shiv.readthedocs.io/ https://shiv.readthedocs.io/): it leverages the concept of zipapp, see PEP 441 from 2013, meaning the ability that python has to execute code inside a zip file. It's a successor to pex. Basically it lets you bundle your code + all deps inside a zip, like a Java .war file. You can then run the resulting zip, a .pyz file, like if it were a regular .py file. It will unzip on the first execution automatically and run transparently. It makes deployment almost as easy as with golang. • nuitka (https://nuitka.net/ https://nuitka.net/): takes your code and all dependencies, turns them into C, and compiles it. Although it does require a bit of setup, since it needs headers and a compiler, it results reliably in a standalone compiled executable that will run on the same architecture with no need for anything else. Also it will speed up your Python program, up to 4 times. In my experience, it's also easier and more robust than pyinstaller, cx_freeze and so on. • pyodide (https://pyodide.org/en/stable/ https://pyodide.org/en/stable/): python compiled to WASM to run in the Web browser. Useless for web programming given the huge size of the runtime, but great for teaching, as it allows students to basically access a zero install full featured python dev env by clicking a link. Try it out, it's awesome, you can even create and query a sqlite db, thanks to the virtual FS: https://notebook.basthon.fr/ https://notebook.basthon.fr/
- derriz 5y agoI probably haven't thought this fully through but wouldn't it be simpler to just have a system venv - root protected - perhaps distributed using the system's package manager? Then if you mess up site-packages at least you wouldn't break the system tools.
- BiteCode_dev 5y agoIt's already the case, that's why you see some mistaken tutorials telling you to "sudo pip install". A venv is just a dir and some path, and your os already have some dedicated to their own python install, even if it's not called a venv, but something like dist-packages + manual PATH fudging. This proposal makes the distinction clearer by putting safeguard to NOT to mess up with the system stdlib. You should be using "--user" or a local venv, and "-m" to call commands. See my other comments for more tooling.
- derriz 5y agoI'm missing something I guess - I don't understand your comment in the context of this claim near the start of the article: "The root cause of the problem is that distribution package managers and Python package managers ("pip" is shorthand to refer to those throughout the rest of the article) often share the same "site‑packages" directory for storing installed packages." Having a system venv - managed by the system's package manager - would mean this "root cause" would go away, no? Of course by poking around the filesystem tree and messing with managed files as root, you could mess up the system venv but that's possible with any installed package.
- faho 5y ago>Having a system venv - managed by the system's package manager - would mean this "root cause" would go away, no? Oh, but you need to explain to pip that it's managed by the system's package manager. Which is what this does with the "EXTERNALLY-MANAGED" file. And you need to do that because the system's package manager and pip have separate sources - with `pip` you get the packages from pypi, with the system's package manager from its repo.
- JonathanBeuys 5y agoOn my computer, I only use Python software that is in the Debian repos. So all dependencies are handled by Debian. Everything else I run in a Docker container. This way I never need a venv. I hope these changes will not make my workflow harder. Running 3rd party software in containers not only makes dependency handling easier. It also is a security thing. Can you really trust the whole dependency tree below any python code that you want to run? I prefer to keep it separate from my main OS.
- BiteCode_dev 5y agoNo, it fact it will make your workflow easier: no matter what, no script using pip will break your install.
- JonathanBeuys 5y agoI don't know if I actually have something you would call an "install". I run 3rd party software in fresh Debian containers. So there is nothing in there I would be afraid to "break".
- BiteCode_dev 5y agoThe container has a system python stdlib, and it's used by apt. If it breaks, everything breaks. I've destroyed it twice, with style I must say. If any script, or yourself, run "pip install" using admin rights (the default on a lot of container techs), you have a chance to break it.
- JonathanBeuys 5y ago"Chance to break" what? It is not as if I manually work in a terminal inside a container that is somehow permanent and valuable. I just once create a dockerfile with the commands to set up the application I want to use. That's it. From then on, I use that container whenever I want to use that application.
- 5y ago
- deleted 5y ago[deleted]
- gigatexal 5y agoWhat’s still annoying to this day is Python imports. I have been doing Python for a while now and I still get tripped up when some bit of code in a folder isn’t importable. I’ve found adding the package dir that I’m working on to PYTHON_PATH works but then my editor doesn’t resolve packages for autocomplete.
- georgyo 5y agoIf you have foo.py and bar.py in a directory and you want to import foo.py from bar.py you only need to add an empty __init__.py You would then `import .foo`
- terom 5y agohttps://www.python.org/dev/peps/pep-0420/ https://www.python.org/dev/peps/pep-0420/ Python 3.3+ no longer requires the `__init__.py` file to make a package. > You would then `import .foo` The `import .foo` is a syntax error, and `from . import foo` does not work for scripts [1]. The basic `import foo` works if the directory is on your PYTHONPATH. That will be the case if you run `bar.py` as a script. It gets more difficult if you want to separate scripts and importable modules into separate sibling directories (bin vs lib). You can't use relative imports to get around that, you need to use a virtualenv (or manual PYTHONPATH wrangling). [1] https://stackoverflow.com/a/14132912 https://stackoverflow.com/a/14132912
- BiteCode_dev 5y agoUse virtualenvs, that's the problem they solve. Python now ships with the venv cli tool. If you point your editor to the python interpreter in the venv, it will detect everything automatically.
- icegreentea2 5y agoAs sibling says, you really should use virtualenvs. But if you're not and you're using vscode, this is the relevant thingy to do to get vscode to play nice: https://code.visualstudio.com/docs/python/environments#_use-of-the-pythonpath-variable https://code.visualstudio.com/docs/python/environments#_use-...
- psd1 5y agoI don't understand why dependency management is hard in python. In powershell, there's a PSModulePath which works like Path: search folders for dependencies. This can be modified within the session. You import dependencies by specification, which might just be the name or may contain version constraints and/or a module guid. Granted, powershell only got a package manager in the last decade. Not long enough to thrash out a contentious pep.
- kortex 5y agoPython is an odd duck in terms of dependencies because it gets very integrated into the system; think of it almost like "portable scripted C." There's the pure python packages, which because of how python's search and import process works, means you need a separate library root for each set of incompatible dependencies, and these must be orthogonal. Then, you have compiled extensions, which have the above issues, plus now you gotta get some binary distribution machinery involved. Then, you have FFI/so/dll/dylib type extensions, which have the above complexities, plus the linker/LD_LIBRARY_PATH involved. Then, you have the interaction of site-packages, since python is used by my OSes for scripting.
- mark_l_watson 5y agoThis sounds like an improvement. In my work related laptops, I still favor conda because it seems to do really robust dependency checking. I do many short experiments that are usually not work related (in the sense that it is not paid for work). I strongly favor spinning up fresh VPNs for these experiments and when I need Python (and often Hy/hylang, a Lisp syntax for Python), I usually just use the system python3 installation and use “—user” for pip installs to my home directory. There are probably many reasons not to do this, but I justify this bit of sloppiness because I tend to discard VPNs fairly quickly.
- vlmutolo 5y agoDo you purchase a license for Anaconda? Last time I checked, it seemed like conda required a license for professional use. I may have misread it though.
- deleted 5y ago[deleted]
- mark_l_watson 5y agoNo, I have never purchased a license. If a paid for license was required for my personal laptops (which I use professionally) then I would probably just use virtualenv.
- pwang 5y agoYou only need a commercial license under certain circumstances, and it only pertains to accessing the default package repository at repo.anaconda.com. You can still use conda-forge etc. Conda itself (the packaging tool) remains open source. Part of the proceeds from the cost go to fund the Anaconda Dividend Program, which gives money to various open source projects in the pydata/scipy ecosystem.
- hitpointdrew 5y ago>Using virtual environments with pip, instead of installing packages system-wide, has been the recommended practice to avoid conflicts with OS-installed packages for quite some time. But it is not generally mandatory, so users sometimes still run into problems. I use virtualenv any time I use Python. That said, it would make way more sense if OS-installed packages used virtual environments, so that a user can pip install whatever they want on the default python install and not have to worry that it could bork dnf or any other system utility.
- chriswarbo 5y agoThere shouldn't be a "python install" at all, let alone a "default". Whether an application uses this-or-that Python interpreter, or Ruby/Bash/Scheme/whatever should make about as much difference as whether a C program was compiled with GCC or Clang: it's an implementation detail, which doesn't need to be co-ordinated with everything else on the OS (although the package maintainers may want to coordinate, e.g. to allow re-use of files via soft-/hard-links, bind-mounts, etc.). When I "install foo-1.0 into my python environment", that's an annoying implementation detail (which is also imperative and non-atomic, i.e. there is a time where the python environment exists without foo-1.0). What I really want is "run a 'python' command where 'import foo' will have foo-1.0". That shouldn't break other applications which just-so-happen to use foo-0.5, or whatever.
- uranusjr 5y agoThis is similar to the static vs dynamic linking debate. It’s easiest to deploy things if they are all self contained, but Linux distros have a strong tradition of dynamic linking and really don’t want to have (say) two separate paycopg2 installations if they can be shared. And once there’s a place to share stuff, users are going to inject new stuffs into it and give the package manager all sorts of trouble.
- cycomanic 5y agoWhy do all languages have to reinvent packaging and typically do it poorly. If instead pip and co could be configured to output an rpm or deb file which would then be installed everything would work much easier. Instead we have to deal with the mess that's virtualenvs, which causes python devs to often be way too specific in version specs, which only causes the mess that virtualenvs are trying to solve. Seriously the only sane way language package managers should work is by hooking into the system package management, maybe with an option to install in the user directory.
- binarybanana 5y agoFWIW for portage (Gentoo) there is g-sorcery[0], which can create ebuilds for Emacs (m/elpa) and python packages automatically. Similarly there is also cargo-ebuild[1] which can create ebuilds for rust programs/libraries, including a list of all dependencies with hashes. I've successfully used cargo-ebuild in the past to create ebuilds automatically, it's a breeze. I'd be surprised if similar tools didn't exist for deb/rpm based distros. I'm always surprised how many problems people have with python on other distributions. On Gentoo I can install multiple versions at the same time, install libraries for any of those or just a specific python version of my choosing (including pypy), set a default python implementation (pypy) and of course I can pick the version of the libraries I need, too. It's not as flexible as nix, but quite close. Not a python dev though, so YMMV. [0]:https://github.com/jauhien/g-sorcery https://github.com/jauhien/g-sorcery [1]: https://github.com/cardoe/cargo-ebuild https://github.com/cardoe/cargo-ebuild
- maple3142 5y agoBecause many system package managers doesn't support multiple version of a same package easily. Also, there are system (used to) without a package manager. > causes python devs to often be way too specific in version specs I don't think anything could fix that, because it is totally up to library developers to decide whether they want to make breaking change or not. So being able to use different exact version of dependencies for different projects are important. BTW, I find it annoying that many distro package manager doesn't make it easy for user to install package for themselves without root. Even Nix requires you to create /nix IIRC.
- uranusjr 5y ago(Disclosure: I’m one of the PEP authors.) I want to clarify a point since many comments seem to go “I do container/Nix/virtualenv so this is not a problem”. That’s absolutely correct, nothing will ever go wrong if you do any of those things. This is a solved problem. But there are a significant number of people that do not do any of these and insist on installing things into the system-provided Python and expect smooth sailing. This is the “problem” for pip and distro maintainers, and what this PEP aims to solve, to guide people toward one of those solutions.
- disgruntledphd2 5y agoThank you for your work on this! Some context, I'm a data scientist who grew up with R, and run Linux. When attempting to learn Python, this kept causing me massive problems as I'd have a weird mixture of system installed and pip installed packages. I think it probably stopped me from really getting into Python for 1-2 years. So, I think this work is really, really important and I'm really glad someone is working on it.
- jjgreen 5y agoIt's too late now, Python has been destroyed by its own packaging. Burn it to the ground, then burn the ashes, then bury them. Then start a clean rewrite of the language. We will agree never to speak of this episode again.
- curryst 5y agoI don't think it's that catastrophic. I think they should have forced a hard cut with Python 3, but adoption there was hard enough already so maybe they made the right choice. Another part of the issue is that it makes system packages worse. The system Python works fine if, and only if, you use some kind of external package manager. Then ensuring that the libraries you have work with all the applications you use is up to the package manager, which works okay. If we burn that down and force virtualenvs, system packages become less optimal. Packages from package managers will either have to bundle their own libraries (meaning duplication and larger package sizes), or we have to have a "system virtualenv" where we manage those packages. A "system virtualenv" is basically what we have now, except there would be more steps to actually be able to use it. What we really need is for OS package maintainers to take a hard line that people should not be installing packages into the system Python without using apt/pacman/yum/dnf/whatever. Attempting to run "sudo pip install $thing" should print an error like: "Installing packages into the system Python without using yum is unsupported. You should use a virtualenv or alternative tooling to prevent potentially system-breaking incompatibilities. See https://example.documentation.site.com https://example.documentation.site.com for more information. Add the --i-dont-care-if-im-unsupported flag if you're sure you know what you're doing and want to proceed anyways." And I really think it should come with a scary and long flag like that. It empowers a lot of engineers to make a case that they need to get away from that pattern quickly. Pip is packaged (at least I know it is for Ubuntu), so the package manager can definitely add code for that. I don't think we need to stop people from shooting themselves in the foot, but there are not nearly enough warnings that you shouldn't be doing this. I don't think there are any currently, even though I think it's basically unanimous at this point that it's technically better to use virtualenvs. I don't even think it's something that linters check for.