14 ms·
Python setuptools v50 breaks pip installation on Debian/Ubuntu
- XzAeRosho 6y agoHow Python dependency management can be improved at this point? Should we hold our breath for Python 4 to solve it?
- ddavis 6y agoThe current paradigm feels OK to me. Unfortunately there are a lot of old crappy "intro to Python" blog posts and tutorials still telling people to use 'sudo pip install' to install packages on their system Python. pip/setuptools do not support that workflow. I think it's slowly getting more widely recognized.
- dheera 6y agoIt seems to me like Python 3.8 has broken a LOT of stuff, violating semantic versioning. Like, for example, RealSense cameras don't work anymore.
- nuclear_eclipse 6y agoPython has never used semantic versioning. There is a long history of small, breaking changes between "minor" releases of Python. Eg, Python 3.7 made `async` and `await` global keywords, breaking any case where old code used `async` as a variable name.
- eesmith 6y agoIf I understand https://github.com/IntelRealSense/librealsense/issues/6296 https://github.com/IntelRealSense/librealsense/issues/6296 and https://github.com/IntelRealSense/librealsense/issues/5777#issuecomment-582480988 https://github.com/IntelRealSense/librealsense/issues/5777#i... correctly, what happened is the RealSense developers don't provide a pre-compiled version for 3.8 on PyPI. Python is "broken" in that the C API does not maintain backwards compatibility. (See https://www.python.org/dev/peps/pep-0620/#the-c-api-blocks-cpython-evolutions https://www.python.org/dev/peps/pep-0620/#the-c-api-blocks-c... for a proposed solution.) RealSense is "broken" in that they don't provide builds for all the versions of Python their users use. But that's because "Officially, Librealsense is only currently supported up to Ubuntu 18.04 LTS" Perhaps also your process is "broken" for depending only on pre-compiled builds; the published solution is to build from source. https://github.com/IntelRealSense/librealsense/issues/6296#issuecomment-644291191 https://github.com/IntelRealSense/librealsense/issues/6296#i... says it's possible. Or "broken" for using an unsupported environment and expecting it will work. Software in general is broken. It's amazing that it works as well as it does.
- dheera 6y ago> the published solution is to build from source I tried that, it didn't work, and they keep closing issues about it instead of addressing it. They are a company with a 200B market cap, and it's already 5 months since the release of 20.04 and they still can't get their drivers straight. People like me want to use their hardware. I don't have time to be debugging their drivers -- I have other work to do. I don't think it's unreasonable ask for them to support the latest LTS release of the most common Linux distribution in a timely manner with pre-compiled binaries that just work, in return for the $179 that I and tens of thousands of others paid for the device. This isn't some open source community software that I'd be glad to pitch in some time and help debug. This is a driver for a device that I paid a for-profit company for and I expect timely support especially for an exceedingly common OS that I use.
- eesmith 6y agoYou wrote "Python 3.8 has broken a LOT of stuff", as if it were only Python's problem. Clearly other things are broken too.
- neckardt 6y agoI'm one of those people using 'sudo pip install'. What should I be using instead?
- CapriciousCptl 6y agoPython -m venv in your project directory. Then source ./bin/activate. Pip install from the project directory as your regular user (no sudo). You can also set up venv (virtual environments) in home or somewhere else instead of for each project. This is called python virtual environments and very easy to adopt.
- ydnaclementine 6y agoRuby has this figured out well by comparison. Install the bundler gem (which you're going to be installing anyway to get dependencies), cd into the project, and prefix your commands with `bundle exec` to use project gems Remembering to start/stop virtualenv and remembering the names of your virtual envs is one too many steps
- easton 6y agoOr just use poetry/pipenv shell (or poetry/pipenv run for one-off commands). Both provide a good wrapper, although poetry seems a bit more stable these days.
- thewrinklyninja 6y agoPerl has copied this as well with Carton. Works really well and as standard keeps the modules in the project dir. https://metacpan.org/pod/Carton https://metacpan.org/pod/Carton
- viraptor 6y agoIf you prefer "bundle exec foo", then "bin/python -mfoo" is pretty much the equivalent and doesn't require running activate before.
- kevin_thibedeau 6y ago
- corty 6y agoJust don't use pip, use apt. What isn't packaged isn't worth it anyways. Non-distro package managers, a dozen per language are a disease.
- Joker_vD 6y agoI think you meant to write "just use pre-built Docker images".
- mistrial9 6y agothe reality is that there are binary needs at the OS link level, and then there is pure-python or ML with drivers or whatnot, per user.. its not actually easy to divide which is which ..
- im3w1l 6y agoIf a python tool wants to be in apt, then it can only use apt dependencies. So that's something to think about.
- jnwatson 6y agoapt is completely inappropriate unless you're using 3 year old dependencies. Ubuntu 20.04 (released in April) doesn't have Python 3.8, which was released in October 2019.
- pritambaral 6y agoThis is patently false; the default python3 on Ubuntu 20.04[1] is 3.8.2. A separate package python3.8 is also available in built-in repositories for all supported versions of Ubuntu going back all the way to 18.04[2]. 1: https://packages.ubuntu.com/focal/python3 https://packages.ubuntu.com/focal/python3 2: https://packages.ubuntu.com/search?keywords=python3.8 https://packages.ubuntu.com/search?keywords=python3.8
- avhon1 6y agoApt is perfectly able to deliver recent versions. Here is Debian's official Python3 package, currently version 3.8.2-3: https://packages.debian.org/bullseye/python3 https://packages.debian.org/bullseye/python3
- akersten 6y agoHonestly, just throw everything in a venv and forget anything about needing to manage platform versions again. Unless, of course, virtualenv can't be installed on the host in the first place due to setuptools/pip issues :)
- abiogenesis 6y agoPython comes with built-in venv support since 3.3: https://docs.python.org/3/library/venv.html https://docs.python.org/3/library/venv.html
- AlphaSite 6y agoDeviant packaging breaks/removes this.
- erfgh 6y agoThat's why it's deviant.
- Conan_Kudo 6y agoI assume the poster meant "Debian packaging", but yes. Debian and Ubuntu remove venv support along with distutils from the standard library by default.
- regularfry 6y agoThis is annoying, but it does encourage the principle of "the system python is for the system, and only incidentally for the users of that system". I'm almost always in a non-system Python for anything interesting, unless I'm writing something to integrate with the OS.
- Conan_Kudo 6y agoIt breaks interrogation of the Python interpreter, since those functions are part of distutils. It also breaks creating virtualenvs if it relies on the venv module.
- pdonis 6y agoI have never really understood how Python dependency management got broken in the first place. The basic model is the same as the basic Unix shell model: you have a path, which is just a list of places where executable thingies can be found, and whatever wants to execute the thingies looks in each place in the list, in order, until it finds the thingie it wants, or it bails out with an error if it's gone through the entire list and can't find what it's looking for. In a Unix shell, the list is $PATH (or maybe $LD_LIBRARY_PATH, since most Python dependencies are libraries); in Python, it's sys.path. Installation of an executable thingie then is simply a matter of "make sure this ends up in one of the places listed in the path". What is so hard about this?
- misnome 6y agoWhat about different packages with different version dependencies?
- brendawalsh 6y agoSo, my most recent success has been using pyenv and pipenv. I install pyenv first, then install any specific point releases I need, then switch to the python version I want to use, and then use pipenv to install dependencies or run commands in the virtual environment.
- zelphirkalt 6y agoI think lately pipenv also interacts with pyenv to do this automatically.
- pdonis 6y agoVersion numbering and labeling of the packages--the same way Unix solves the same problem with shared libraries. A program that wants a specific version of a library looks for it by specific version.
- Joker_vD 6y agoI remember there was time when pip3 used to break apt... or was it the other way around? Or maybe it was pipenv? Whatever that was, it was pretty baffling to see python's stacktrace of an ImportError when you tried to run "apt whatever".
- dheera 6y agoAnd then there was pip2, and now "pip" on my system actually runs pip3. It's a mess.
- josteink 6y agoWhile I'll be first in line to criticize the overall Python packaging-story, you also have to consider how Python itself is being used to implement the underlying platform which the OS's package-manager depends on. If you allow the pip to make changes to things already present on your system, it make break expectations other parts of your OS may have. And especially given Python's messy package-management story, combined with the less-than-ideal Python2 to 3 compatibility-story, transitioning a full OS and platform from Python 2 to Python 3 without breaking stuff is no small task. I'm amazed it even works, and that's before you allow users to mess around with packages using pip.
- Waterluvian 6y agoIt broke docker-compose today which messed up some robot deployment stuff for an hour. Lesson learned by that team: find a way to hard freeze the entire dependency tree or don’t use a tool.
- peterwwillis 6y agoImmutable Infrastructure as Code. Docker containers are the modern gold standard. You can get fairly far on a stock OS with a combination of vendored binaries/libraries and virtualenv/pip, but containers are much simpler.
- Waterluvian 6y agoI dunno if your plan is containers all the way down but for us it was literally docker-compose that was broken. More important than immutable containers is repeatable builds. I need to be able to go back to a version in 12 months and build the exact same output avec a few changes. Containers don’t help with that.
- zelphirkalt 6y agoI am quite fed up with people thinking: "Oh, I'll use docker compose, it'll ve simpler." adding another layerof abstraction on top of something that does not need another layer and the having issues, but not documenting their precise docker run commmands, so that I need to reverse engineer them from the docker compose yaml files.
- peterwwillis 6y agoLiterally any change in your system can break. It matters what you do once it breaks, and what I describe makes the whole thing much more repeatable and easier to recover from. Docker containers builds are extremely repeatable. You run some software called Artifactory, which keeps the built versions of your artifacts, including base image tags. You store your applications in there, as well as the Docker containers that you built. If you need to go back a version, you pull it out of Artifactory. This is industry standard at this point.
- pyuser583 6y agoWell, shit.
- uranusjr 6y agoFor context, the root cause to this is actually in Debian, not setuptools (or pip). Debian vendor-patches the stdlib distutils, which setuptools used to depend on, but lost in v50 since it does not depend on the stdlib distutils anymore. This issue was known for quite a while, and IIRC more recent Debian (and thus Ubuntu) versions contain a fix. But the change was not backported for other reasons I am not personally familar with.
- Iv 6y agoWhile we are at it, can we finally get rid of python 2? And find some sane way to not need python3.6 python3.7 and python3.8 laying around and installing each their own libs versions?
- simcop2387 6y agoI have done this on my laptop running debian nextstable, so it is happening and will be the default soon.
- Iv 6y agoFinally a good news!
- xorcist 6y agoPython has to implement all the same mistakes Java did. All they have to do is release a new major version twice a year and then they're done. Sorry, just joking (I hope). Also, we didn't have to suffer UTF-16.
- jnxx 6y agoA few days ago, I learned that the Python interpreter does not specifically uses semantic versioning. Which means that a change in the minor version number (like 3.8 to 3.9) could break backward compatibility.
- Iv 6y ago"could" but does it, actually? The breaking changes they did from 2 to 3 caused enough headaches that I think no one wants to go through that again.
- misnome 6y agoThis update seems to have broken things all over. Our CI conda-based builds all broke overnight as Conda-forge deployed, and then undeployed v50.
- rjeli 6y agoIMO, either use system python and system python packages (`apt install python-opencv`), or if you need pip packages that aren’t debian packaged, use pyenv python which takes care of all the pip prefixes. Mixing the two has never gone well for me.
- ohazi 6y agoThat's not always possible. If you install a package from the system repository that depends on a bunch of python packages, the dependency manager will only be satisfied if they're installed via the repository, since it can't see your pip packages. So you might need the latest python-opencv from pypi for work, and some desktop camera app needs the one from apt to not have broken dependencies.
- Conan_Kudo 6y agoYou can create virtualenvs that can "see" system-installed Python modules, so this strategy does in fact work. System modules can be used as dependencies for non-system applications with this.
- viraptor 6y agoYou're not going to run into this issue though. Apt-installed python packages will see each other and pip-installed venv packages will live in a separate area. They don't conflict.
- laingc 6y agoThe first thing I do (or rather, ansible does) on any new PC or server is install Miniconda to a location inside my home folder, and set it as the first item in my PATH. I’ve been down most of the python installation and packaging rabbit holes, and this practice has served me well.
- ur-whale 6y agoThe fact that anaconda actually has to hack your path variables and your bashrc doesn't strike you as a recipe for disaster?
- Redoubts 6y agoYeah it always did squick me out that it needed to do that, while also having a special binary manage all the commands. Explicitly, why is 'conda' in my path, but then I need to run 'conda init $shell' to make it really work. It should be handling that internally all by itself, without any help...
- laingc 6y agoI mean, no more so than having poetry or pyenv do some special magic to hopefully give me the right environment? Maybe it’s more brittle than I’m imagining, but I’ve been doing this for years and have never had it break.
- fractalb 6y agoI think the maintainers thought this is only Debian/Ubuntu problem, but in reality, it affects a lot of other systems. Fedora, Conda distributions seem to have been affected too.
- echelon 6y agoIf there is ever a Python 4, the number one goal of the project should be to make Python completely hermetic, repeatable, and redistributable. And there should only be one way to do it. pip, requirements.txt, and venv are madness. Look at how much suffering they cause! I was dealing with (different) pip issues today myself, and it's not an infrequent occurrence. Rust's Cargo is a good reference point.
- turbonoobie 6y agopipenv solves some of the chaos. I do agree though. One right way of setting things up would do wonders.
- zelphirkalt 6y agoAnd pipenv is getting better. Recently thete has been a push of improvement. It still has its issues though. Pipenv or Poetry? Time will tell. Poetry needs to parallelize.
- pydry 6y agoLast time I tried to use pipenv it was a mess. Poetry worked fine though.
- ohazi 6y agoWasn't pipenv abandoned for like a year or two? Is it actually back now?
- wirrbel 6y agoThe original author has a habit of abandoning software projects after a while. But I think they managed to get it into the pypa docs (python packaging 'authority'), and so I guess someone has brought it back.
- 6y ago
- bregma 6y agoPip is already seriously broken, it's just that the breakage has hit one of the main targets for the project. Part of my day job is maintaining Python for an embedded OS. For self-evident reasons, software for the embedded OS needs to be cross compiled, and Python makes it impossible to cross-compile pip. Explicitly. Upstream's reaction is "you can't cross compile pip, you're on your own." To make matters worse, pip doesn't support non-native Python once it is built. Of course, this is just the tip of the Python portability nightmare. Python is great and completely portable as long as your target is Linux running on x86_64 or aarch64. It has second-class support for Mac OS and Windows and there are private port projects for a few other system but they generally require heavy patching. For example, the Python code violates POSIX standards in favour of assuming that everything is Linux (hey guys, time_t is not a signed integer that can be used for arbitrary arithmetic!). Executing a whole lot of system code between fork() and exec() is always broken, although it appears to mostly work on Linux as long as you disable certain kernel features, and guess which code base relies on that? Yeah. Pip is broken and now it's showing up on Debian too. Maybe it'll just get better by itself?
- sfoley 6y agoPip is a pure python library, you don’t compile it.
- w_t_payne 6y agoAnd this is why I vendor every single dependency manually and build them all from source.
- bumblebritches5 6y agolol.
- jnxx 6y agoFrom the github issue: > In short, packaged Debian/ubuntu Python installs things into dist-packages. site-packages is left alone in-case you like to build your own upstream python and install it. Debuntu overrides the distutils it ships to redirect installs to dist-packages. It gets blurry when you install pip from upstream. But it would call setuptools, which would use the patched distutils so things worked. > > Now setuptools vendors distutils this and always installs things into site-packages, which isn't on the packaged python interpreter path, so it can't find modules. Now, who thinks that vendoring distutils is a good idea? And installing into the base system?
- johnchristopher 6y agoThere are things I can't uninstall or install because some needs python2 and others python3 and uninstall python2 or python3 removes one or the other (notably calibre, and I just read the author won't move to py3 and that he can maintain py2 anyway). For instance I can't install mycli without removing calibre. Of course it's most likely possible to get around that with virtualenv or whatever is used at the moment but that's a huge cognitive load just to run some apps.