4 ms·

Just an en-passe mention (under "Documentation") to the new Bookworm way of handling Python modules which already pissed me off^W^W^W surprised me in Ubuntu 23.

Just an en-passe mention (under "Documentation") to the new Bookworm way of handling Python modules which already pissed me off^W^W^W surprised me in Ubuntu 23.04. Basically you cannot install modules with `pip3 install module` (or `python3 -m pip install module`) without using a virtualenv or using the .deb packaged version in the repos. I've got used to it but it was a headache for a few days after the first upgrade I did.

extraduder_ire 3y ago
Can you use ensurepip to install pip, and then use pip from there? I recently had to deal with missing pip on the last ubuntu LTS version, and that's what I had to do. Makes some amount of sense to stop novices messing up their python install right off the bat, much like I have on most of my machines.
imp0cat 3y ago
I think `pip install --break-system-packages` will still work if you insist on mixing pip and deb.
josephcsible 3y ago
> you cannot install modules with `pip3 install module` (or `python3 -m pip install module`) without using a virtualenv or using the .deb packaged version in the repos. Wasn't that always a really bad idea to do, though, since it was so easy to break your system Python that way?
- darkwater 3y ago
  I'm not saying it was the best way but if you just ran some small 3rd party script or wrote some small Python script with non packaged deps, well, it was easy, fast and usually safe. But forcing this new behavior is surely a good decision.
- deleted 3y ago
  [deleted]
- asddubs 3y ago
  I'm fairly ignorant of the python ecosystem beyond extremely casual use for a few scripts, how does it break your system to do this?
  - sjsdaiuasgdia 3y ago
    There's a few ways Python packages can be installed. They can be installed at the system level, making the package available to any Python program on the system. This can get problematic if not all of your Python usage can function with the same versions of all system-installed packages. They can be installed at the user level, sitting in your homedir instead of somewhere under /usr. This gives some isolation from the system defaults, but may just move the problem down a level - you still have the potential for version conflicts. They can also be installed at the virtualenv level, and now the package install is specific to Python programs launched from that virtualenv. Upgrading a package can only break stuff in that virtualenv, it can't spread beyond it. If you mostly use stuff from the standard library and stable packages that don't change their APIs much, you can go a long way without venvs before it actually bites you.
    - ramses0 3y ago
      The thing that _really_ irritates me about debian (almost the only thing, actually) is how flipping hard it is to use it as a development environment/system. It's _great_ as a system, but for the love of $DIETY, why isn't there some sort of way to say: `sudo apt install $ALL_THE_COMPILERS@[latest||version]` ...semi-similarly to the "exceptions" made for frequently updated packages like `firefox`, `chromium`, etc... I know of `asdf`, but it seems like a big miss that it's sometimes so hard to use debian "properly" (ie: via `apt ...`) to actually develop. Or similarly, wouldn't it be great if similar to `LSP-language-server-protocol`, there was something like `PSP-package-server-protocol` where `npm`, `go.mod`, `pip`, etc... were all like: "yeah, the debian system packages for python, perl, etc. are great for debian-on-debian", but we should also support pulling an arbitrary project at an arbitrary version and make a best-effort attempt at being able to isolate, share, manage, upgrade packages between them.
      - 3y ago
- eternityforest 3y ago
  Yeah it's insanely easy to break stuff.
adql 3y ago
That's a very good change. You should never have a reason to vomit your app's deps into system, far too easy to cause problems on upgrade
elsjaako 3y ago
Do you know if `pip3 install --user module` still works?
- Phrodo_00 3y ago
  Suse Tumbleweed made the same change and it doesn't work there (I use it through WSL if it changes anything)
geerlingguy 3y ago
You can by deleting the EXTERNALLY-MANAGED file [1] I know some Python maintainers think users are too dumb to work with system Python, but there are also some packages that are really hard to deal with on a system level if you try installing them in a venv. Plus venv's are dumb (I use Docker if I need more than one environment for specific apps or dev work), since it's another layer of abstraction I'd need to learn just to use Python. I'm not forced to use virtual environments for PHP, Ruby, Java, etc... why for Python? [1] https://www.jeffgeerling.com/blog/2023/how-solve-error-externally-managed-environment-when-installing-pip3 https://www.jeffgeerling.com/blog/2023/how-solve-error-exter...
- nicolaslem 3y ago
  I think that what bothers you with virtual environments is simply the name. A venv is literally just a directory with files in it. Most of the languages you mentioned actually use a very similar approach to installing dependencies, they just don't call it "virtual", they call it GOPATH, node_modules or something similar. As a developer working mostly with Python I am so glad to have way to install a dependency in a directory that won't break my OS and without having to use docker. I still use docker for lots of things, but not as a way to resolve issues that can be resolved by creating a new directory.
- delfinom 3y ago
  >I know some Python maintainers think users are too dumb to work with system Python, No, this isn't python maintainers thinking users are too dumb. This is distro maintainers fed up with false bug reports and broken systems caused by python's inability of versioning the packages fed to programs without venv.
- nsteel 3y ago
  Using --break-system-packages is a far clearer way to do the same thing.
delfinom 3y ago
This is fundamentally a python problem. python with pip has no ability to version modules correctly in the main package store and feed them to applications by version. If one system installed package needs numpy 1.26.0 but another system installed package that'll brick your system needs 1.13.5. The result is, "WTF THESE BUG REPORTS IN OUR DEBIAN ISSUE TRACKER". So virtualenv is now enforced, for everyone because that's python's only mechanism to feed versioned packages that avoid breaking packaged system software But hey, at least they added the arg `--break-system-packages` to pip. LOL You can also edit/create a pip.conf and add [global] break-system-packages = true This actually improves the user experience for people who acknowledge what they are doing.
- imp0cat 3y ago
  Or for people who just don't care, ie. if you're just trying something in a docker image.
zeagle 3y ago
The way I run my NAS I have a different user for different purposes so just install modules user level with --break-system-packages. Is there a better way?

extraduder_ire 3y ago

Can you use ensurepip to install pip, and then use pip from there? I recently had to deal with missing pip on the last ubuntu LTS version, and that's what I had to do. Makes some amount of sense to stop novices messing up their python install right off the bat, much like I have on most of my machines.

imp0cat 3y ago

I think `pip install --break-system-packages` will still work if you insist on mixing pip and deb.

josephcsible 3y ago

> you cannot install modules with `pip3 install module` (or `python3 -m pip install module`) without using a virtualenv or using the .deb packaged version in the repos. Wasn't that always a really bad idea to do, though, since it was so easy to break your system Python that way?

darkwater 3y ago
I'm not saying it was the best way but if you just ran some small 3rd party script or wrote some small Python script with non packaged deps, well, it was easy, fast and usually safe. But forcing this new behavior is surely a good decision.
deleted 3y ago
[deleted]
asddubs 3y ago
I'm fairly ignorant of the python ecosystem beyond extremely casual use for a few scripts, how does it break your system to do this?
- sjsdaiuasgdia 3y ago
  There's a few ways Python packages can be installed. They can be installed at the system level, making the package available to any Python program on the system. This can get problematic if not all of your Python usage can function with the same versions of all system-installed packages. They can be installed at the user level, sitting in your homedir instead of somewhere under /usr. This gives some isolation from the system defaults, but may just move the problem down a level - you still have the potential for version conflicts. They can also be installed at the virtualenv level, and now the package install is specific to Python programs launched from that virtualenv. Upgrading a package can only break stuff in that virtualenv, it can't spread beyond it. If you mostly use stuff from the standard library and stable packages that don't change their APIs much, you can go a long way without venvs before it actually bites you.
  - ramses0 3y ago
    The thing that _really_ irritates me about debian (almost the only thing, actually) is how flipping hard it is to use it as a development environment/system. It's _great_ as a system, but for the love of $DIETY, why isn't there some sort of way to say: `sudo apt install $ALL_THE_COMPILERS@[latest||version]` ...semi-similarly to the "exceptions" made for frequently updated packages like `firefox`, `chromium`, etc... I know of `asdf`, but it seems like a big miss that it's sometimes so hard to use debian "properly" (ie: via `apt ...`) to actually develop. Or similarly, wouldn't it be great if similar to `LSP-language-server-protocol`, there was something like `PSP-package-server-protocol` where `npm`, `go.mod`, `pip`, etc... were all like: "yeah, the debian system packages for python, perl, etc. are great for debian-on-debian", but we should also support pulling an arbitrary project at an arbitrary version and make a best-effort attempt at being able to isolate, share, manage, upgrade packages between them.
    - 3y ago
eternityforest 3y ago
Yeah it's insanely easy to break stuff.

adql 3y ago

That's a very good change. You should never have a reason to vomit your app's deps into system, far too easy to cause problems on upgrade

elsjaako 3y ago

Do you know if `pip3 install --user module` still works?

Phrodo_00 3y ago
Suse Tumbleweed made the same change and it doesn't work there (I use it through WSL if it changes anything)

geerlingguy 3y ago

You can by deleting the EXTERNALLY-MANAGED file [1] I know some Python maintainers think users are too dumb to work with system Python, but there are also some packages that are really hard to deal with on a system level if you try installing them in a venv. Plus venv's are dumb (I use Docker if I need more than one environment for specific apps or dev work), since it's another layer of abstraction I'd need to learn just to use Python. I'm not forced to use virtual environments for PHP, Ruby, Java, etc... why for Python? [1] https://www.jeffgeerling.com/blog/2023/how-solve-error-externally-managed-environment-when-installing-pip3 https://www.jeffgeerling.com/blog/2023/how-solve-error-exter...

nicolaslem 3y ago
I think that what bothers you with virtual environments is simply the name. A venv is literally just a directory with files in it. Most of the languages you mentioned actually use a very similar approach to installing dependencies, they just don't call it "virtual", they call it GOPATH, node_modules or something similar. As a developer working mostly with Python I am so glad to have way to install a dependency in a directory that won't break my OS and without having to use docker. I still use docker for lots of things, but not as a way to resolve issues that can be resolved by creating a new directory.
delfinom 3y ago
>I know some Python maintainers think users are too dumb to work with system Python, No, this isn't python maintainers thinking users are too dumb. This is distro maintainers fed up with false bug reports and broken systems caused by python's inability of versioning the packages fed to programs without venv.
nsteel 3y ago
Using --break-system-packages is a far clearer way to do the same thing.

delfinom 3y ago

This is fundamentally a python problem. python with pip has no ability to version modules correctly in the main package store and feed them to applications by version. If one system installed package needs numpy 1.26.0 but another system installed package that'll brick your system needs 1.13.5. The result is, "WTF THESE BUG REPORTS IN OUR DEBIAN ISSUE TRACKER". So virtualenv is now enforced, for everyone because that's python's only mechanism to feed versioned packages that avoid breaking packaged system software But hey, at least they added the arg `--break-system-packages` to pip. LOL You can also edit/create a pip.conf and add [global] break-system-packages = true This actually improves the user experience for people who acknowledge what they are doing.

imp0cat 3y ago
Or for people who just don't care, ie. if you're just trying something in a docker image.

zeagle 3y ago

The way I run my NAS I have a different user for different purposes so just install modules user level with --break-system-packages. Is there a better way?