4 ms·

I'm extremely ill informed about these things, so is anyone able to explain why Gentoo actually needs to package Python libs? Or are they talking about Python i

I'm extremely ill informed about these things, so is anyone able to explain why Gentoo actually needs to package Python libs? Or are they talking about Python itself? Over the years I've found distribution packages like `python3-pycurl` etc. to be more of a hinderance than a help when I want to run Python on linux.

anentropic 5y ago
I think this or something similar was posted a week or two ago I also wondered the same thing, but the issue seems to be mostly for people who want to distribute tools built in Python via Linux package managers i.e. it should be possible to do that and not have to explain to users that it even uses Python, and certainly not that you have to first set up a virtualenv, then pip install dependencies etc It's a use case I've never had, and mostly just accepted that Python was ill-suited for, but I can see it would be nice if it worked and at the same time didn't compromise other uses too much
- cobby 5y ago
  > I think this or something similar was posted a week or two ago I believe you are referring to this post https://news.ycombinator.com/item?id=29238700 https://news.ycombinator.com/item?id=29238700
- sseagull 5y ago
  This is correct. If you have an application (lets say offlineimap, which is one I use), then you have two options: 1.) Install via a package manager (if they have it) 2.) Install via pip (or whatever) But #2 is a step backwards. Imagine having to learn the ecosystem and tools for every programming language just to get common apps running. And then being responsible for updating all the virtual environments, etc. Package managers are a very useful abstraction for end users who don't always care and just want shit to work. But fragmentation is increasing, and each language is going the blackjack and hookers route (Futurama reference). I don't see an easy solution, unfortunately.
  - drcongo 5y ago
    Thank you, that's an excellent explanation. Being primarily a Python dev I've been looking at it entirely backwards.
  - lucky_cloud 5y ago
    For something like a command line tool, you can package a dedicated python runtime with external dependencies preinstalled and treat it as a virtualenv. Maybe not the most elegant solution, but it works. iTerm2 does this for its Python scripting API. Another option is to just include the required libraries in your package and have the executable add the paths via sys.addsitedir. Pipenv installed via Homebrew does this.
    - CameronNemo 5y ago
      As a user, I would have to install fat binaries with questionable security. As a developer, I have to take special care to distribute the application in a way that works for all kinds of setups. I trust distros to do the integration work a lot better. For tools like youtube-dl or quodlibet, distro packages work quite well.
  - richardwhiuk 5y ago
    Bundle all of the wheels required for the package into a single package, rather than having a 1-1 mapping between OS packages and Python packages. Then run it all in a Venv.
    - tjader 5y ago
      Then you have multiple versions of every dependency for every application you use. Good luck fixing security bugs in all of those. Also, you have to do that for python apps, for node apps, for go apps, for rust apps, etc. As an user, I just want to run "apt install application" and have it work with dependencies kept up to date by the OS package manager and not have to micromanage the dependencies for all applications I use.
      - 5y ago
pseudalopex 5y ago
Python apps depend on Python libs.
ringworld 5y ago
A significant amount of systems tooling has migrated from perl to python in modern Linux, RHEL relies heavily on it when you start looking under the covers. The basic python interpreter installation ("stdlib") does not include a few key libraries like Requests (think curl) or YAML parsing; these must be installed as complementary (extra) packages on top. In the article the author makes note of the TOML parser (basically an enhanced INI file design); if a TOML parser is required to install a library (pyproject.toml instructions), and no TOML parser is in the stdlib, how do you install python3-toml (sic) to provide it? It's a circular dependency, chicken & egg problem caused by removing the legacy Setuptools abilty to install a library using only stdlib functionality. This is only one example, others exist - the OS detection library (needed to know which flavor of Linux you're on) is external and has similar (but not identical) needs, as the python installation paths are different on RHEL-like systems than Debian-like. This one has solutions for it but the author is pointing out those solutions might break based on the current trajectory of upstream thinking.
- jfrunyon 5y ago
  > like Requests (think curl) https://docs.python.org/3/library/urllib.request.html https://docs.python.org/3/library/urllib.request.html ?
  - jogu 5y ago
    No, this requests: https://github.com/psf/requests https://github.com/psf/requests
    - jfrunyon 5y ago
      ... why would I use that when I could use https://docs.python.org/3/library/urllib.request.html https://docs.python.org/3/library/urllib.request.html ?
      - jogu 5y ago
        Requests generally provides a simpler interface and absorbs some of the complexities if you were trying to use the stdlib directly e.g. pooling, redirects, retries, proxies, etc. It also smooths over some of the differences in the stdlib across Python versions, though that's less of an issue as of late as older versions are being deprecated.
deleted 5y ago
[deleted]
Taywee 5y ago
Gentoo, in particular, is very heavily dependent on Python. Its native package manager, Portage, is written entirely in Python, so the base OS install depends on functioning Python and many particular Python libs: https://en.wikipedia.org/wiki/Portage_(software) https://en.wikipedia.org/wiki/Portage_(software) Gentoo is in a special place with packaging Python as well, with Portage's slot system allowing many parallel python versions, and having each python lib installed for all installed python versions at once. Removing a python version from PYTHON_TARGETS and uninstalling that python version will cause all the installed python libs to eventually rebuild and remove their installed packages for that version. Adding a python version to PYTHON_TARGETS will do the inverse. So Gentoo needs to package python libs because the OS depends on them. A lot of other distros have dependency on Python and some Python libs as well, or have very close-to-core use depend on it. A lot of the SELinux tools are Python, last I checked. So there needs to be at least some mechanism for including python libs in-distro.
- twelvechairs 5y ago
  This is right, but also at a fundamental level Gentoo's 'killer feature' has always been a package manager (Portage) that is flexible, modular, portable, maintainable, highly-compatible, efficient. Another package management system over the top of this (pip, flit for python) that doesn't share the same values - in this case doesn't care so much about backwards-compatibility or efficient builds, really breaks Gentoo's potential to provide that experience to its users that is the headline of why people use Gentoo. And the Gentoo devs are therefore forced into filling some of that gap and there's some uneasy compromises in the process.
noobermin 5y ago
A package manager in theory should manage things installed under /usr, no?

anentropic 5y ago

I think this or something similar was posted a week or two ago I also wondered the same thing, but the issue seems to be mostly for people who want to distribute tools built in Python via Linux package managers i.e. it should be possible to do that and not have to explain to users that it even uses Python, and certainly not that you have to first set up a virtualenv, then pip install dependencies etc It's a use case I've never had, and mostly just accepted that Python was ill-suited for, but I can see it would be nice if it worked and at the same time didn't compromise other uses too much

cobby 5y ago
> I think this or something similar was posted a week or two ago I believe you are referring to this post https://news.ycombinator.com/item?id=29238700 https://news.ycombinator.com/item?id=29238700
sseagull 5y ago
This is correct. If you have an application (lets say offlineimap, which is one I use), then you have two options: 1.) Install via a package manager (if they have it) 2.) Install via pip (or whatever) But #2 is a step backwards. Imagine having to learn the ecosystem and tools for every programming language just to get common apps running. And then being responsible for updating all the virtual environments, etc. Package managers are a very useful abstraction for end users who don't always care and just want shit to work. But fragmentation is increasing, and each language is going the blackjack and hookers route (Futurama reference). I don't see an easy solution, unfortunately.
- drcongo 5y ago
  Thank you, that's an excellent explanation. Being primarily a Python dev I've been looking at it entirely backwards.
- lucky_cloud 5y ago
  For something like a command line tool, you can package a dedicated python runtime with external dependencies preinstalled and treat it as a virtualenv. Maybe not the most elegant solution, but it works. iTerm2 does this for its Python scripting API. Another option is to just include the required libraries in your package and have the executable add the paths via sys.addsitedir. Pipenv installed via Homebrew does this.
  - CameronNemo 5y ago
    As a user, I would have to install fat binaries with questionable security. As a developer, I have to take special care to distribute the application in a way that works for all kinds of setups. I trust distros to do the integration work a lot better. For tools like youtube-dl or quodlibet, distro packages work quite well.
- richardwhiuk 5y ago
  Bundle all of the wheels required for the package into a single package, rather than having a 1-1 mapping between OS packages and Python packages. Then run it all in a Venv.
  - tjader 5y ago
    Then you have multiple versions of every dependency for every application you use. Good luck fixing security bugs in all of those. Also, you have to do that for python apps, for node apps, for go apps, for rust apps, etc. As an user, I just want to run "apt install application" and have it work with dependencies kept up to date by the OS package manager and not have to micromanage the dependencies for all applications I use.
    - 5y ago

pseudalopex 5y ago

Python apps depend on Python libs.

ringworld 5y ago

A significant amount of systems tooling has migrated from perl to python in modern Linux, RHEL relies heavily on it when you start looking under the covers. The basic python interpreter installation ("stdlib") does not include a few key libraries like Requests (think curl) or YAML parsing; these must be installed as complementary (extra) packages on top. In the article the author makes note of the TOML parser (basically an enhanced INI file design); if a TOML parser is required to install a library (pyproject.toml instructions), and no TOML parser is in the stdlib, how do you install python3-toml (sic) to provide it? It's a circular dependency, chicken & egg problem caused by removing the legacy Setuptools abilty to install a library using only stdlib functionality. This is only one example, others exist - the OS detection library (needed to know which flavor of Linux you're on) is external and has similar (but not identical) needs, as the python installation paths are different on RHEL-like systems than Debian-like. This one has solutions for it but the author is pointing out those solutions might break based on the current trajectory of upstream thinking.

jfrunyon 5y ago
> like Requests (think curl) https://docs.python.org/3/library/urllib.request.html https://docs.python.org/3/library/urllib.request.html ?
- jogu 5y ago
  No, this requests: https://github.com/psf/requests https://github.com/psf/requests
  - jfrunyon 5y ago
    ... why would I use that when I could use https://docs.python.org/3/library/urllib.request.html https://docs.python.org/3/library/urllib.request.html ?
    - jogu 5y ago
      Requests generally provides a simpler interface and absorbs some of the complexities if you were trying to use the stdlib directly e.g. pooling, redirects, retries, proxies, etc. It also smooths over some of the differences in the stdlib across Python versions, though that's less of an issue as of late as older versions are being deprecated.

deleted 5y ago

[deleted]

Taywee 5y ago

Gentoo, in particular, is very heavily dependent on Python. Its native package manager, Portage, is written entirely in Python, so the base OS install depends on functioning Python and many particular Python libs: https://en.wikipedia.org/wiki/Portage_(software) https://en.wikipedia.org/wiki/Portage_(software) Gentoo is in a special place with packaging Python as well, with Portage's slot system allowing many parallel python versions, and having each python lib installed for all installed python versions at once. Removing a python version from PYTHON_TARGETS and uninstalling that python version will cause all the installed python libs to eventually rebuild and remove their installed packages for that version. Adding a python version to PYTHON_TARGETS will do the inverse. So Gentoo needs to package python libs because the OS depends on them. A lot of other distros have dependency on Python and some Python libs as well, or have very close-to-core use depend on it. A lot of the SELinux tools are Python, last I checked. So there needs to be at least some mechanism for including python libs in-distro.

twelvechairs 5y ago
This is right, but also at a fundamental level Gentoo's 'killer feature' has always been a package manager (Portage) that is flexible, modular, portable, maintainable, highly-compatible, efficient. Another package management system over the top of this (pip, flit for python) that doesn't share the same values - in this case doesn't care so much about backwards-compatibility or efficient builds, really breaks Gentoo's potential to provide that experience to its users that is the headline of why people use Gentoo. And the Gentoo devs are therefore forced into filling some of that gap and there's some uneasy compromises in the process.

noobermin 5y ago

A package manager in theory should manage things installed under /usr, no?