3 ms·

Just use the system packages! Fedora, Debian, AUR, brew/macports on macOS, etc are all a thing, use them.

by rwmj 2y ago

atemerev 2y ago
Nope. System packages and language dependencies are better not to be mixed, and the horrors if you do that are very real.
zahlman 2y ago
There are many problems with this idea. * Windows exists. * The system package manager is meant for the system environment. The typical Python environment is separate from that - a container or a venv. The system Python installation is there to make the OS work, not to provide a working dev environment, and it's often not suitable as such (especially on Debian). * There simply are far too many packages out there to make it workable. * Part of the point of how Pip works is that compiled code can be pre-built for specific platforms. A system package manager typically loses that advantage and needs to rebuild from source. * Installing Python packages "from source" commonly involves running arbitrary Python code at install time, by design. (The build system simply doesn't provide any standardized way to invoke compilers, nor to apply conditional logic - you're stuck with Turing-complete, unsandboxed Python. Python is really hard to sandbox anyway.) For obvious reasons it's not appropriate to do that as root, which rather interferes with using a system package manager. Yes, obviously packages that mirror PyPI distributions and are made to be installed with the system package manager, will be adapted to avoid security issues. But this requires significant, parallel maintenance effort. The devs of the individual projects likely don't have resources to spare to help distro maintainers, either.
- cozzyd 2y ago
  Yeah but do you really need a separate copy of numpy/matplotlib/scipy/etc. for every single project? I don't have infinite disk space...
  - zahlman 2y ago
    If your projects are using different versions of the library and/or different Python versions, then yes. Especially for libraries like NumPy/SciPy that rely on compiled code. It must be re-compiled against the API of a specific Python minor version, for many reasons (e.g., the bytecode format changes with almost every minor version). If your projects can get away with having the same exact set of libraries installed, then no - just reuse the same virtual environment. In between is, I would argue, mostly Pip's fault. There's no facility for installing via symlinks, or hard links, or `.pth` files, even though all of those would work fine in Python. Granted, using `.pth` files causes `sys.path` to blow up at startup, which some people claim is a performance issue (it's searched linearly) - but it's honestly probably not a meaningful problem in remotely normal cases (especially considering that module imports are cached in `sys.modules`). Pip also unpacks wheels, even though Python could theoretically import from them directly (they're renamed zip files). This is partly for performance reasons but largely because it allows installed packages to read their own data files using relative paths and the ordinary file I/O API. For packages that don't need to do that, the egg format implemented a `zip_safe` flag, but for whatever reason, wheels don't have this.
ForHackernews 2y ago
Doesn't brew compile from source every time on the user's machine?
- duskwuff 2y ago
  It mostly uses precompiled packages nowadays. (Which are hosted for it by GitHub.)
- notpushkin 2y ago
  They do have binary packages now I think.
candiddevmike 2y ago
I can't tell if this is sarcasm.
- HelloNurse 2y ago
  On MacOS they are casks of sarcasm.
BiteCode_dev 2y ago
There are currently 570K+ projects in pypi, 60k+ in debian repos. It can take several months of work to approve one single package to the official repos, for a single distribution. And each have different rules and setup. Now explain to me how you think this is going to work. Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix? Do you want to do that also for JS, Ruby and PHP?
- pxc 2y ago
  You only need the system package manager to provide the non-Python dependencies of Python packages. I'd expect that those by and large are already packaged in Debian and elsewhere, perhaps with some exceptions among smalltime C (or Rust) projects that only exist to accelerate Python packages. > Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix? Nix is a good fit for this, but so is Guix and probably Spack. With a sane implementation of PEP-725 (which I mention and describe in another comment on this post), users could freely choose whatever package manager they like to supply non-Python deps.
- rwmj 2y ago
  > There are currently 570K+ projects in pypi, 60k+ in debian repos. Not all PyPi projects require C code. > It can take several months of work to approve one single package to the official repos This is a massive exaggeration. For Fedora it takes a couple of days, all of it being necessary review of the code and licenses. And yes, you do have to do that work, it's done by the distros themselves too.
kbolino 2y ago
AUR is not a source of system packages. It is a community-maintained source of package build scripts which each user must download and build themselves.
echelon 2y ago
Python builds weren't really hermetic before, but now your entire operating system is a part of the software definition. This is a slide backwards.
- aragilar 2y ago
  The OS always was part of the software definition, and always will be (unless you run without an OS/write your own).
  - echelon 2y ago
    How much of the host system, though? You want it to be as thin as possible. And ideally it wouldn't be a part of the definition at all.
    - aragilar 2y ago
      Why though? Rather than using your own bespoke graphics system (requiring the user to separately configure it) and your own custom DNS server (as opposed to the system one), you could just use the OS. That is all part of the definition of the software, no software is an island, and those that try to be tend to be frustrating to deal with.
- cozzyd 2y ago
  Plenty of python packages depend on various system packages outside of PyPI anyway.
  - kristoff_it 2y ago
    Yeah, and that's another source of jankyness that could be avoided
    - cozzyd 2y ago
      Not easily on all platforms...
- deleted 2y ago
cozzyd 2y ago
I wish there was an easy way for pip to list what it's missing so it's easier to install dependencies via the system package manager. Also it's sad how setup tools bdist_rpm seems to be considered deprecated.
- zahlman 2y ago
  >I wish there was an easy way for pip to list what it's missing so it's easier to install dependencies via the system package manager. PEP 725 is trying to be exactly that (https://peps.python.org/pep-0725/ https://peps.python.org/pep-0725/). It just took this long, since the implementation of the new `pyproject.toml` system, to get around to it. But aside from that, the system package manager is only appropriate for "dependencies" that need it - i.e. things that need to actually be part of the local system, not packages side-by-side in a virtual environment somewhere. >Also it's sad how setup tools bdist_rpm seems to be considered deprecated. Historically, Setuptools tried to be a complete solution for everything except actually fetching code. People were even expected to setup.py install locally after downloading something from GitHub or whatever. Nowadays it's supposed to be just a build backend, for the kind of building processes that can be automated under Pip's direction. `bdist_rpm` simply doesn't fit that mold. You're supposed to use a third-party build system instead.
tempfile 2y ago
This is how it should be, but it isn't how it is. The system packages are generally very outdated, and don't support multiple versions on a single system (which makes dependency locking a fantasy). I dream of the day where we only have one package manager on a single machine.

Just use the system packages! Fedora, Debian, AUR, brew/macports on macOS, etc are all a thing, use them.

by rwmj 2y ago

Just use the system packages! Fedora, Debian, AUR, brew/macports on macOS, etc are all a thing, use them.

Nope. System packages and language dependencies are better not to be mixed, and the horrors if you do that are very real.

zahlman 2y ago

There are many problems with this idea. * Windows exists. * The system package manager is meant for the system environment. The typical Python environment is separate from that - a container or a venv. The system Python installation is there to make the OS work, not to provide a working dev environment, and it's often not suitable as such (especially on Debian). * There simply are far too many packages out there to make it workable. * Part of the point of how Pip works is that compiled code can be pre-built for specific platforms. A system package manager typically loses that advantage and needs to rebuild from source. * Installing Python packages "from source" commonly involves running arbitrary Python code at install time, by design. (The build system simply doesn't provide any standardized way to invoke compilers, nor to apply conditional logic - you're stuck with Turing-complete, unsandboxed Python. Python is really hard to sandbox anyway.) For obvious reasons it's not appropriate to do that as root, which rather interferes with using a system package manager. Yes, obviously packages that mirror PyPI distributions and are made to be installed with the system package manager, will be adapted to avoid security issues. But this requires significant, parallel maintenance effort. The devs of the individual projects likely don't have resources to spare to help distro maintainers, either.

cozzyd 2y ago
Yeah but do you really need a separate copy of numpy/matplotlib/scipy/etc. for every single project? I don't have infinite disk space...
- zahlman 2y ago
  If your projects are using different versions of the library and/or different Python versions, then yes. Especially for libraries like NumPy/SciPy that rely on compiled code. It must be re-compiled against the API of a specific Python minor version, for many reasons (e.g., the bytecode format changes with almost every minor version). If your projects can get away with having the same exact set of libraries installed, then no - just reuse the same virtual environment. In between is, I would argue, mostly Pip's fault. There's no facility for installing via symlinks, or hard links, or `.pth` files, even though all of those would work fine in Python. Granted, using `.pth` files causes `sys.path` to blow up at startup, which some people claim is a performance issue (it's searched linearly) - but it's honestly probably not a meaningful problem in remotely normal cases (especially considering that module imports are cached in `sys.modules`). Pip also unpacks wheels, even though Python could theoretically import from them directly (they're renamed zip files). This is partly for performance reasons but largely because it allows installed packages to read their own data files using relative paths and the ordinary file I/O API. For packages that don't need to do that, the egg format implemented a `zip_safe` flag, but for whatever reason, wheels don't have this.

ForHackernews 2y ago

Doesn't brew compile from source every time on the user's machine?

duskwuff 2y ago
It mostly uses precompiled packages nowadays. (Which are hosted for it by GitHub.)
notpushkin 2y ago
They do have binary packages now I think.

candiddevmike 2y ago

I can't tell if this is sarcasm.

HelloNurse 2y ago
On MacOS they are casks of sarcasm.

BiteCode_dev 2y ago

There are currently 570K+ projects in pypi, 60k+ in debian repos. It can take several months of work to approve one single package to the official repos, for a single distribution. And each have different rules and setup. Now explain to me how you think this is going to work. Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix? Do you want to do that also for JS, Ruby and PHP?

pxc 2y ago
You only need the system package manager to provide the non-Python dependencies of Python packages. I'd expect that those by and large are already packaged in Debian and elsewhere, perhaps with some exceptions among smalltime C (or Rust) projects that only exist to accelerate Python packages. > Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix? Nix is a good fit for this, but so is Guix and probably Spack. With a sane implementation of PEP-725 (which I mention and describe in another comment on this post), users could freely choose whatever package manager they like to supply non-Python deps.
rwmj 2y ago
> There are currently 570K+ projects in pypi, 60k+ in debian repos. Not all PyPi projects require C code. > It can take several months of work to approve one single package to the official repos This is a massive exaggeration. For Fedora it takes a couple of days, all of it being necessary review of the code and licenses. And yes, you do have to do that work, it's done by the distros themselves too.

kbolino 2y ago

AUR is not a source of system packages. It is a community-maintained source of package build scripts which each user must download and build themselves.

echelon 2y ago

Python builds weren't really hermetic before, but now your entire operating system is a part of the software definition. This is a slide backwards.

aragilar 2y ago
The OS always was part of the software definition, and always will be (unless you run without an OS/write your own).
- echelon 2y ago
  How much of the host system, though? You want it to be as thin as possible. And ideally it wouldn't be a part of the definition at all.
  - aragilar 2y ago
    Why though? Rather than using your own bespoke graphics system (requiring the user to separately configure it) and your own custom DNS server (as opposed to the system one), you could just use the OS. That is all part of the definition of the software, no software is an island, and those that try to be tend to be frustrating to deal with.
cozzyd 2y ago
Plenty of python packages depend on various system packages outside of PyPI anyway.
- kristoff_it 2y ago
  Yeah, and that's another source of jankyness that could be avoided
  - cozzyd 2y ago
    Not easily on all platforms...
deleted 2y ago

cozzyd 2y ago

I wish there was an easy way for pip to list what it's missing so it's easier to install dependencies via the system package manager. Also it's sad how setup tools bdist_rpm seems to be considered deprecated.

zahlman 2y ago
>I wish there was an easy way for pip to list what it's missing so it's easier to install dependencies via the system package manager. PEP 725 is trying to be exactly that (https://peps.python.org/pep-0725/ https://peps.python.org/pep-0725/). It just took this long, since the implementation of the new `pyproject.toml` system, to get around to it. But aside from that, the system package manager is only appropriate for "dependencies" that need it - i.e. things that need to actually be part of the local system, not packages side-by-side in a virtual environment somewhere. >Also it's sad how setup tools bdist_rpm seems to be considered deprecated. Historically, Setuptools tried to be a complete solution for everything except actually fetching code. People were even expected to setup.py install locally after downloading something from GitHub or whatever. Nowadays it's supposed to be just a build backend, for the kind of building processes that can be automated under Pip's direction. `bdist_rpm` simply doesn't fit that mold. You're supposed to use a third-party build system instead.

tempfile 2y ago

This is how it should be, but it isn't how it is. The system packages are generally very outdated, and don't support multiple versions on a single system (which makes dependency locking a fantasy). I dream of the day where we only have one package manager on a single machine.