3 ms·

Eww. I get the rationale, but Python's packaging/import logic is already ridiculously convoluted. The underlying problem is using global mutable state (e.g. /us

Eww. I get the rationale, but Python's packaging/import logic is already ridiculously convoluted. The underlying problem is using global mutable state (e.g. /usr/lib/.../site-packages). Giving each application its own package directory is better; there are Python-specific solutions to that (like virtualenv mentioned in the article), but I prefer Nix since it's language-agnostic (it can handle non-Python dependencies too). I'm also not a fan of all the Rube Goldberg machines being cobbled together to appease Docker's fundamentally broken way of working: > Distros that produce official images for single-application containers (e.g., Docker container images) should remove the EXTERNALLY‑MANAGED file, preferably in a way that makes it not come back if a user of that image installs package updates inside their image (think RUN apt‑get dist‑upgrade). On dpkg-based systems, using dpkg‑divert ‑‑local to persistently rename the file would work. On other systems, there may need to be some configuration flag available to a post-install script to re-remove the EXTERNALLY‑MANAGED file. Here a "single-application container" is assumed to contain an entire OS rather than, you know, a single application. That was supposed to make dependency management easier, since we can tailor the OS to just that one application; but based on this article it sounds like even that was a lie. Should we revisit the assumption that Docker makes dependency management easier? No, the OS maintainers are now expected to change their distros, to add another layer to the Rube Goldberg machine. I also find it pretty alarming that 'apt-get dist-upgrade' is given as an example of something we might want to do to an "official image". What's the point sanctioning a snapshot as "official" if we're just going to immediately, and non-determinisitcally, overwrite arbitrary parts of it based on external-server-state du jour?

BiteCode_dev 5y ago
While I agree about Nix being a cleaner solution, it will never be adopted, because... it's a cleaner solution. Nix is pretty much incompatible with everything, because our entire legacy of software stack has state, and we need to accommodate for it. So we keep adding those hacks because we favor compat above purity. Even nix has to, after all, there plenty of bash calls in NixOS config scripts as soon as you do something non trivial.
- chriswarbo 5y ago
  Sure, but I'm not saying everyone should adopt Nix; I'm saying the solution to problems caused by global mutable state is to avoid adding more global mutable state. If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, and (b) why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.
  - BiteCode_dev 5y ago
    > If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, debian packagers won't use pip for obvious reasons. dev need pip because it's portable, and doesn't need packagers validation. It would have been better if something like nix would have been adopted 30 years ago by both community, but it hasn't been so we have several packages managers. It's even worst now with poetry, conda, snap, flatpack and docker. > why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies. pip now uses wheels, and basically does that. Whl files are zip, and pip just unpack it at a know location. The problem is, this location is currently shared with the OS if you don't use --user or a venv. This article address that problems. My other comment also talk about some other solution. It's very hacky, because, well legacy and all that.
    - georgyo 5y ago
      You missed what he was saying. Nix solves this problem in multiple different ways. One of which is packaging, but also the entire /nix/store is mounted read-only. A sudo pip install would have no ability to break the system or remove something that was installed by nix.
      - BiteCode_dev 5y ago
        I get that, but since our entire legacy stack is stateful, there is no way around that unless you nixify the entire stuff. Immutable works only if the entire chain is.
olau 5y ago
> Giving each application its own package directory is better One simple way to do that is cd projectfoo mkdir libs pip install -t libs PYTHONPATH=libs python foo.py That way you can use system packages like a database connection library together with the local application dependency farm.

BiteCode_dev 5y ago

While I agree about Nix being a cleaner solution, it will never be adopted, because... it's a cleaner solution. Nix is pretty much incompatible with everything, because our entire legacy of software stack has state, and we need to accommodate for it. So we keep adding those hacks because we favor compat above purity. Even nix has to, after all, there plenty of bash calls in NixOS config scripts as soon as you do something non trivial.

chriswarbo 5y ago
Sure, but I'm not saying everyone should adopt Nix; I'm saying the solution to problems caused by global mutable state is to avoid adding more global mutable state. If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, and (b) why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.
- BiteCode_dev 5y ago
  > If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, debian packagers won't use pip for obvious reasons. dev need pip because it's portable, and doesn't need packagers validation. It would have been better if something like nix would have been adopted 30 years ago by both community, but it hasn't been so we have several packages managers. It's even worst now with poetry, conda, snap, flatpack and docker. > why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies. pip now uses wheels, and basically does that. Whl files are zip, and pip just unpack it at a know location. The problem is, this location is currently shared with the OS if you don't use --user or a venv. This article address that problems. My other comment also talk about some other solution. It's very hacky, because, well legacy and all that.
  - georgyo 5y ago
    You missed what he was saying. Nix solves this problem in multiple different ways. One of which is packaging, but also the entire /nix/store is mounted read-only. A sudo pip install would have no ability to break the system or remove something that was installed by nix.
    - BiteCode_dev 5y ago
      I get that, but since our entire legacy stack is stateful, there is no way around that unless you nixify the entire stuff. Immutable works only if the entire chain is.

olau 5y ago

> Giving each application its own package directory is better One simple way to do that is cd projectfoo mkdir libs pip install -t libs PYTHONPATH=libs python foo.py That way you can use system packages like a database connection library together with the local application dependency farm.