6 ms·

This article really comes at some of the crux of packaging pains for many languages. The "distro ships the software" idea is something so many modern language t

by rtpg 2y ago

This article really comes at some of the crux of packaging pains for many languages. The "distro ships the software" idea is something so many modern language toolkits are pretty far away from (and, as someone constantly burned by Debian packaging choices, something I'm pretty happy with as well). But without a good answer, there's going to be frustration. When distros are charging themselves with things like security, shared libraries do become a bit load bearing. And for users, the idea that you update some lib once instead of "number of software vendor" times is tempting! But as a software developer, I really really enjoy how I can ship a thing with exactly a certain set of versions, so it's almost an anti-feature to have a distro swap out functionality out from under me. Of course there can be balances and degrees to this. But a part of me feels like the general trend of software packaging is leaning towards "you bundle one thing at a time", away from "you have a system running N things", and in that model I don't know where distro packagers are (at least for server software).

oconnor663 2y ago
I think it would be pretty natural for a distro to run it's own crates.io/PyPI/NPM mirror to build packages against, and have a unified set of library versions for the entire OS that way. Maybe not even a server, maybe just a giant Git repo or whatever, filtered down to the set of packages they actually use. Have any of them tried this?
- steveklabnik 2y ago
  I’m not aware of any that have; this would require them to become experts in running parallel infra, in many languages and with varied setups, and would mean that they’re not longer self-contained. That feels antithetical to their goals to me at least.
- rtpg 2y ago
  I mean Ubuntu has a bunch of python libs. The problem is that you need to make a universe of mutually compatible libs, so if _any_ of the libs have old-ish dependencies, that holds back so much. I think what happens in practice is package maintainers do the work to get things up to date, and so only include a subset.
- thristian 2y ago
  Debian effectively does this. It doesn't actually implement the crates.io API, but when building a Rust package they tell cargo "look in this directory for dependencies, don't talk to the network". As I understand it, the process goes something like this: - Debian wants to package a Rust tool - They examine the tool's Cargo.toml file to determine its immediate dependencies - Those Rust dependencies are converted into Debian package names - Generate a Debian package definition that lists those dependencies under "Build-Depends" - When building the Debian package in an isolated environment, Build-Depends packages are installed, providing things like `rustc` and `cargo` as well as putting the dependencies' source into the "local dependencies" directory - Cargo runs, configured to only look in the "local dependencies" directory for dependencies - the resulting binary is scooped up and packaged
  - actionfromafar 2y ago
    Is there some documentation on how to do this yourself? It sounds very useful for building software meant to be binary distributed.
    - mkesper 2y ago
      The whole process seems to be compiled in this Readme: https://salsa.debian.org/rust-team/debcargo-conf/blob/master/README.rst https://salsa.debian.org/rust-team/debcargo-conf/blob/master... How to use only local packages can be found in the last paragraph.
  - nesarkvechnep 2y ago
    Is this documented anywhere?
    - thristian 2y ago
      See https://wiki.debian.org/Teams/RustPackaging/Policy https://wiki.debian.org/Teams/RustPackaging/Policy
  - josephg 2y ago
    Is this automated? Does doing this manually do anything for Debian other than create a headache and a mountain of work? If it were up to me, I’d consider carving off part of the Debian package namespace - like cargo—* for cargo packages, and just auto-create them (recursively) based on the package name in cargo. There are a few things to figure out - like security, and manually patched packages, and integrating with cargo itself. But it seems like it should be possible to make something like this work with minimal human intervention.
- oefrha 2y ago
  Haskell has Stackage[1] which maintains globally consistent sets of Haskell packages that build together. It’s of course a huge pain to maintain, probably exponentially harder wrt <size of package registry> / <size of Hackage>. Also highly depends on whether the language and ecosystem helps and encourages library authors to maintain compatibility; I don’t think Rust does. [1] https://www.stackage.org/ https://www.stackage.org/
Scramblejams 2y ago
> I really really enjoy how I can ship a thing with exactly a certain set of versions Just curious, how do you handle security monitoring and updates when doing this? I've been exposed to two major approaches: At $FAANG we're on a treadmill where company CI force builds your stuff every time anything in your requirements.txt releases a new version (semver respected, at least) because in the absence of targeted CVE monitoring (which itself won't cover you 100%), that's the only practical way they see to keep up with security updates. If a dependency change breaks your project and you leave it broken for too long, peeps above you start getting autocut tickets. This leads to a lot of ongoing maintenance work for the life of the project because eventually you get forced into newer minor and eventually major versions whether you like it or not. Outside of $FAANG I've gotten a whole lot of mileage building customer software on top of Debian- or Ubuntu-packaged dependencies. This allows me to rely on the distro for security updates without the churn, and I'm only forced to take newer dependencies when the distro version goes EOL. Obviously this constrains my library selection a lot, but if you can do it there's very little maintenance work compared to the other approach. I'd like to hear how others handle this because both approaches obviously have considerable downsides.
- rtpg 2y ago
  To be honest I'm fortunate enough to where maintenance work by sticking to "the latest and greatest" for application-level stuff is not hard. For system-level stuff in general it's been about relying on Debian or Ubuntu stuff, but then application-level management for the top layer ("the application", so to speak). I've only rarely been in situations where Debian packaging outright meant we couldn't move forward on something (one funny and annoying one was gettext being too old, but gettext is so coupled with libc for some reason that we couldn't just compile a fresh version...) The joys of not having to deal wit the extremes of any problem...
- jjnoakes 2y ago
  I strive for the latter approach as much as possible. I'm happy doing more work as a developer (like using a slightly older version of libraries or languages) to ensure that I build on a stable base that gets security updates without tons of major version api churn.
giancarlostoro 2y ago
I think Debian's system should have a system only environment with their pre-frozen packages, and then a userland specific environment, where you can install any version of anything, and none of the system packages are affected by these software packages, but when you open a terminal for dev work, you can choose a more modern Python, or what have you. I'm seeing an uptick in what they call "atomic" systems, where this is exactly the case, but last I tried to install one, it didnt even register correctly on boot up, so until I find one that boots up at all, I'll be on POP OS.
- yjftsjthsd-h 2y ago
  The way immutable distros handle it is by running those things inside containers using distrobox or the like. You can always do the same on a traditional system; just run your Python stuff in docker/podman on pop-os, or use full distrobox if you like.

by rtpg 2y ago

oconnor663 2y ago

I think it would be pretty natural for a distro to run it's own crates.io/PyPI/NPM mirror to build packages against, and have a unified set of library versions for the entire OS that way. Maybe not even a server, maybe just a giant Git repo or whatever, filtered down to the set of packages they actually use. Have any of them tried this?

steveklabnik 2y ago
I’m not aware of any that have; this would require them to become experts in running parallel infra, in many languages and with varied setups, and would mean that they’re not longer self-contained. That feels antithetical to their goals to me at least.
rtpg 2y ago
I mean Ubuntu has a bunch of python libs. The problem is that you need to make a universe of mutually compatible libs, so if _any_ of the libs have old-ish dependencies, that holds back so much. I think what happens in practice is package maintainers do the work to get things up to date, and so only include a subset.
thristian 2y ago
Debian effectively does this. It doesn't actually implement the crates.io API, but when building a Rust package they tell cargo "look in this directory for dependencies, don't talk to the network". As I understand it, the process goes something like this: - Debian wants to package a Rust tool - They examine the tool's Cargo.toml file to determine its immediate dependencies - Those Rust dependencies are converted into Debian package names - Generate a Debian package definition that lists those dependencies under "Build-Depends" - When building the Debian package in an isolated environment, Build-Depends packages are installed, providing things like `rustc` and `cargo` as well as putting the dependencies' source into the "local dependencies" directory - Cargo runs, configured to only look in the "local dependencies" directory for dependencies - the resulting binary is scooped up and packaged
- actionfromafar 2y ago
  Is there some documentation on how to do this yourself? It sounds very useful for building software meant to be binary distributed.
  - mkesper 2y ago
    The whole process seems to be compiled in this Readme: https://salsa.debian.org/rust-team/debcargo-conf/blob/master/README.rst https://salsa.debian.org/rust-team/debcargo-conf/blob/master... How to use only local packages can be found in the last paragraph.
- nesarkvechnep 2y ago
  Is this documented anywhere?
  - thristian 2y ago
    See https://wiki.debian.org/Teams/RustPackaging/Policy https://wiki.debian.org/Teams/RustPackaging/Policy
- josephg 2y ago
  Is this automated? Does doing this manually do anything for Debian other than create a headache and a mountain of work? If it were up to me, I’d consider carving off part of the Debian package namespace - like cargo—* for cargo packages, and just auto-create them (recursively) based on the package name in cargo. There are a few things to figure out - like security, and manually patched packages, and integrating with cargo itself. But it seems like it should be possible to make something like this work with minimal human intervention.
oefrha 2y ago
Haskell has Stackage[1] which maintains globally consistent sets of Haskell packages that build together. It’s of course a huge pain to maintain, probably exponentially harder wrt <size of package registry> / <size of Hackage>. Also highly depends on whether the language and ecosystem helps and encourages library authors to maintain compatibility; I don’t think Rust does. [1] https://www.stackage.org/ https://www.stackage.org/

Scramblejams 2y ago

> I really really enjoy how I can ship a thing with exactly a certain set of versions Just curious, how do you handle security monitoring and updates when doing this? I've been exposed to two major approaches: At $FAANG we're on a treadmill where company CI force builds your stuff every time anything in your requirements.txt releases a new version (semver respected, at least) because in the absence of targeted CVE monitoring (which itself won't cover you 100%), that's the only practical way they see to keep up with security updates. If a dependency change breaks your project and you leave it broken for too long, peeps above you start getting autocut tickets. This leads to a lot of ongoing maintenance work for the life of the project because eventually you get forced into newer minor and eventually major versions whether you like it or not. Outside of $FAANG I've gotten a whole lot of mileage building customer software on top of Debian- or Ubuntu-packaged dependencies. This allows me to rely on the distro for security updates without the churn, and I'm only forced to take newer dependencies when the distro version goes EOL. Obviously this constrains my library selection a lot, but if you can do it there's very little maintenance work compared to the other approach. I'd like to hear how others handle this because both approaches obviously have considerable downsides.

rtpg 2y ago
To be honest I'm fortunate enough to where maintenance work by sticking to "the latest and greatest" for application-level stuff is not hard. For system-level stuff in general it's been about relying on Debian or Ubuntu stuff, but then application-level management for the top layer ("the application", so to speak). I've only rarely been in situations where Debian packaging outright meant we couldn't move forward on something (one funny and annoying one was gettext being too old, but gettext is so coupled with libc for some reason that we couldn't just compile a fresh version...) The joys of not having to deal wit the extremes of any problem...
jjnoakes 2y ago
I strive for the latter approach as much as possible. I'm happy doing more work as a developer (like using a slightly older version of libraries or languages) to ensure that I build on a stable base that gets security updates without tons of major version api churn.

giancarlostoro 2y ago

I think Debian's system should have a system only environment with their pre-frozen packages, and then a userland specific environment, where you can install any version of anything, and none of the system packages are affected by these software packages, but when you open a terminal for dev work, you can choose a more modern Python, or what have you. I'm seeing an uptick in what they call "atomic" systems, where this is exactly the case, but last I tried to install one, it didnt even register correctly on boot up, so until I find one that boots up at all, I'll be on POP OS.

yjftsjthsd-h 2y ago
The way immutable distros handle it is by running those things inside containers using distrobox or the like. You can always do the same on a traditional system; just run your Python stuff in docker/podman on pop-os, or use full distrobox if you like.