7 ms·

The author writes: > I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the r

The author writes: > I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users. Followed by a rationale. However, the rationale rings hollow to me: * Compiling a Rust package outside of its declared semver might not cause security problems in the form of exploitable memory corruption, but it's almost certainly going to cause stability problems with extremely undesirable characteristics (impossible for the upstream to triage, and definitionally unsupported). * The assumption that semantic changes to APIs are encoded within types is incorrect in the general case: Rust has its fair share of "take a string, return a string" APIs where critical semantic changes can occur without any changes to the public interface. These again are unlikely to cause memory safety issues but they can result in logical security issues, especially if the project's actual version constraints are intended to close off versions that do work but perform their operation in an undesirable way. As a contrived example of the above: `frobulator v1.0.0` might have `fn frob(&str) -> &str` which shells out to `bash` for no good reason. `frobulator v1.0.1` removes the subshell but doesn't change the API signature or public behavior; Debian ends up using v1.0.0 as the build dependency despite the upstream maintainer explicitly requesting v1.0.1 or later. This would presumably go unnoticed until someone files a RUSTSEC advisory on v1.0.0 or similar, which I think is a risky assumption given the size (and growth) of the Rust ecosystem and its tendency for large dep trees. The author is right that, in practice, this will work 99% of the time. But I think the 1% will cause a lot of unnecessary downstream heartburn, all because of a path dependency (assumptions around deps and dynamic linkage) that isn't categorically relevant to the Rust ecosystem.

jvanderbot 2y ago
I really don't understand this problem at all. I can cargo-deb and get a reasonable package with virtually no effort. What is uncompliant about that?
- woodruffw 2y ago
  My understanding (which could be wrong) is that this is an attempt to preserve Debian's "global" view of dependencies, wherein each Rust package has a set of dependencies that's consistent with every other Rust package (or, if not every, as many as possible). This is similar to the C/C++ packaging endeavor, where dependencies on libraries are handled via dynamic linkage to a single packaged version that's compatible with all dependents. If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages). I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.
  - wakawaka28 2y ago
    Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I think that's why Debian rejects language-specific package distribution. How bad would it suck if every Python app you installed needed to have its own venv for example? A distro might have hundreds of these applications. As a maintainer you need to try to support installing them all efficiently with as few conflicts as possible. A properly maintained global environment can do that. Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
    - woodruffw 2y ago
      > Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Or, they could maintain immutability but sacrifice their package-level global view (but not metadata-level view) by having Debian Rust source packages contain their fully vendored dependency set. The former would be a lot of work, especially given how manual the distribution packaging process is today. The latter seems more tractable, but requires distributions to readjust their approach to dependency tracking in ecosystems that fundamentally don't behave like C or C++ (Rust, Go, Python, etc.). > How bad would it suck if every Python app you installed needed to have its own venv for example? Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment. > A properly maintained global environment can do that. Agreed. The problem is the "properly maintained" part; I would argue that ignoring upstream semver constraints challenges the overall project :-)
      - 2y ago
- guappa 2y ago
  Bad developers (aka most of the developers) want to be able to break compatibility every 3 days or so, and pinning a precise version lets them do that. Some users commenting here are employed by python, which also has a policy of breaking compatibility all the time. It's very fun to develop in this way, but of course completely insecure. These developers might not care because they aren't in charge of security, while distributions care about security and distro maintainers understand that shipping thousands of copies of the same code means you can't fix vulnerabilities (and it's also terribly inefficient). My suspicion is that most developers think their own software is a special snowflake, while everyone else's software is still expected to behave normally because who's got the time to deal with that anyway?
  - woodruffw 2y ago
    AFAICT, nobody in this thread is employed by the PSF (I assume that's what you meant by "python"). And, to the best of my knowledge, there is no "break things just because" policy within Python.
    - guappa 2y ago
      I don't know who is signing your checks, I know what you are paid to work on :) The distinction between consultant and employee is pretty moot if you're not the tax office.
rcxdude 2y ago
It is also something that will make upstream hate them, possibly to the point of actively making their life difficult in retribution. (Problems along this line already are why the bcachefs dev told users to avoid debian and the packager eventually dropped the package: https://www.reddit.com/r/bcachefs/comments/1em2vzf/psa_avoid_debian/ https://www.reddit.com/r/bcachefs/comments/1em2vzf/psa_avoid...)
- Hemospectrum 2y ago
  > ...and the packager eventually dropped the package Related HN discussion from August/September: https://news.ycombinator.com/item?id=41407768 https://news.ycombinator.com/item?id=41407768
- woodruffw 2y ago
  I'm one of those potential upstreams. It's all OSS and I understand the pain of path dependency, so I don't think it would be fair to go out of my way to make any downstream distribution maintainer's life harder than necessary. At the same time, I suspect this kind of policy will make my life harder than necessary: users who hit bugs due to distribution packaging quirks will be redirected to me for triage, and I'll have a very hard time understanding (much less fixing) quirks that are tied to transitive subdependencies that my own builds intentionally exclude.
- kelnos 2y ago
  > It is also something that will make upstream hate them This was my thought. Xubuntu decided to ship unstable development pre-releases of the core Xfce components in their last stable OS release, and I got really annoyed getting bug reports from users who were running a dev release that was 4 releases behind current, where things were fixed. This was completely unnecessary and wasted a bunch of my time.
- Arnavion 2y ago
  Users ought to report bugs to their distro first, and the distro package maintainer should then decide whether it's an upstream bug or a distro packaging bug. That would cut out all the noise that upstream hates. The problem is users have gotten savvy about talking to upstream directly. It helps that upstream is usually on easy-to-use and search-engine-indexed platforms like GitHub these days vs mailing lists. It would still be fine if the user did the diligence of checking whether the problem exists in upstream or just the distro package, but they often don't do that either.
amluto 2y ago
The rationale doesn’t even seem like a rationale — it’s more like an argument that this choice might plausibly work. But why? Those pure source build-time dependencies don’t even seem useful to package. As a user of Debian or a downstream distro, there is approximately zero chance that I would want to use these packages for any purpose. This is a completely different situation from C/C++ and from building a distro out of C/C++ source. C does not have a useful package manager, and it’s handy if my distro has C libraries packaged. My distro might use them! (I might also use them for my own unpackaged work! But then I might get grumpy that the result of building my own code isn’t actually useful outside that version of that distro, and then I risk having to use containers (sigh), when what I actually wanted was just a build artifact that could run on other distros.)

jvanderbot 2y ago

I really don't understand this problem at all. I can cargo-deb and get a reasonable package with virtually no effort. What is uncompliant about that?

woodruffw 2y ago
My understanding (which could be wrong) is that this is an attempt to preserve Debian's "global" view of dependencies, wherein each Rust package has a set of dependencies that's consistent with every other Rust package (or, if not every, as many as possible). This is similar to the C/C++ packaging endeavor, where dependencies on libraries are handled via dynamic linkage to a single packaged version that's compatible with all dependents. If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages). I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.
- wakawaka28 2y ago
  Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I think that's why Debian rejects language-specific package distribution. How bad would it suck if every Python app you installed needed to have its own venv for example? A distro might have hundreds of these applications. As a maintainer you need to try to support installing them all efficiently with as few conflicts as possible. A properly maintained global environment can do that. Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
  - woodruffw 2y ago
    > Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Or, they could maintain immutability but sacrifice their package-level global view (but not metadata-level view) by having Debian Rust source packages contain their fully vendored dependency set. The former would be a lot of work, especially given how manual the distribution packaging process is today. The latter seems more tractable, but requires distributions to readjust their approach to dependency tracking in ecosystems that fundamentally don't behave like C or C++ (Rust, Go, Python, etc.). > How bad would it suck if every Python app you installed needed to have its own venv for example? Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment. > A properly maintained global environment can do that. Agreed. The problem is the "properly maintained" part; I would argue that ignoring upstream semver constraints challenges the overall project :-)
    - 2y ago
guappa 2y ago
Bad developers (aka most of the developers) want to be able to break compatibility every 3 days or so, and pinning a precise version lets them do that. Some users commenting here are employed by python, which also has a policy of breaking compatibility all the time. It's very fun to develop in this way, but of course completely insecure. These developers might not care because they aren't in charge of security, while distributions care about security and distro maintainers understand that shipping thousands of copies of the same code means you can't fix vulnerabilities (and it's also terribly inefficient). My suspicion is that most developers think their own software is a special snowflake, while everyone else's software is still expected to behave normally because who's got the time to deal with that anyway?
- woodruffw 2y ago
  AFAICT, nobody in this thread is employed by the PSF (I assume that's what you meant by "python"). And, to the best of my knowledge, there is no "break things just because" policy within Python.
  - guappa 2y ago
    I don't know who is signing your checks, I know what you are paid to work on :) The distinction between consultant and employee is pretty moot if you're not the tax office.

rcxdude 2y ago

It is also something that will make upstream hate them, possibly to the point of actively making their life difficult in retribution. (Problems along this line already are why the bcachefs dev told users to avoid debian and the packager eventually dropped the package: https://www.reddit.com/r/bcachefs/comments/1em2vzf/psa_avoid_debian/ https://www.reddit.com/r/bcachefs/comments/1em2vzf/psa_avoid...)

Hemospectrum 2y ago
> ...and the packager eventually dropped the package Related HN discussion from August/September: https://news.ycombinator.com/item?id=41407768 https://news.ycombinator.com/item?id=41407768
woodruffw 2y ago
I'm one of those potential upstreams. It's all OSS and I understand the pain of path dependency, so I don't think it would be fair to go out of my way to make any downstream distribution maintainer's life harder than necessary. At the same time, I suspect this kind of policy will make my life harder than necessary: users who hit bugs due to distribution packaging quirks will be redirected to me for triage, and I'll have a very hard time understanding (much less fixing) quirks that are tied to transitive subdependencies that my own builds intentionally exclude.
kelnos 2y ago
> It is also something that will make upstream hate them This was my thought. Xubuntu decided to ship unstable development pre-releases of the core Xfce components in their last stable OS release, and I got really annoyed getting bug reports from users who were running a dev release that was 4 releases behind current, where things were fixed. This was completely unnecessary and wasted a bunch of my time.
Arnavion 2y ago
Users ought to report bugs to their distro first, and the distro package maintainer should then decide whether it's an upstream bug or a distro packaging bug. That would cut out all the noise that upstream hates. The problem is users have gotten savvy about talking to upstream directly. It helps that upstream is usually on easy-to-use and search-engine-indexed platforms like GitHub these days vs mailing lists. It would still be fine if the user did the diligence of checking whether the problem exists in upstream or just the distro package, but they often don't do that either.

amluto 2y ago

The rationale doesn’t even seem like a rationale — it’s more like an argument that this choice might plausibly work. But why? Those pure source build-time dependencies don’t even seem useful to package. As a user of Debian or a downstream distro, there is approximately zero chance that I would want to use these packages for any purpose. This is a completely different situation from C/C++ and from building a distro out of C/C++ source. C does not have a useful package manager, and it’s handy if my distro has C libraries packaged. My distro might use them! (I might also use them for my own unpackaged work! But then I might get grumpy that the result of building my own code isn’t actually useful outside that version of that distro, and then I risk having to use containers (sigh), when what I actually wanted was just a build artifact that could run on other distros.)