10 ms·

Debian's approach to Rust – Dependency handling (2022)

69 points by zdw 2y ago

PeterWhittaker 2y ago
[flagged]
- Ar-Curunir 2y ago
  Is this an AI-generated comment? It adds nothing at all to the discussion
  - PeterWhittaker 2y ago
    Hardly. I just thought it was a great write up.
woodruffw 2y ago
The author writes: > I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users. Followed by a rationale. However, the rationale rings hollow to me: * Compiling a Rust package outside of its declared semver might not cause security problems in the form of exploitable memory corruption, but it's almost certainly going to cause stability problems with extremely undesirable characteristics (impossible for the upstream to triage, and definitionally unsupported). * The assumption that semantic changes to APIs are encoded within types is incorrect in the general case: Rust has its fair share of "take a string, return a string" APIs where critical semantic changes can occur without any changes to the public interface. These again are unlikely to cause memory safety issues but they can result in logical security issues, especially if the project's actual version constraints are intended to close off versions that do work but perform their operation in an undesirable way. As a contrived example of the above: `frobulator v1.0.0` might have `fn frob(&str) -> &str` which shells out to `bash` for no good reason. `frobulator v1.0.1` removes the subshell but doesn't change the API signature or public behavior; Debian ends up using v1.0.0 as the build dependency despite the upstream maintainer explicitly requesting v1.0.1 or later. This would presumably go unnoticed until someone files a RUSTSEC advisory on v1.0.0 or similar, which I think is a risky assumption given the size (and growth) of the Rust ecosystem and its tendency for large dep trees. The author is right that, in practice, this will work 99% of the time. But I think the 1% will cause a lot of unnecessary downstream heartburn, all because of a path dependency (assumptions around deps and dynamic linkage) that isn't categorically relevant to the Rust ecosystem.
- jvanderbot 2y ago
  I really don't understand this problem at all. I can cargo-deb and get a reasonable package with virtually no effort. What is uncompliant about that?
  - woodruffw 2y ago
    My understanding (which could be wrong) is that this is an attempt to preserve Debian's "global" view of dependencies, wherein each Rust package has a set of dependencies that's consistent with every other Rust package (or, if not every, as many as possible). This is similar to the C/C++ packaging endeavor, where dependencies on libraries are handled via dynamic linkage to a single packaged version that's compatible with all dependents. If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages). I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.
    - wakawaka28 2y ago
      Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I think that's why Debian rejects language-specific package distribution. How bad would it suck if every Python app you installed needed to have its own venv for example? A distro might have hundreds of these applications. As a maintainer you need to try to support installing them all efficiently with as few conflicts as possible. A properly maintained global environment can do that. Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
      - 2y ago
spenczar5 2y ago
> I am not aware of any dependency system that has an explicit machine-readable representation for the “unknown” state, so that they can say something like “A is known to depend on B; versions of B before v1 are known to break; version v2 is known to work”. Isn’t this what Go’s MVS does? Or am I misunderstanding?
- arccy 2y ago
  go's MVS only means >=, capped at major version, selected from all declared versions (excluding external latest released). >= doesn't really mean anything below it will break
wakawaka28 2y ago
This sounds like a very unsafe approach for this allegedly "safe" language. You should not categorically ignore the advice of the library and app authors in favor of some QA-driven time wasting scheme. If the authors are not releasing software stable enough to put together a distribution, then talk to them and get them to commit to stable releases and support for stable versions of libraries. I know they might not cooperate but at that point you just wash your hands of these problems, perhaps by hosting your own mirror of the language-specific repo. When users start having problems due to space-wasting unruly apps, you can explain to them that the problem lies upstream.
01HNNWZ0MV43FF 2y ago
> The resulting breakages will be discovered by automated QA It's a bold move, Cotton. Wiping `Cargo.lock` and updating within semver seems sort-of reasonable if you're talking about security updates like for a TLS lib. "Massaging" `Cargo.toml` and hoping that automated tests catch breakage seems doomed to fail. Of course this was 3 years ago and this person is no fool. I wonder what I'm missing. Did this ever go into production?
Macha 2y ago
To be blunt: If you compile my Rust library against dependencies that are not compatible as declared in Cargo.toml, the result is on you. If you want to regenerate a compatible Cargo.lock, that would be more understandable, and I don't go out of my way to specify "=x.y.z" dependencies in my Cargo.toml, so I have effectively given permission for that anyway. They give the example of Debian "bumping" dependencies, but given the relative "freshness" of your typical Rust package vs say... Debian stable, I imagine the more likely outcome would be downgrading dependencies. This reminds me of the time Debian developers "knew better" with the openssh key handling...
- MuffinFlavored 2y ago
  > If you want to regenerate a compatible Cargo.lock I only read the comments here so far and not the article but it sounds to me that people are up-in-arms about the "how do we handles" related to: * primarily C/GNU makefile ecosystem then you now want to add Rust to it and Cargo.lock on cares about Cargo.toml versions, not what other crates `build.rs` are linking against provided by the system external to Rust?
- yjftsjthsd-h 2y ago
  > If you compile my Rust library against dependencies that are not compatible as declared in Cargo.toml, the result is on you. The annoying thing is that that was always supposed to be true. Users of distro packages are already supposed to report all bugs to the package maintainers, and then they deal with it. Sometimes part of dealing with it is finding that a bug did in fact originate upstream and reporting it, but that was never supposed to be step 1.
  - XorNot 2y ago
    The thing is user's want a solution: if a user is reporting a bug then already two things are true: (1) they're experiencing it and it's blocking them from using the software and (2) it's significant enough to their process they didn't just install something else. So "user reporting a bug" is already an unusually highly motivated individual compared to the alternatives. Which leads to (3) they were trying to use the software right then, and a response from package maintainers of "this is an upstream bug" doesn't solve the problem, and in fact barely implies that a solution might be being requested from upstream. Which also leads to (4): the most common immediate resolution anyone is going to present that user is "use the latest version / master / whatever". Which means the user is already going to make the transition to "now I'm a direct upstream user" (and equally the high probability the bug might still exist in upstream anyway).
devit 2y ago
The correct solution to packaging Rust crates for a distribution seems to be to compile them as Rust dynamic libraries and use package names like "librust1.82-regex-1+aho-corasick" for the regex crate, semver major 1, compiled for the Rust 1.82 ABI, for the given set of features (which should be maximal, excluding conflicting features). For unclear reasons it seems Debian only packages the sources and not the binaries and thus doesn't need the Rust version, and also apparently only packages the latest semver version (which might make it problematic to compile old versions but is not an issue at runtime).
- progval 2y ago
  Are Rust ABIs even guaranteed to be stable for a given compiler version and library features? ie. doesn't the compiler allow itself to change the ABI based on external factors, like how other dependents use it?
  - woodruffw 2y ago
    The Rust ABI is unstable, but is consistent within a single Rust version when a crate is built with `dylib` as its type. So Debian could build shared libraries for various crates, but at the cost of a lot of shared library use in a less common/ergonomic configuration for consumers.
    - zozbot234 2y ago
      The thing is that even `dylib` is just not very useful given the extent that monomorphized generic code is used in Rust "library" crates. You end up having to generate your binary code at the application level anyway, when all your generic types and functions are finally instantiated with their proper types. Which is also why it's similarly not sensible to split out a C-compatible API/ABI from just any random Rust library crate, which would otherwise be the "proper" solution here (avoiding the need to rebuild-the-world whenever the system rustc compiler changes).
      - woodruffw 2y ago
atoav 2y ago
As a software developer that works in Rust as well I applaud any effort to unify dependecies where possible — if you took one of my projects however and you went all like yolo and changed my chosen dependencies in a major distro I would get all the heat for a choice I didn't make. This means if you change my chosen dependencies you are responsible to ensure testing exists. Unless debian packagers provide the tests to avoid logic errors
davexunit 2y ago
It's frustrating to see the "rewrite it in Rust" meme continue spread to all sorts of projects when there is no reasonable packaging story for the language and no solution in sight because the community does not see it as a problem. Cargo has introduced a huge problem for distros that didn't exist with languages like C. Compared to Rust, packaging C projects for distros is easy. Because of these problems, developers are receiving more issue reports from distro maintainers and are becoming more hostile to distro packaging, adopting a "use my binary or go away" mentality. Of course, this is a general problem that has been happening for all languages that come with a language-specific package manager; developers love them but they're typically unaware of the massive downstream problems they cause. It's getting harder to harder to exercise freedom 2 of free software and that's a shame.
- ahupp 2y ago
  It’s not like this is unique to rust; you see similar issues with node and python. Distributions have many jobs, but one was solving the lack of package management in C. Now that every modern language a package manager, trying to apply the C package management philosophy is untenable. Specifically, the idea of a single version, globally installed, and producing distro packages for every language specific packages.
  - llm_trw 2y ago
    Apart from the fact building it the 'non-C way' results in a magic meat package that you have no idea what it contains: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Guix is also a distro that allows for any number of versions of the same package globally, something that language specific dependancy managers do not. Distors are there for a reason, and anyone who doesn't understand that reason is just another contributor to the ongoing collapse of the tower of abstractions we've built.
    - ahupp 2y ago
      Using language-native packaging doesn't imply that you have to use binaries from wherever. In the pytorch example you can still build it as a regular part of the distribution, using the C++ dependencies/toolchain, it just means you don't try to stuff it into a versioning/distribution/install model that doesn't match the languages expectations.
josephcsible 2y ago
> I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users. That sounds like a recipe for a lot more security vulnerabilities like https://news.ycombinator.com/item?id=30614766 https://news.ycombinator.com/item?id=30614766
FridgeSeal 2y ago
TBH I think the distro model probably works really nicely for C-and-friends, from days of yore. When there was no package manager for C, resources like disk space were limited and the “ecosystem scale” of programming was a bit smaller. In this situation the benefits provided by distro-package-managers are large. However, I think programming and the ecosystem is different these days and I am beginning to think the “single globally shared dylibs” model isn’t a good fit for many of the more recent programming languages and ecosystems. We’ve seen it with the bcacheFS stuff and I think we’ll see it with more stuff in the future. I suspect there’s fundamental culture differences. As mentioned elsewhere in the thread, I’d be interested in a distro model that split out “OS” and “app” level management and does not concern itself with “user”/“app” level dependencies, beyond maybe tracking the installed versions of apps, and little else.
- fulafel 2y ago
  You can get the OS / app management separation with Snap/Flatpak/AppImage.
- oneshtein 2y ago
  > I’d be interested in a distro model that split out “OS” and “app” level management and does not concern itself with “user”/“app” level dependencies Switch to Android/Linux. :-/
deleted 2y ago
[deleted]
manquer 2y ago
Author says build dependencies calculation is not common in free software world . Aren’t emerge and other systems in source based distributions like gentoo doing this for 20 odd years now ? https://devmanual.gentoo.org/general-concepts/dependencies/ https://devmanual.gentoo.org/general-concepts/dependencies/ I don’t understand how the RDEPEND , BDEPEND setup doesn’t handle this, and the problem should be hardly unique for just rust ? Perhaps handling this in the package layer in a distribution not designed to handle build dependencies properly is the problem ?

69 points by zdw 2y ago

PeterWhittaker 2y ago

[flagged]

Ar-Curunir 2y ago
Is this an AI-generated comment? It adds nothing at all to the discussion
- PeterWhittaker 2y ago
  Hardly. I just thought it was a great write up.

woodruffw 2y ago

The author writes: > I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users. Followed by a rationale. However, the rationale rings hollow to me: * Compiling a Rust package outside of its declared semver might not cause security problems in the form of exploitable memory corruption, but it's almost certainly going to cause stability problems with extremely undesirable characteristics (impossible for the upstream to triage, and definitionally unsupported). * The assumption that semantic changes to APIs are encoded within types is incorrect in the general case: Rust has its fair share of "take a string, return a string" APIs where critical semantic changes can occur without any changes to the public interface. These again are unlikely to cause memory safety issues but they can result in logical security issues, especially if the project's actual version constraints are intended to close off versions that do work but perform their operation in an undesirable way. As a contrived example of the above: `frobulator v1.0.0` might have `fn frob(&str) -> &str` which shells out to `bash` for no good reason. `frobulator v1.0.1` removes the subshell but doesn't change the API signature or public behavior; Debian ends up using v1.0.0 as the build dependency despite the upstream maintainer explicitly requesting v1.0.1 or later. This would presumably go unnoticed until someone files a RUSTSEC advisory on v1.0.0 or similar, which I think is a risky assumption given the size (and growth) of the Rust ecosystem and its tendency for large dep trees. The author is right that, in practice, this will work 99% of the time. But I think the 1% will cause a lot of unnecessary downstream heartburn, all because of a path dependency (assumptions around deps and dynamic linkage) that isn't categorically relevant to the Rust ecosystem.

jvanderbot 2y ago
I really don't understand this problem at all. I can cargo-deb and get a reasonable package with virtually no effort. What is uncompliant about that?
- woodruffw 2y ago
  My understanding (which could be wrong) is that this is an attempt to preserve Debian's "global" view of dependencies, wherein each Rust package has a set of dependencies that's consistent with every other Rust package (or, if not every, as many as possible). This is similar to the C/C++ packaging endeavor, where dependencies on libraries are handled via dynamic linkage to a single packaged version that's compatible with all dependents. If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages). I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.
  - wakawaka28 2y ago
    Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I think that's why Debian rejects language-specific package distribution. How bad would it suck if every Python app you installed needed to have its own venv for example? A distro might have hundreds of these applications. As a maintainer you need to try to support installing them all efficiently with as few conflicts as possible. A properly maintained global environment can do that. Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
    - 2y ago

spenczar5 2y ago

> I am not aware of any dependency system that has an explicit machine-readable representation for the “unknown” state, so that they can say something like “A is known to depend on B; versions of B before v1 are known to break; version v2 is known to work”. Isn’t this what Go’s MVS does? Or am I misunderstanding?

arccy 2y ago
go's MVS only means >=, capped at major version, selected from all declared versions (excluding external latest released). >= doesn't really mean anything below it will break

wakawaka28 2y ago

This sounds like a very unsafe approach for this allegedly "safe" language. You should not categorically ignore the advice of the library and app authors in favor of some QA-driven time wasting scheme. If the authors are not releasing software stable enough to put together a distribution, then talk to them and get them to commit to stable releases and support for stable versions of libraries. I know they might not cooperate but at that point you just wash your hands of these problems, perhaps by hosting your own mirror of the language-specific repo. When users start having problems due to space-wasting unruly apps, you can explain to them that the problem lies upstream.

01HNNWZ0MV43FF 2y ago

> The resulting breakages will be discovered by automated QA It's a bold move, Cotton. Wiping `Cargo.lock` and updating within semver seems sort-of reasonable if you're talking about security updates like for a TLS lib. "Massaging" `Cargo.toml` and hoping that automated tests catch breakage seems doomed to fail. Of course this was 3 years ago and this person is no fool. I wonder what I'm missing. Did this ever go into production?

Macha 2y ago

To be blunt: If you compile my Rust library against dependencies that are not compatible as declared in Cargo.toml, the result is on you. If you want to regenerate a compatible Cargo.lock, that would be more understandable, and I don't go out of my way to specify "=x.y.z" dependencies in my Cargo.toml, so I have effectively given permission for that anyway. They give the example of Debian "bumping" dependencies, but given the relative "freshness" of your typical Rust package vs say... Debian stable, I imagine the more likely outcome would be downgrading dependencies. This reminds me of the time Debian developers "knew better" with the openssh key handling...

MuffinFlavored 2y ago
> If you want to regenerate a compatible Cargo.lock I only read the comments here so far and not the article but it sounds to me that people are up-in-arms about the "how do we handles" related to: * primarily C/GNU makefile ecosystem then you now want to add Rust to it and Cargo.lock on cares about Cargo.toml versions, not what other crates `build.rs` are linking against provided by the system external to Rust?
yjftsjthsd-h 2y ago
> If you compile my Rust library against dependencies that are not compatible as declared in Cargo.toml, the result is on you. The annoying thing is that that was always supposed to be true. Users of distro packages are already supposed to report all bugs to the package maintainers, and then they deal with it. Sometimes part of dealing with it is finding that a bug did in fact originate upstream and reporting it, but that was never supposed to be step 1.
- XorNot 2y ago
  The thing is user's want a solution: if a user is reporting a bug then already two things are true: (1) they're experiencing it and it's blocking them from using the software and (2) it's significant enough to their process they didn't just install something else. So "user reporting a bug" is already an unusually highly motivated individual compared to the alternatives. Which leads to (3) they were trying to use the software right then, and a response from package maintainers of "this is an upstream bug" doesn't solve the problem, and in fact barely implies that a solution might be being requested from upstream. Which also leads to (4): the most common immediate resolution anyone is going to present that user is "use the latest version / master / whatever". Which means the user is already going to make the transition to "now I'm a direct upstream user" (and equally the high probability the bug might still exist in upstream anyway).

devit 2y ago

The correct solution to packaging Rust crates for a distribution seems to be to compile them as Rust dynamic libraries and use package names like "librust1.82-regex-1+aho-corasick" for the regex crate, semver major 1, compiled for the Rust 1.82 ABI, for the given set of features (which should be maximal, excluding conflicting features). For unclear reasons it seems Debian only packages the sources and not the binaries and thus doesn't need the Rust version, and also apparently only packages the latest semver version (which might make it problematic to compile old versions but is not an issue at runtime).

progval 2y ago
Are Rust ABIs even guaranteed to be stable for a given compiler version and library features? ie. doesn't the compiler allow itself to change the ABI based on external factors, like how other dependents use it?
- woodruffw 2y ago
  The Rust ABI is unstable, but is consistent within a single Rust version when a crate is built with `dylib` as its type. So Debian could build shared libraries for various crates, but at the cost of a lot of shared library use in a less common/ergonomic configuration for consumers.
  - zozbot234 2y ago
    The thing is that even `dylib` is just not very useful given the extent that monomorphized generic code is used in Rust "library" crates. You end up having to generate your binary code at the application level anyway, when all your generic types and functions are finally instantiated with their proper types. Which is also why it's similarly not sensible to split out a C-compatible API/ABI from just any random Rust library crate, which would otherwise be the "proper" solution here (avoiding the need to rebuild-the-world whenever the system rustc compiler changes).
    - woodruffw 2y ago

atoav 2y ago

As a software developer that works in Rust as well I applaud any effort to unify dependecies where possible — if you took one of my projects however and you went all like yolo and changed my chosen dependencies in a major distro I would get all the heat for a choice I didn't make. This means if you change my chosen dependencies you are responsible to ensure testing exists. Unless debian packagers provide the tests to avoid logic errors

davexunit 2y ago

It's frustrating to see the "rewrite it in Rust" meme continue spread to all sorts of projects when there is no reasonable packaging story for the language and no solution in sight because the community does not see it as a problem. Cargo has introduced a huge problem for distros that didn't exist with languages like C. Compared to Rust, packaging C projects for distros is easy. Because of these problems, developers are receiving more issue reports from distro maintainers and are becoming more hostile to distro packaging, adopting a "use my binary or go away" mentality. Of course, this is a general problem that has been happening for all languages that come with a language-specific package manager; developers love them but they're typically unaware of the massive downstream problems they cause. It's getting harder to harder to exercise freedom 2 of free software and that's a shame.

ahupp 2y ago
It’s not like this is unique to rust; you see similar issues with node and python. Distributions have many jobs, but one was solving the lack of package management in C. Now that every modern language a package manager, trying to apply the C package management philosophy is untenable. Specifically, the idea of a single version, globally installed, and producing distro packages for every language specific packages.
- llm_trw 2y ago
  Apart from the fact building it the 'non-C way' results in a magic meat package that you have no idea what it contains: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Guix is also a distro that allows for any number of versions of the same package globally, something that language specific dependancy managers do not. Distors are there for a reason, and anyone who doesn't understand that reason is just another contributor to the ongoing collapse of the tower of abstractions we've built.
  - ahupp 2y ago
    Using language-native packaging doesn't imply that you have to use binaries from wherever. In the pytorch example you can still build it as a regular part of the distribution, using the C++ dependencies/toolchain, it just means you don't try to stuff it into a versioning/distribution/install model that doesn't match the languages expectations.

josephcsible 2y ago

> I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users. That sounds like a recipe for a lot more security vulnerabilities like https://news.ycombinator.com/item?id=30614766 https://news.ycombinator.com/item?id=30614766

FridgeSeal 2y ago

TBH I think the distro model probably works really nicely for C-and-friends, from days of yore. When there was no package manager for C, resources like disk space were limited and the “ecosystem scale” of programming was a bit smaller. In this situation the benefits provided by distro-package-managers are large. However, I think programming and the ecosystem is different these days and I am beginning to think the “single globally shared dylibs” model isn’t a good fit for many of the more recent programming languages and ecosystems. We’ve seen it with the bcacheFS stuff and I think we’ll see it with more stuff in the future. I suspect there’s fundamental culture differences. As mentioned elsewhere in the thread, I’d be interested in a distro model that split out “OS” and “app” level management and does not concern itself with “user”/“app” level dependencies, beyond maybe tracking the installed versions of apps, and little else.

fulafel 2y ago
You can get the OS / app management separation with Snap/Flatpak/AppImage.
oneshtein 2y ago
> I’d be interested in a distro model that split out “OS” and “app” level management and does not concern itself with “user”/“app” level dependencies Switch to Android/Linux. :-/

deleted 2y ago

[deleted]

manquer 2y ago

Author says build dependencies calculation is not common in free software world . Aren’t emerge and other systems in source based distributions like gentoo doing this for 20 odd years now ? https://devmanual.gentoo.org/general-concepts/dependencies/ https://devmanual.gentoo.org/general-concepts/dependencies/ I don’t understand how the RDEPEND , BDEPEND setup doesn’t handle this, and the problem should be hardly unique for just rust ? Perhaps handling this in the package layer in a distribution not designed to handle build dependencies properly is the problem ?