5 ms·

I really don't understand this problem at all. I can cargo-deb and get a reasonable package with virtually no effort. What is uncompliant about that?

woodruffw 2y ago
My understanding (which could be wrong) is that this is an attempt to preserve Debian's "global" view of dependencies, wherein each Rust package has a set of dependencies that's consistent with every other Rust package (or, if not every, as many as possible). This is similar to the C/C++ packaging endeavor, where dependencies on libraries are handled via dynamic linkage to a single packaged version that's compatible with all dependents. If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages). I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.
- wakawaka28 2y ago
  Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I think that's why Debian rejects language-specific package distribution. How bad would it suck if every Python app you installed needed to have its own venv for example? A distro might have hundreds of these applications. As a maintainer you need to try to support installing them all efficiently with as few conflicts as possible. A properly maintained global environment can do that. Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
  - woodruffw 2y ago
    > Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Or, they could maintain immutability but sacrifice their package-level global view (but not metadata-level view) by having Debian Rust source packages contain their fully vendored dependency set. The former would be a lot of work, especially given how manual the distribution packaging process is today. The latter seems more tractable, but requires distributions to readjust their approach to dependency tracking in ecosystems that fundamentally don't behave like C or C++ (Rust, Go, Python, etc.). > How bad would it suck if every Python app you installed needed to have its own venv for example? Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment. > A properly maintained global environment can do that. Agreed. The problem is the "properly maintained" part; I would argue that ignoring upstream semver constraints challenges the overall project :-)
    - wakawaka28 2y ago
      >I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Debian is a binary-first distro so this would obligate them to produce probably 5x the binary packages for the same thing. Then you have higher chances of conflicts, unless I'm missing something. C and C++ shared libraries support coexistence of multiple versions via semver-based name schemes. I don't know if Rust packages are structured that well. >Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment. These are typically not used to install everything that goes into a whole desktop or server operating system. They're used to install a handful of applications that the user wants. If you want to support as many systems as possible, you need to be mindful of resource usage. >I would argue that ignoring upstream semver constraints challenges the overall project :-) Yes it's a horrible idea. "Let's programmatically add a ton of bugs and wait for victims to report the bugs back to us in the future" is what I'm reading. A policy like that can be exploited by malicious actors. At minimum they need to ship the correct required versions of everything, if they ship anything.
- MrBuddyCasino 2y ago
  This seems like a fools errand to me. Unlike C/C++, Rust apps typically have many small-ish dependencies, trying to align them to a dist-global approved version seems pointless and laborious. Pointless because Rust programs will have a lot less CVEs that would warrant such an approach.
  - sshine 2y ago
    Laborious but not pointless. Rust programs have fewer CVEs for two reasons: its safe design, and its experienced user base. As it grows more widespread, more thoughtless programmers will create insecure programs in Rust. They just won’t often be caused by memory bugs.
  - arccy 2y ago
    I'd think logic bugs are the majority of CVEs, and rust doesn't magically make those go away
    - woodruffw 2y ago
      The "majority of CVEs" isn't a great metric, since (1) anybody can file a CVE, and (2) CNAs can be tied to vendors, who are incentivized to pre-filter CVEs or not issue CVEs at all for internal incidents. Thankfully, we have better data sources. Chromium estimates that 70% of serious security bugs in their codebases stem from memory unsafety[1], and MSRC estimates a similar number for Microsoft's codebases[2]. (General purpose programming languages can't prevent logic bugs. However, I would separately argue that idiomatic Rust programs are less likely to experience classes of logic bugs that are common in C and C++ programs, in part because strong type systems can make invalid states unrepresentable.) [1]: https://www.chromium.org/Home/chromium-security/memory-safety/ https://www.chromium.org/Home/chromium-security/memory-safet... [2]: https://msrc.microsoft.com/blog/2019/07/we-need-a-safer-systems-programming-language/ https://msrc.microsoft.com/blog/2019/07/we-need-a-safer-syst...
- Asooka 2y ago
  This is a general Linux distro problem, which is entirely self-inflicted. The distro should only carry the software for the OS itself and any applications for the user should come with all their dependencies, like on Windows. Yes, it kind of sucks that I have 3 copies of Chrome (Electron) and 7 copies of Qt on my Windows system, but that sure works a hell of a lot better than trying to synchronise the dependencies of a dozen applications. The precise split between OS service and user application can be argued endlessly, but that's for the maintainers to decide. The OS should not be a vehicle for delivering enduser applications. Yes, some applications should remain as a nice courtesy (e.g. GNU chess), but the rest should be dropped. Basically the split should be "does this project want to commit to working with major distros to keep its dependencies reasonable". I really hope we can move most Linux software to flatpak and have it updated and maintained separately from the OS. After decades of running both Linux and Windows, the Windows model of an application coming with all its dependencies in a single folder really is a lot better.
  - yjftsjthsd-h 2y ago
    > Yes, it kind of sucks that I have 3 copies of Chrome (Electron) and 7 copies of Qt on my Windows system, but that sure works a hell of a lot better than trying to synchronise the dependencies of a dozen applications. Does it? Even if 2 of those chromiums and all of the QTs have actively exploited vulnerabilities and it's anyone's guess if/when the application authors might bother updating?
    - Xylakant 2y ago
      A common downside is that the distribution picks the lowest common denominator of some dependency and all apps that require a newer version are held behind. That version may well be out of support and not receive fixes at all any more, which leaves the burden of maintenance on the distribution. Depending on the package maintainer, results may vary. (We sysadmins still remember debians effort to backport a fix to OpenSSL and breaking key generation.) This is clearly a tradeoff with no easy win for either side.
      - 2y ago
- jppittma 2y ago
  I think what I’m missing is why all of this is necessary in a world without dynamic linking. We’re talking about purely build time dependencies, who cares about those matching across static binaries? If it’s this much of a headache, I’d rather just containerize the build and call it a day.
  - woodruffw 2y ago
    This is pretty much my thought as well -- FWICT the ultimate problem here isn't a technical one, but a philosophical disagreement between how Rust tooling expects to be built and how Debian would like to build the world. Debian could adopt Rust's approach with minor technical accommodations (as evidenced by `cargo-deb`), but to do so would be to shed Debian's commitment to global version management.
  - mook 2y ago
    I believe that's usually so they can track when a library has a security vulnerability and needs to be updated, regardless of whether the upstream package itself has a version that uses the fixed library.
  - cpuguy83 2y ago
    Because they are also maintaining all those build time dependencies, including dealing with CVE's, back porting patches, etc.
  - oneshtein 2y ago
    World without dynamic linking was ended eons ago. You are just too young to remember that. But, if you like it, find like-minded people here on HN and create a successful distro with static linking only. Maybe you will success where others failed. Thank you in advance.
- iknowstuff 2y ago
  The sheer amount of useless busywork needed for this, only to end up with worse results and piss off application creators, is peak fucking Linux. If I was paid to do this kind of useless work for a living I’d probably be well on the way to off myself.
  - XorNot 2y ago
    Software bill-of-materials work has become a huge focus in the security/SRE space lately, due to the existence of actually attempted or successful supply chain attacks. In that context, creating an ever increasing menagerie of dependencies of subtly different versions of the same thing is the pointless busy work because for the tiny bit of developer convenience, you're opening an exponentially growing amount of audit work to prove the dependencies are safe.
    - flomo 2y ago
      You make a great point, but I think we're going to see stuff like Microsoft selling 'supported NPM' to corps, rather than a zillion volunteers showing up to do monkey packaging work for Debian. (In fact, inserting some rando to fork the software makes the problem worse.)
      - XorNot 2y ago
        Right but the point is then open source by necessity needs to constrain it's dependencies to sensible, auditable sets. But I'd note that a hypothetical "verified NPM" would also result in the same thing: Microsoft does not have infinite resources for such a thing, so you'd just have a limited set of approved deps yet again (which would in fact make piggy backing them relatively easy for distros). I can't see a way to slice it where it's reasonable to expect such a world to just support enormous nests of dependency versions.
        LtWorf 2y ago
        The verified npm will be like the verified pypi: "this thing was built on github, but we actually have no fucking clue if it's a bitcoin miner or a legit library"
- NobodyNada 2y ago
  Cargo even supports multiple incompatible versions of the same library coexisting within a single binary. E.g. an application I maintain depended on both winit 0.28 and 0.29 for a while, since one of my direct dependencies required winit 0.28, another required winit 0.29, and the API changed significantly between the two versions. So if the set of Rust applications packaged by Debian grows large enough, this "let's just pick one canonical version of each package" approach seems like a complete non-starter. I'd expect it to be much less effort for Debian to use the dependencies the application says it's compatible with, and build tooling allowing them to track/audit/vendor/upgrade Cargo dependencies as needed; rather than trying to shove the Cargo ecosystem into their existing tooling that is fundamentally different (and thus incompatible).
guappa 2y ago
Bad developers (aka most of the developers) want to be able to break compatibility every 3 days or so, and pinning a precise version lets them do that. Some users commenting here are employed by python, which also has a policy of breaking compatibility all the time. It's very fun to develop in this way, but of course completely insecure. These developers might not care because they aren't in charge of security, while distributions care about security and distro maintainers understand that shipping thousands of copies of the same code means you can't fix vulnerabilities (and it's also terribly inefficient). My suspicion is that most developers think their own software is a special snowflake, while everyone else's software is still expected to behave normally because who's got the time to deal with that anyway?
- woodruffw 2y ago
  AFAICT, nobody in this thread is employed by the PSF (I assume that's what you meant by "python"). And, to the best of my knowledge, there is no "break things just because" policy within Python.
  - guappa 2y ago
    I don't know who is signing your checks, I know what you are paid to work on :) The distinction between consultant and employee is pretty moot if you're not the tax office.

woodruffw 2y ago

My understanding (which could be wrong) is that this is an attempt to preserve Debian's "global" view of dependencies, wherein each Rust package has a set of dependencies that's consistent with every other Rust package (or, if not every, as many as possible). This is similar to the C/C++ packaging endeavor, where dependencies on libraries are handled via dynamic linkage to a single packaged version that's compatible with all dependents. If the above is right this contortion is similar to what's happened with Python packaging in Debian and similar, where distributions tried hard to maintain compatibility inside of a single global environment instead of allowing distinct incompatible resolutions within independent environments (which is what Python encourages). I think the "problem" with cargo-deb is that it bundles all-in-one, i.e. doesn't devolve the dependencies back to the distribution. In other words, it's technically sound but not philosophically compatible with what Debian wants to do.

wakawaka28 2y ago
Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I think that's why Debian rejects language-specific package distribution. How bad would it suck if every Python app you installed needed to have its own venv for example? A distro might have hundreds of these applications. As a maintainer you need to try to support installing them all efficiently with as few conflicts as possible. A properly maintained global environment can do that. Edit: I explained lower down but I also want to mention here, static linkage of binaries is a huge burden and waste of resources for a Linux distro. That's why they all tend to lean heavily on shared libraries unless it is too difficult to do so.
- woodruffw 2y ago
  > Packaging a distribution efficiently requires sharing as many dependencies as possible, and ideally hosting as much of the stuff as possible in an immutable state. I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Or, they could maintain immutability but sacrifice their package-level global view (but not metadata-level view) by having Debian Rust source packages contain their fully vendored dependency set. The former would be a lot of work, especially given how manual the distribution packaging process is today. The latter seems more tractable, but requires distributions to readjust their approach to dependency tracking in ecosystems that fundamentally don't behave like C or C++ (Rust, Go, Python, etc.). > How bad would it suck if every Python app you installed needed to have its own venv for example? Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment. > A properly maintained global environment can do that. Agreed. The problem is the "properly maintained" part; I would argue that ignoring upstream semver constraints challenges the overall project :-)
  - wakawaka28 2y ago
    >I don't think any of this precludes immutability: my understanding is Debian could package every version variant (or find common variants without violating semver) and maintain both immutability and their global view. Debian is a binary-first distro so this would obligate them to produce probably 5x the binary packages for the same thing. Then you have higher chances of conflicts, unless I'm missing something. C and C++ shared libraries support coexistence of multiple versions via semver-based name schemes. I don't know if Rust packages are structured that well. >Empirically, not that badly. It's what tools like `uv` and `pipx` do by default, and it results in a markedly better net user experience (since Python tools actually behave like hermetic tools, and not implicit modifiers of global resolution state). It's also what Homebrew does -- every packaged Python formula in Homebrew gets shipped in its own virtual environment. These are typically not used to install everything that goes into a whole desktop or server operating system. They're used to install a handful of applications that the user wants. If you want to support as many systems as possible, you need to be mindful of resource usage. >I would argue that ignoring upstream semver constraints challenges the overall project :-) Yes it's a horrible idea. "Let's programmatically add a ton of bugs and wait for victims to report the bugs back to us in the future" is what I'm reading. A policy like that can be exploited by malicious actors. At minimum they need to ship the correct required versions of everything, if they ship anything.
MrBuddyCasino 2y ago
This seems like a fools errand to me. Unlike C/C++, Rust apps typically have many small-ish dependencies, trying to align them to a dist-global approved version seems pointless and laborious. Pointless because Rust programs will have a lot less CVEs that would warrant such an approach.
- sshine 2y ago
  Laborious but not pointless. Rust programs have fewer CVEs for two reasons: its safe design, and its experienced user base. As it grows more widespread, more thoughtless programmers will create insecure programs in Rust. They just won’t often be caused by memory bugs.
- arccy 2y ago
  I'd think logic bugs are the majority of CVEs, and rust doesn't magically make those go away
  - woodruffw 2y ago
    The "majority of CVEs" isn't a great metric, since (1) anybody can file a CVE, and (2) CNAs can be tied to vendors, who are incentivized to pre-filter CVEs or not issue CVEs at all for internal incidents. Thankfully, we have better data sources. Chromium estimates that 70% of serious security bugs in their codebases stem from memory unsafety[1], and MSRC estimates a similar number for Microsoft's codebases[2]. (General purpose programming languages can't prevent logic bugs. However, I would separately argue that idiomatic Rust programs are less likely to experience classes of logic bugs that are common in C and C++ programs, in part because strong type systems can make invalid states unrepresentable.) [1]: https://www.chromium.org/Home/chromium-security/memory-safety/ https://www.chromium.org/Home/chromium-security/memory-safet... [2]: https://msrc.microsoft.com/blog/2019/07/we-need-a-safer-systems-programming-language/ https://msrc.microsoft.com/blog/2019/07/we-need-a-safer-syst...
Asooka 2y ago
This is a general Linux distro problem, which is entirely self-inflicted. The distro should only carry the software for the OS itself and any applications for the user should come with all their dependencies, like on Windows. Yes, it kind of sucks that I have 3 copies of Chrome (Electron) and 7 copies of Qt on my Windows system, but that sure works a hell of a lot better than trying to synchronise the dependencies of a dozen applications. The precise split between OS service and user application can be argued endlessly, but that's for the maintainers to decide. The OS should not be a vehicle for delivering enduser applications. Yes, some applications should remain as a nice courtesy (e.g. GNU chess), but the rest should be dropped. Basically the split should be "does this project want to commit to working with major distros to keep its dependencies reasonable". I really hope we can move most Linux software to flatpak and have it updated and maintained separately from the OS. After decades of running both Linux and Windows, the Windows model of an application coming with all its dependencies in a single folder really is a lot better.
- yjftsjthsd-h 2y ago
  > Yes, it kind of sucks that I have 3 copies of Chrome (Electron) and 7 copies of Qt on my Windows system, but that sure works a hell of a lot better than trying to synchronise the dependencies of a dozen applications. Does it? Even if 2 of those chromiums and all of the QTs have actively exploited vulnerabilities and it's anyone's guess if/when the application authors might bother updating?
  - Xylakant 2y ago
    A common downside is that the distribution picks the lowest common denominator of some dependency and all apps that require a newer version are held behind. That version may well be out of support and not receive fixes at all any more, which leaves the burden of maintenance on the distribution. Depending on the package maintainer, results may vary. (We sysadmins still remember debians effort to backport a fix to OpenSSL and breaking key generation.) This is clearly a tradeoff with no easy win for either side.
    - 2y ago
jppittma 2y ago
I think what I’m missing is why all of this is necessary in a world without dynamic linking. We’re talking about purely build time dependencies, who cares about those matching across static binaries? If it’s this much of a headache, I’d rather just containerize the build and call it a day.
- woodruffw 2y ago
  This is pretty much my thought as well -- FWICT the ultimate problem here isn't a technical one, but a philosophical disagreement between how Rust tooling expects to be built and how Debian would like to build the world. Debian could adopt Rust's approach with minor technical accommodations (as evidenced by `cargo-deb`), but to do so would be to shed Debian's commitment to global version management.
- mook 2y ago
  I believe that's usually so they can track when a library has a security vulnerability and needs to be updated, regardless of whether the upstream package itself has a version that uses the fixed library.
- cpuguy83 2y ago
  Because they are also maintaining all those build time dependencies, including dealing with CVE's, back porting patches, etc.
- oneshtein 2y ago
  World without dynamic linking was ended eons ago. You are just too young to remember that. But, if you like it, find like-minded people here on HN and create a successful distro with static linking only. Maybe you will success where others failed. Thank you in advance.
iknowstuff 2y ago
The sheer amount of useless busywork needed for this, only to end up with worse results and piss off application creators, is peak fucking Linux. If I was paid to do this kind of useless work for a living I’d probably be well on the way to off myself.
- XorNot 2y ago
  Software bill-of-materials work has become a huge focus in the security/SRE space lately, due to the existence of actually attempted or successful supply chain attacks. In that context, creating an ever increasing menagerie of dependencies of subtly different versions of the same thing is the pointless busy work because for the tiny bit of developer convenience, you're opening an exponentially growing amount of audit work to prove the dependencies are safe.
  - flomo 2y ago
    You make a great point, but I think we're going to see stuff like Microsoft selling 'supported NPM' to corps, rather than a zillion volunteers showing up to do monkey packaging work for Debian. (In fact, inserting some rando to fork the software makes the problem worse.)
    - XorNot 2y ago
      Right but the point is then open source by necessity needs to constrain it's dependencies to sensible, auditable sets. But I'd note that a hypothetical "verified NPM" would also result in the same thing: Microsoft does not have infinite resources for such a thing, so you'd just have a limited set of approved deps yet again (which would in fact make piggy backing them relatively easy for distros). I can't see a way to slice it where it's reasonable to expect such a world to just support enormous nests of dependency versions.
      - LtWorf 2y ago
        The verified npm will be like the verified pypi: "this thing was built on github, but we actually have no fucking clue if it's a bitcoin miner or a legit library"
NobodyNada 2y ago
Cargo even supports multiple incompatible versions of the same library coexisting within a single binary. E.g. an application I maintain depended on both winit 0.28 and 0.29 for a while, since one of my direct dependencies required winit 0.28, another required winit 0.29, and the API changed significantly between the two versions. So if the set of Rust applications packaged by Debian grows large enough, this "let's just pick one canonical version of each package" approach seems like a complete non-starter. I'd expect it to be much less effort for Debian to use the dependencies the application says it's compatible with, and build tooling allowing them to track/audit/vendor/upgrade Cargo dependencies as needed; rather than trying to shove the Cargo ecosystem into their existing tooling that is fundamentally different (and thus incompatible).

guappa 2y ago

Bad developers (aka most of the developers) want to be able to break compatibility every 3 days or so, and pinning a precise version lets them do that. Some users commenting here are employed by python, which also has a policy of breaking compatibility all the time. It's very fun to develop in this way, but of course completely insecure. These developers might not care because they aren't in charge of security, while distributions care about security and distro maintainers understand that shipping thousands of copies of the same code means you can't fix vulnerabilities (and it's also terribly inefficient). My suspicion is that most developers think their own software is a special snowflake, while everyone else's software is still expected to behave normally because who's got the time to deal with that anyway?

woodruffw 2y ago
AFAICT, nobody in this thread is employed by the PSF (I assume that's what you meant by "python"). And, to the best of my knowledge, there is no "break things just because" policy within Python.
- guappa 2y ago
  I don't know who is signing your checks, I know what you are paid to work on :) The distinction between consultant and employee is pretty moot if you're not the tax office.