6 ms·

It’s not like this is unique to rust; you see similar issues with node and python. Distributions have many jobs, but one was solving the lack of package manage

by ahupp 2y ago

It’s not like this is unique to rust; you see similar issues with node and python. Distributions have many jobs, but one was solving the lack of package management in C. Now that every modern language a package manager, trying to apply the C package management philosophy is untenable. Specifically, the idea of a single version, globally installed, and producing distro packages for every language specific packages.

llm_trw 2y ago
Apart from the fact building it the 'non-C way' results in a magic meat package that you have no idea what it contains: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Guix is also a distro that allows for any number of versions of the same package globally, something that language specific dependancy managers do not. Distors are there for a reason, and anyone who doesn't understand that reason is just another contributor to the ongoing collapse of the tower of abstractions we've built.
- ahupp 2y ago
  Using language-native packaging doesn't imply that you have to use binaries from wherever. In the pytorch example you can still build it as a regular part of the distribution, using the C++ dependencies/toolchain, it just means you don't try to stuff it into a versioning/distribution/install model that doesn't match the languages expectations.
- tcfhgj 2y ago
  > Distors are there for a reason for me: make an os out of the kernel
- pjmlp 2y ago
  Making an userspace to Linux kernel, and sorting out disagreements by creating yet another fork.
- kpcyrd 2y ago
  This is outdated information. Debian (and other distros) already had their own SBOM format called buildinfo files that encodes this kind of information. In Debian stable ripgrep on amd64 is currently on version 13.0.0-4+b2. The relevant buildinfo file can be found here: https://buildinfos.debian.net/buildinfo-pool/r/rust-ripgrep/rust-ripgrep_13.0.0-4+b2_amd64.buildinfo https://buildinfos.debian.net/buildinfo-pool/r/rust-ripgrep/... It encodes the entire Rust dependency graph that was used for this binary, with exact versions of each crate.
XorNot 2y ago
Except from a management and maintenance perspective...this is a nightmare. When a security vulnerability drops somewhere, everywhere needs to be patched ASAP. Distros (and the people who run most scales of IT org) want to be able to deploy and verify that the fix is in place - and its a huge advantage if it's a linked library that you can just deploy an upgrade for. But if it's tons and tons of monolithic binaries, then the problem goes viral - every single one has to be recompiled, redeployed etc. And frequently at the cost of "are you only compatible with this specific revision, or was it just really easy to put that in?" It's worth noting that docker and friends also while still suffering from this problem, don't quite suffer from it in the same way - they're shipping entire dynamically linked environments, so while not as automatic, being able to simply scan for and replace the library you know is bad is a heck of a lot easier then recompiling a statically linked exe. People are okay with really specific dependencies when it's part of the business critical application they're supporting - i.e. the nodejs or python app which runs the business, that can do anything it wants we'll keep it running no matter what. Having this happen to the underlying distributions though? (of note: I've run into this issue with Go - love the static deploys, but if someone finds a vulnerability in the TLS stack of Go suddenly we're rushing out rebuilds).
- curt15 2y ago
  >When a security vulnerability drops somewhere, everywhere needs to be patched ASAP. Is that ultimately the responsibility of the application developer or the OS developer?
  - XorNot 2y ago
    It's "whoever turns up to do the work" but I would point out that distros generally have more people in the process who can pick up the work. The issue is one way or another it needs to happen ASAP: so either the distro is haranguing upstream to "approve" a change, or they're going to be going in and carrying patches to Cargo.toml anyway - so this idea of "don't you dare touch my dependencies" lasts exactly as long as until you need a fix in ASAP.
    - 2y ago
rlpb 2y ago
> Now that every modern language a package manager... ...they fail to integrate with dependencies written in any other language. It's fine if you just want to sit a monoculture language software stack on top of a multilingual base platform. You can't make a functional system with one language alone, yet those who criticise distribution packaging architecture do so while simultaneously depending on this ability that language-specific package managers do not have. There is no viable alternative today. Most critics think they understand the general problem but only have narrow practical experience, so end up believing that their solution is superior while not considering the general multilingual software supply problem. Nix isn't a solution either, because in the general case Nix isn't security-supporting arbitrary and multiple dependency versions either.

by ahupp 2y ago

llm_trw 2y ago

Apart from the fact building it the 'non-C way' results in a magic meat package that you have no idea what it contains: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Guix is also a distro that allows for any number of versions of the same package globally, something that language specific dependancy managers do not. Distors are there for a reason, and anyone who doesn't understand that reason is just another contributor to the ongoing collapse of the tower of abstractions we've built.

ahupp 2y ago
Using language-native packaging doesn't imply that you have to use binaries from wherever. In the pytorch example you can still build it as a regular part of the distribution, using the C++ dependencies/toolchain, it just means you don't try to stuff it into a versioning/distribution/install model that doesn't match the languages expectations.
tcfhgj 2y ago
> Distors are there for a reason for me: make an os out of the kernel
pjmlp 2y ago
Making an userspace to Linux kernel, and sorting out disagreements by creating yet another fork.
kpcyrd 2y ago
This is outdated information. Debian (and other distros) already had their own SBOM format called buildinfo files that encodes this kind of information. In Debian stable ripgrep on amd64 is currently on version 13.0.0-4+b2. The relevant buildinfo file can be found here: https://buildinfos.debian.net/buildinfo-pool/r/rust-ripgrep/rust-ripgrep_13.0.0-4+b2_amd64.buildinfo https://buildinfos.debian.net/buildinfo-pool/r/rust-ripgrep/... It encodes the entire Rust dependency graph that was used for this binary, with exact versions of each crate.

XorNot 2y ago

Except from a management and maintenance perspective...this is a nightmare. When a security vulnerability drops somewhere, everywhere needs to be patched ASAP. Distros (and the people who run most scales of IT org) want to be able to deploy and verify that the fix is in place - and its a huge advantage if it's a linked library that you can just deploy an upgrade for. But if it's tons and tons of monolithic binaries, then the problem goes viral - every single one has to be recompiled, redeployed etc. And frequently at the cost of "are you only compatible with this specific revision, or was it just really easy to put that in?" It's worth noting that docker and friends also while still suffering from this problem, don't quite suffer from it in the same way - they're shipping entire dynamically linked environments, so while not as automatic, being able to simply scan for and replace the library you know is bad is a heck of a lot easier then recompiling a statically linked exe. People are okay with really specific dependencies when it's part of the business critical application they're supporting - i.e. the nodejs or python app which runs the business, that can do anything it wants we'll keep it running no matter what. Having this happen to the underlying distributions though? (of note: I've run into this issue with Go - love the static deploys, but if someone finds a vulnerability in the TLS stack of Go suddenly we're rushing out rebuilds).

curt15 2y ago
>When a security vulnerability drops somewhere, everywhere needs to be patched ASAP. Is that ultimately the responsibility of the application developer or the OS developer?
- XorNot 2y ago
  It's "whoever turns up to do the work" but I would point out that distros generally have more people in the process who can pick up the work. The issue is one way or another it needs to happen ASAP: so either the distro is haranguing upstream to "approve" a change, or they're going to be going in and carrying patches to Cargo.toml anyway - so this idea of "don't you dare touch my dependencies" lasts exactly as long as until you need a fix in ASAP.
  - 2y ago

rlpb 2y ago

> Now that every modern language a package manager... ...they fail to integrate with dependencies written in any other language. It's fine if you just want to sit a monoculture language software stack on top of a multilingual base platform. You can't make a functional system with one language alone, yet those who criticise distribution packaging architecture do so while simultaneously depending on this ability that language-specific package managers do not have. There is no viable alternative today. Most critics think they understand the general problem but only have narrow practical experience, so end up believing that their solution is superior while not considering the general multilingual software supply problem. Nix isn't a solution either, because in the general case Nix isn't security-supporting arbitrary and multiple dependency versions either.