4 ms·

I think it would be pretty natural for a distro to run it's own crates.io/PyPI/NPM mirror to build packages against, and have a unified set of library versions

I think it would be pretty natural for a distro to run it's own crates.io/PyPI/NPM mirror to build packages against, and have a unified set of library versions for the entire OS that way. Maybe not even a server, maybe just a giant Git repo or whatever, filtered down to the set of packages they actually use. Have any of them tried this?

steveklabnik 2y ago
I’m not aware of any that have; this would require them to become experts in running parallel infra, in many languages and with varied setups, and would mean that they’re not longer self-contained. That feels antithetical to their goals to me at least.
rtpg 2y ago
I mean Ubuntu has a bunch of python libs. The problem is that you need to make a universe of mutually compatible libs, so if _any_ of the libs have old-ish dependencies, that holds back so much. I think what happens in practice is package maintainers do the work to get things up to date, and so only include a subset.
thristian 2y ago
Debian effectively does this. It doesn't actually implement the crates.io API, but when building a Rust package they tell cargo "look in this directory for dependencies, don't talk to the network". As I understand it, the process goes something like this: - Debian wants to package a Rust tool - They examine the tool's Cargo.toml file to determine its immediate dependencies - Those Rust dependencies are converted into Debian package names - Generate a Debian package definition that lists those dependencies under "Build-Depends" - When building the Debian package in an isolated environment, Build-Depends packages are installed, providing things like `rustc` and `cargo` as well as putting the dependencies' source into the "local dependencies" directory - Cargo runs, configured to only look in the "local dependencies" directory for dependencies - the resulting binary is scooped up and packaged
- actionfromafar 2y ago
  Is there some documentation on how to do this yourself? It sounds very useful for building software meant to be binary distributed.
  - mkesper 2y ago
    The whole process seems to be compiled in this Readme: https://salsa.debian.org/rust-team/debcargo-conf/blob/master/README.rst https://salsa.debian.org/rust-team/debcargo-conf/blob/master... How to use only local packages can be found in the last paragraph.
- nesarkvechnep 2y ago
  Is this documented anywhere?
  - thristian 2y ago
    See https://wiki.debian.org/Teams/RustPackaging/Policy https://wiki.debian.org/Teams/RustPackaging/Policy
- josephg 2y ago
  Is this automated? Does doing this manually do anything for Debian other than create a headache and a mountain of work? If it were up to me, I’d consider carving off part of the Debian package namespace - like cargo—* for cargo packages, and just auto-create them (recursively) based on the package name in cargo. There are a few things to figure out - like security, and manually patched packages, and integrating with cargo itself. But it seems like it should be possible to make something like this work with minimal human intervention.
  - thristian 2y ago
    Yes, it's heavily automated.
- IshKebab 2y ago
  Why not just vendor the source for each Rust tool separately? A lot simpler and more reliable. The only downside I can see is disk space usage on the builder machines, but it's source code... is it really that big?
oefrha 2y ago
Haskell has Stackage[1] which maintains globally consistent sets of Haskell packages that build together. It’s of course a huge pain to maintain, probably exponentially harder wrt <size of package registry> / <size of Hackage>. Also highly depends on whether the language and ecosystem helps and encourages library authors to maintain compatibility; I don’t think Rust does. [1] https://www.stackage.org/ https://www.stackage.org/

steveklabnik 2y ago

I’m not aware of any that have; this would require them to become experts in running parallel infra, in many languages and with varied setups, and would mean that they’re not longer self-contained. That feels antithetical to their goals to me at least.

rtpg 2y ago

I mean Ubuntu has a bunch of python libs. The problem is that you need to make a universe of mutually compatible libs, so if _any_ of the libs have old-ish dependencies, that holds back so much. I think what happens in practice is package maintainers do the work to get things up to date, and so only include a subset.

thristian 2y ago

Debian effectively does this. It doesn't actually implement the crates.io API, but when building a Rust package they tell cargo "look in this directory for dependencies, don't talk to the network". As I understand it, the process goes something like this: - Debian wants to package a Rust tool - They examine the tool's Cargo.toml file to determine its immediate dependencies - Those Rust dependencies are converted into Debian package names - Generate a Debian package definition that lists those dependencies under "Build-Depends" - When building the Debian package in an isolated environment, Build-Depends packages are installed, providing things like `rustc` and `cargo` as well as putting the dependencies' source into the "local dependencies" directory - Cargo runs, configured to only look in the "local dependencies" directory for dependencies - the resulting binary is scooped up and packaged

actionfromafar 2y ago
Is there some documentation on how to do this yourself? It sounds very useful for building software meant to be binary distributed.
- mkesper 2y ago
  The whole process seems to be compiled in this Readme: https://salsa.debian.org/rust-team/debcargo-conf/blob/master/README.rst https://salsa.debian.org/rust-team/debcargo-conf/blob/master... How to use only local packages can be found in the last paragraph.
nesarkvechnep 2y ago
Is this documented anywhere?
- thristian 2y ago
  See https://wiki.debian.org/Teams/RustPackaging/Policy https://wiki.debian.org/Teams/RustPackaging/Policy
josephg 2y ago
Is this automated? Does doing this manually do anything for Debian other than create a headache and a mountain of work? If it were up to me, I’d consider carving off part of the Debian package namespace - like cargo—* for cargo packages, and just auto-create them (recursively) based on the package name in cargo. There are a few things to figure out - like security, and manually patched packages, and integrating with cargo itself. But it seems like it should be possible to make something like this work with minimal human intervention.
- thristian 2y ago
  Yes, it's heavily automated.
IshKebab 2y ago
Why not just vendor the source for each Rust tool separately? A lot simpler and more reliable. The only downside I can see is disk space usage on the builder machines, but it's source code... is it really that big?

oefrha 2y ago

Haskell has Stackage[1] which maintains globally consistent sets of Haskell packages that build together. It’s of course a huge pain to maintain, probably exponentially harder wrt <size of package registry> / <size of Hackage>. Also highly depends on whether the language and ecosystem helps and encourages library authors to maintain compatibility; I don’t think Rust does. [1] https://www.stackage.org/ https://www.stackage.org/