3 ms·

Indeed, and that's a good reason to avoid third-party dependencies. But that's irrelevant to the choice of programming language; a language with a bad dependenc

by kibwen 2y ago

Indeed, and that's a good reason to avoid third-party dependencies. But that's irrelevant to the choice of programming language; a language with a bad dependency manager might force you to build everything yourself, but you can always just do that, even in a language with a good dependency manager, you just choose to build everything yourself if you care. Perplexingly, the original commenter seems to understand that this doesn't matter, and then handwaves away the correct conclusion.

rectang 2y ago
It remains relevant to programming language choice because the "best in class" libraries in Rust often have lots of dependencies, thanks to Rust culture and cargo's design. I'd like to be able to pick a few libraries without incurring a huge ongoing audit burden. If I have to exclude many popular libraries because they have oodles of dependencies, that both makes searching more laborious and limits my choices.
- kibwen 2y ago
  I still don't understand what alternative people are arguing in favor of. When I think of those "best in class" libraries like regex, serde, etc, those are multiple crates that are developed by the same teams. Having one massive crate or one hundred tiny crates is irrelevant here, because if they're all developed by the same contributors it does not increase your trusted computing base.
  - estebank 2y ago
    I do think that there's some work that can be done to improve reporting on our side: cargo should be able to report not just "how many crates are in the dep tree" but rather "how many owners am I depending on" and "how many repositories am I depending on". For example, I just noticed that there's no way to see in crates.io other crates that live in the same repository, like it does per owner, even though it has that information available.
    - rectang 2y ago
      > "how many owners am I depending on" Yes, knowing that would be helpful! Is there a way to whitelist owners/publishers in Cargo?
      - estebank 2y ago
        No tool for that exists afaik, but all the pieces to make it are there.
      - 0x457 2y ago
        There is `cargo-deny` that handles some enforcement: https://github.com/EmbarkStudios/cargo-deny https://github.com/EmbarkStudios/cargo-deny. Doesn't handle authors, but I suspect it's easy to add? There is really just a handful of crates that nearly often get pulled in and probably like 5 authors across them. Supply chain harderning is pretty easy in rust: caro-deny, cargo-suply-chain, cargo-crev, cargo-vet, cargo-{s}bom and probably a few more I can't remember.
    - claytonwramsey 2y ago
      Cargo currently has `cargo tree`, which prints out a dependency tree. There's an extension to cargo which also shows how many people have the ability to push to your dependencies, titled `cargo-supply-chain`. https://github.com/rust-secure-code/cargo-supply-chain/ https://github.com/rust-secure-code/cargo-supply-chain/
    - nicoburns 2y ago
      Yeh, this would be great. I'd also love to see the ability to publish multiple library crates as a single package.
  - ncruces 2y ago
    I'd settle for most dependencies not having any dependencies at all; at a minimum making a serious effort to only add dependencies that really pull their own weight. This starts from explaining outright which dependencies they have and why. It's not so much direct dependencies that bother me: it's an exponential explosion of transitive dependencies. Also, seeing an “end product” with dozens of dependencies doesn't bother me much; a library does.
    - FridgeSeal 2y ago
      Is each web framework expected to reimplement regex, path matching and HTTP protocol logic themselves? Should every physics/ML/etc library have to write their own array abstractions and operators? If every networking library had to write and ship their own async executor logic, I can safely posit that would be about 100x worse than it is now. The alternative to “zero dependencies” isn’t full “JS/NPM-dumpster-fire”.
- ViewTrick1002 2y ago
  The best in class libraries depend on many crates. But crates are often used in workspaces to speed up compilation or split up independent parts. So how many dependencies are there truly when you peel away the first layer of the onion? https://doc.rust-lang.org/cargo/reference/workspaces.html https://doc.rust-lang.org/cargo/reference/workspaces.html
  - rectang 2y ago
    I dunno! Sounds complicated. The obvious answer is "N crates is N dependencies", because each crate represents a discrete sequence of atomic software release packages. In the absence of a standardized mechanism to group crates together, we have to fall back to informal methods, like "I know all these authors personally because I'm an insider", or "these crates seem to be related even though I'm unsure how to guarantee they'll stay that way". You can take a hard line and insist that nobody should run a single line of code they haven't reviewed, but that severely constrains the ability of a typical org to use the wider ecosystem at all. Not every org has the expertise on staff to pore over diverse Rust code and confidently state that it has no issues, and even those that do have to consider whether paying that cost is good risk management. It would be nice if there was a more reliable way to simplify the evaluation of publisher trust centers, especially for orgs who aren't going to audit code but don't want to blindly take in anything.
infogulch 2y ago
What is the shape of these dependency trees? Is it really hundreds of single-type + single-function crates? Could there ever be a path to scrub out the smaller dependencies and integrate them into larger crates with more concrete functionality? What's the status of potential distributed code review systems like cargo-crev?
- kibwen 2y ago
  > Is it really hundreds of single-type + single-function crates? No, and I think this is the crucial thing that people who have experience with NPM overlook when it comes to Rust. Rust emphatically does not have a culture of single-function microlibraries, instead libraries are split out by purpose, in the same way you would modularize a C codebase. Remember, Rust crates are not just units of distribution, they are also units of translation (a.k.a. compilation units), so the same pressures that cause people to split C projects into multiple files results in people splitting Rust projects into multiple crates.
  - deleted 2y ago
    [deleted]
  - infogulch 2y ago
    That makes sense. It seems this "problem" is unlikely to be "solved" by reintegrating projects into larger crates, for good technical and social reasons. Then the solution is to reframe the problem. Distributed code review is a brute-force style solution. Republishing collections of crates under a single name/version is a dimensionality-reduction and responsibility-concentration style solution. I suspect pure-PR style solutions will be ineffective. What other kind of solutions are there?

by kibwen 2y ago

rectang 2y ago

It remains relevant to programming language choice because the "best in class" libraries in Rust often have lots of dependencies, thanks to Rust culture and cargo's design. I'd like to be able to pick a few libraries without incurring a huge ongoing audit burden. If I have to exclude many popular libraries because they have oodles of dependencies, that both makes searching more laborious and limits my choices.

kibwen 2y ago
I still don't understand what alternative people are arguing in favor of. When I think of those "best in class" libraries like regex, serde, etc, those are multiple crates that are developed by the same teams. Having one massive crate or one hundred tiny crates is irrelevant here, because if they're all developed by the same contributors it does not increase your trusted computing base.
- estebank 2y ago
  I do think that there's some work that can be done to improve reporting on our side: cargo should be able to report not just "how many crates are in the dep tree" but rather "how many owners am I depending on" and "how many repositories am I depending on". For example, I just noticed that there's no way to see in crates.io other crates that live in the same repository, like it does per owner, even though it has that information available.
  - rectang 2y ago
    > "how many owners am I depending on" Yes, knowing that would be helpful! Is there a way to whitelist owners/publishers in Cargo?
    - estebank 2y ago
      No tool for that exists afaik, but all the pieces to make it are there.
    - 0x457 2y ago
      There is `cargo-deny` that handles some enforcement: https://github.com/EmbarkStudios/cargo-deny https://github.com/EmbarkStudios/cargo-deny. Doesn't handle authors, but I suspect it's easy to add? There is really just a handful of crates that nearly often get pulled in and probably like 5 authors across them. Supply chain harderning is pretty easy in rust: caro-deny, cargo-suply-chain, cargo-crev, cargo-vet, cargo-{s}bom and probably a few more I can't remember.
  - claytonwramsey 2y ago
    Cargo currently has `cargo tree`, which prints out a dependency tree. There's an extension to cargo which also shows how many people have the ability to push to your dependencies, titled `cargo-supply-chain`. https://github.com/rust-secure-code/cargo-supply-chain/ https://github.com/rust-secure-code/cargo-supply-chain/
  - nicoburns 2y ago
    Yeh, this would be great. I'd also love to see the ability to publish multiple library crates as a single package.
- ncruces 2y ago
  I'd settle for most dependencies not having any dependencies at all; at a minimum making a serious effort to only add dependencies that really pull their own weight. This starts from explaining outright which dependencies they have and why. It's not so much direct dependencies that bother me: it's an exponential explosion of transitive dependencies. Also, seeing an “end product” with dozens of dependencies doesn't bother me much; a library does.
  - FridgeSeal 2y ago
    Is each web framework expected to reimplement regex, path matching and HTTP protocol logic themselves? Should every physics/ML/etc library have to write their own array abstractions and operators? If every networking library had to write and ship their own async executor logic, I can safely posit that would be about 100x worse than it is now. The alternative to “zero dependencies” isn’t full “JS/NPM-dumpster-fire”.
ViewTrick1002 2y ago
The best in class libraries depend on many crates. But crates are often used in workspaces to speed up compilation or split up independent parts. So how many dependencies are there truly when you peel away the first layer of the onion? https://doc.rust-lang.org/cargo/reference/workspaces.html https://doc.rust-lang.org/cargo/reference/workspaces.html
- rectang 2y ago
  I dunno! Sounds complicated. The obvious answer is "N crates is N dependencies", because each crate represents a discrete sequence of atomic software release packages. In the absence of a standardized mechanism to group crates together, we have to fall back to informal methods, like "I know all these authors personally because I'm an insider", or "these crates seem to be related even though I'm unsure how to guarantee they'll stay that way". You can take a hard line and insist that nobody should run a single line of code they haven't reviewed, but that severely constrains the ability of a typical org to use the wider ecosystem at all. Not every org has the expertise on staff to pore over diverse Rust code and confidently state that it has no issues, and even those that do have to consider whether paying that cost is good risk management. It would be nice if there was a more reliable way to simplify the evaluation of publisher trust centers, especially for orgs who aren't going to audit code but don't want to blindly take in anything.

infogulch 2y ago

What is the shape of these dependency trees? Is it really hundreds of single-type + single-function crates? Could there ever be a path to scrub out the smaller dependencies and integrate them into larger crates with more concrete functionality? What's the status of potential distributed code review systems like cargo-crev?

kibwen 2y ago
> Is it really hundreds of single-type + single-function crates? No, and I think this is the crucial thing that people who have experience with NPM overlook when it comes to Rust. Rust emphatically does not have a culture of single-function microlibraries, instead libraries are split out by purpose, in the same way you would modularize a C codebase. Remember, Rust crates are not just units of distribution, they are also units of translation (a.k.a. compilation units), so the same pressures that cause people to split C projects into multiple files results in people splitting Rust projects into multiple crates.
- deleted 2y ago
  [deleted]
- infogulch 2y ago
  That makes sense. It seems this "problem" is unlikely to be "solved" by reintegrating projects into larger crates, for good technical and social reasons. Then the solution is to reframe the problem. Distributed code review is a brute-force style solution. Republishing collections of crates under a single name/version is a dimensionality-reduction and responsibility-concentration style solution. I suspect pure-PR style solutions will be ineffective. What other kind of solutions are there?