5 ms·

Is it normal for all rust projects to have sooo many dependencies? I tried scrolling through them to see what was used for the TUI itself, and was amazed by a n

Is it normal for all rust projects to have sooo many dependencies? I tried scrolling through them to see what was used for the TUI itself, and was amazed by a neverending list of dependencies. Reminds me of the JS dependency hell that exists.. I somehow feel this is DRY gone in the wrong direction.

Groxx 2y ago
Relatively. tbh I think it's a natural outcome of any system where it's easy and reliable to add a dependency...... .. and where there isn't any ongoing feedback about the costs associated with doing so (binary side, build time, etc) so you don't notice things slowly getting worse. It's just "rust builds are slow" not "why does this one thing I don't use take two minutes". I'm not convinced the large number is actually a problem, but the lack of feedback definitely is.
- creshal 2y ago
  > I'm not convinced the large number is actually a problem Supply chain security always matters, and the more random dependencies you introduce, the worse it gets.
  - _flux 2y ago
    But are the dependencies random?
    - creshal 2y ago
      If you have the time, I invite you to take the project's cargo lock file and analyse for every single one of them - what does it to justify its inclusion - who wrote it - who else has commit access to the repositories - how do these people secure their git access - are these people susceptible to state actor pressure - will these people never in the future add new maintainers who are moles/blackmailable/have sloppy security hygiene
      - _flux 2y ago
        I mean just this solely person already could tick all those boxes or I don't have a way to access that information. By glancing the direct dependencies of this project it is pretty easy to see why they have been selected. Yes, it would be more prudent to check all transitive dependencies as well, in addition to reviewing their source code, but frankly that's just too much effort. Does this mean the dependencies are random? But it does seem like it would be cool to have a tool to collect as much of that information as possible.
        Groxx 2y ago
        There also can and should be mechanical checks on libraries, to reduce the number of things that need heavy ongoing human review. E.g. if a library provides a pure function, does it matter if the owner changes? It's still pure. If it becomes not-pure, yes that's immediately concerning, but in the meantime its ability to do anything nefarious is infinitely smaller than most current module systems allow, and your review-budgets should probably be spent elsewhere.
        _flux 2y ago
        Some kind of capability system would be interesting, but to work within function level it would need some pretty sophisticated support from the type system. On package level it might be easier? But then again you need to have quite fine-grained capabilities to describe what a package should be able to do. Of course, in Rust all unsafe code would need to be out (and it would be its own capability), but it still leaves exploiting the compiler bugs. For malicious changes that would of course be the vector to exploit and it might be very difficult to automatically detect them.
pieresqi 2y ago
[dead]
_flux 2y ago
Are you looking at the direct dependencies? Because that list isn't neverending: https://github.com/Troels51/dtui/blob/main/Cargo.toml https://github.com/Troels51/dtui/blob/main/Cargo.toml For TUI it uses ratatui. The transitive dependencies can get quite large, though! I've never found the direct dependencies of any package to have superfluous dependencies (perhaps I haven't looked carefully enough?); they are always seem to be useful or easily provide functionality users expect to see, such as nice command line options and nice usage for them.
- integricho 2y ago
  Yes, but one cannot count transitive dependencies as if they are not your dependencies. They are still all needed to build your project, thus they are effectively yours as well. I find the pure number of dependencies involved worrying tbh (and this is not strictly related to your project, but if this is the case throughout the whole rust ecosystem, then it is a global problem).
  - codr7 2y ago
    I second that reaction to many Rust projects I've compiled. Anything JS these days tends to look similar. I think part of the issue is that it's just so easy to get a package included in "the" central registry, and for many programming adepts that's like a badge of honor. So you get many tiny libraries, and many turn into balls of mud. Then they learn about reuse and happily pull in all dependencies they can think of. In Rust there seems to be a tendency to keep stuff out of the standard library, so everyone depends on their favorite solutions to everyday problems, which depends on their favorites and so on. But it is a problem, because somewhere along the line one of those indirect dependencies is going to start conflicting with something else in the graph; the bigger the graph, the more conflicts. And finally risk, because if a major indirect dependency goes belly up, that may well cause serious problems for your direct dependencies, which then makes it your problem. Also directly related to graph size.
    - dagurp 2y ago
      I feel like the trend is going the other way in modern JS
diath 2y ago
...and the build folder adds up to 700MB, that's quite grim for a program like this.
jeroenhd 2y ago
Depends on what you consider "many dependencies". Does this tool _need_ async tasks, a code generator for argument parsing, variable log level tracing hooks and JSON logging output? Probably not. A lot of this stuff is just developer convenience. I've seen other projects just include entire third party software repositories into their source tree rather than use a dependency manager, which works, I guess, but doesn't make the dependency tree all that obvious. When I use languages that don't have an easy dependency manager, I often still try to use common libraries, but I find that there are fewer easy dependencies. In C, I would just not bother with async I/O rather than import a library that makes it easy to do. I would use a shitty "works-for-me" string replace rather than a compliant parser for formats like JSON or XML. I'd steal some random SHA code rather than link to a library. I don't think this is that much of a problem as long as you're somewhat conservative in the kinds of dependencies you add. There's a trade-off between doing something yourself (and thus correctness) or stealing/vendoring the source code, and making use of a thriving library ecosystem.
resonious 2y ago
Yup, Rust as an ecosystem feels very similar to JS. Part of this might be because some core Rust facilities are lacking. For example, many projects use the "anyhow" library for error handling. It could also be because having a good package manager makes it so easy to add dependencies, and that reduced friction just naturally makes people tend to add many dependencies.
- giyanani 2y ago
  “Lacking” implies it’s an unintentional deficiency. Rust has deliberately taken a “batteries not included” approach due to learned experiences in python, where many std-lib packages are basically abandoned/not recommended anymore. This gives the ecosystem time to develop better approaches to things, rather than ossifying around the initial implementation. Yes, it does lead to a larger set of dependencies, but some of the most common packages (including anyhow) were written and are maintained by members of the rust lang organization/team itself.
WesolyKubeczek 2y ago
When Rust's ecosystem was booming, the JS/npm way of a myriad small libraries was viewed as superior. I think that there have been dozens of articles pre-left-pad about why it is good and why it's the future. I don't quite blame them, remembering how Python's ecosystem tried to mimic Java back in the day, how PHP tried, consciously or not, to do the same. I'm just wondering what the 2024-2025 dominant fad is going to look like.
- steveklabnik 2y ago
  Crates.io never had the behavior that led to left-pad, so it didn’t really change any opinions here. Small dependencies are not inherently good. Large dependencies are not inherently bad.
  - WesolyKubeczek 2y ago
    Oh just you wait. It doesn't need to be a ragequitting and taking the toys away. If you are accustomed to having long chains of small twisty dependencies, all it takes is one Jia Tan of the Rust world to subtly alter a dependency of a dependency of a high profile dependency to own almost anything written in Rust. The said dependency can even ship precompiled bits of who-knows-which provenance, citing the same excuse as the one the author of serde used.
    - steveklabnik 2y ago
      That is a separate, and valid issue.
      - WesolyKubeczek 2y ago
        All I'm saying that in a maze of twisty little dependencies, all alike, it's very tedious to vet them all, and it's much too easy to end up with thousands of crates for the simplest things. And it's very easy to put nefarious things in once there is a will to do so. It all works well if everyone wants to play nice, but this doesn't scale. I don't know if Rust has the same quirk as JS/npm where a project can transitively depend on several versions of a module and they can coexist. Imagine a crate fixing some nasty shit, but some transitive dependency still pinning a vulnerable version. I contrast this with Go that has a rich enough standard library so you rarely need a dependency, and you don't need any for mundane stuff, and it makes you think hard about every dependency you introduce. And they got burned early enough so vendoring is a Good Thing now. And then I compare it with Perl, in that on one hand, it has lots of core modules, but then its standard library is lacking in weird places making you reach for CPAN in 100% of the cases writing a non-trivial program (HTTP/TLS comes to mind), and it would be similar to Rust except that CPAN is very slow-paced these days. I'm curious to see how it all evolves.

Groxx 2y ago

Relatively. tbh I think it's a natural outcome of any system where it's easy and reliable to add a dependency...... .. and where there isn't any ongoing feedback about the costs associated with doing so (binary side, build time, etc) so you don't notice things slowly getting worse. It's just "rust builds are slow" not "why does this one thing I don't use take two minutes". I'm not convinced the large number is actually a problem, but the lack of feedback definitely is.

creshal 2y ago
> I'm not convinced the large number is actually a problem Supply chain security always matters, and the more random dependencies you introduce, the worse it gets.
- _flux 2y ago
  But are the dependencies random?
  - creshal 2y ago
    If you have the time, I invite you to take the project's cargo lock file and analyse for every single one of them - what does it to justify its inclusion - who wrote it - who else has commit access to the repositories - how do these people secure their git access - are these people susceptible to state actor pressure - will these people never in the future add new maintainers who are moles/blackmailable/have sloppy security hygiene
    - _flux 2y ago
      I mean just this solely person already could tick all those boxes or I don't have a way to access that information. By glancing the direct dependencies of this project it is pretty easy to see why they have been selected. Yes, it would be more prudent to check all transitive dependencies as well, in addition to reviewing their source code, but frankly that's just too much effort. Does this mean the dependencies are random? But it does seem like it would be cool to have a tool to collect as much of that information as possible.
      - Groxx 2y ago
        There also can and should be mechanical checks on libraries, to reduce the number of things that need heavy ongoing human review. E.g. if a library provides a pure function, does it matter if the owner changes? It's still pure. If it becomes not-pure, yes that's immediately concerning, but in the meantime its ability to do anything nefarious is infinitely smaller than most current module systems allow, and your review-budgets should probably be spent elsewhere.
        _flux 2y ago
        Some kind of capability system would be interesting, but to work within function level it would need some pretty sophisticated support from the type system. On package level it might be easier? But then again you need to have quite fine-grained capabilities to describe what a package should be able to do. Of course, in Rust all unsafe code would need to be out (and it would be its own capability), but it still leaves exploiting the compiler bugs. For malicious changes that would of course be the vector to exploit and it might be very difficult to automatically detect them.

pieresqi 2y ago

[dead]

_flux 2y ago

Are you looking at the direct dependencies? Because that list isn't neverending: https://github.com/Troels51/dtui/blob/main/Cargo.toml https://github.com/Troels51/dtui/blob/main/Cargo.toml For TUI it uses ratatui. The transitive dependencies can get quite large, though! I've never found the direct dependencies of any package to have superfluous dependencies (perhaps I haven't looked carefully enough?); they are always seem to be useful or easily provide functionality users expect to see, such as nice command line options and nice usage for them.

integricho 2y ago
Yes, but one cannot count transitive dependencies as if they are not your dependencies. They are still all needed to build your project, thus they are effectively yours as well. I find the pure number of dependencies involved worrying tbh (and this is not strictly related to your project, but if this is the case throughout the whole rust ecosystem, then it is a global problem).
- codr7 2y ago
  I second that reaction to many Rust projects I've compiled. Anything JS these days tends to look similar. I think part of the issue is that it's just so easy to get a package included in "the" central registry, and for many programming adepts that's like a badge of honor. So you get many tiny libraries, and many turn into balls of mud. Then they learn about reuse and happily pull in all dependencies they can think of. In Rust there seems to be a tendency to keep stuff out of the standard library, so everyone depends on their favorite solutions to everyday problems, which depends on their favorites and so on. But it is a problem, because somewhere along the line one of those indirect dependencies is going to start conflicting with something else in the graph; the bigger the graph, the more conflicts. And finally risk, because if a major indirect dependency goes belly up, that may well cause serious problems for your direct dependencies, which then makes it your problem. Also directly related to graph size.
  - dagurp 2y ago
    I feel like the trend is going the other way in modern JS

diath 2y ago

...and the build folder adds up to 700MB, that's quite grim for a program like this.

jeroenhd 2y ago

Depends on what you consider "many dependencies". Does this tool _need_ async tasks, a code generator for argument parsing, variable log level tracing hooks and JSON logging output? Probably not. A lot of this stuff is just developer convenience. I've seen other projects just include entire third party software repositories into their source tree rather than use a dependency manager, which works, I guess, but doesn't make the dependency tree all that obvious. When I use languages that don't have an easy dependency manager, I often still try to use common libraries, but I find that there are fewer easy dependencies. In C, I would just not bother with async I/O rather than import a library that makes it easy to do. I would use a shitty "works-for-me" string replace rather than a compliant parser for formats like JSON or XML. I'd steal some random SHA code rather than link to a library. I don't think this is that much of a problem as long as you're somewhat conservative in the kinds of dependencies you add. There's a trade-off between doing something yourself (and thus correctness) or stealing/vendoring the source code, and making use of a thriving library ecosystem.

resonious 2y ago

Yup, Rust as an ecosystem feels very similar to JS. Part of this might be because some core Rust facilities are lacking. For example, many projects use the "anyhow" library for error handling. It could also be because having a good package manager makes it so easy to add dependencies, and that reduced friction just naturally makes people tend to add many dependencies.

giyanani 2y ago
“Lacking” implies it’s an unintentional deficiency. Rust has deliberately taken a “batteries not included” approach due to learned experiences in python, where many std-lib packages are basically abandoned/not recommended anymore. This gives the ecosystem time to develop better approaches to things, rather than ossifying around the initial implementation. Yes, it does lead to a larger set of dependencies, but some of the most common packages (including anyhow) were written and are maintained by members of the rust lang organization/team itself.

WesolyKubeczek 2y ago

When Rust's ecosystem was booming, the JS/npm way of a myriad small libraries was viewed as superior. I think that there have been dozens of articles pre-left-pad about why it is good and why it's the future. I don't quite blame them, remembering how Python's ecosystem tried to mimic Java back in the day, how PHP tried, consciously or not, to do the same. I'm just wondering what the 2024-2025 dominant fad is going to look like.

steveklabnik 2y ago
Crates.io never had the behavior that led to left-pad, so it didn’t really change any opinions here. Small dependencies are not inherently good. Large dependencies are not inherently bad.
- WesolyKubeczek 2y ago
  Oh just you wait. It doesn't need to be a ragequitting and taking the toys away. If you are accustomed to having long chains of small twisty dependencies, all it takes is one Jia Tan of the Rust world to subtly alter a dependency of a dependency of a high profile dependency to own almost anything written in Rust. The said dependency can even ship precompiled bits of who-knows-which provenance, citing the same excuse as the one the author of serde used.
  - steveklabnik 2y ago
    That is a separate, and valid issue.
    - WesolyKubeczek 2y ago
      All I'm saying that in a maze of twisty little dependencies, all alike, it's very tedious to vet them all, and it's much too easy to end up with thousands of crates for the simplest things. And it's very easy to put nefarious things in once there is a will to do so. It all works well if everyone wants to play nice, but this doesn't scale. I don't know if Rust has the same quirk as JS/npm where a project can transitively depend on several versions of a module and they can coexist. Imagine a crate fixing some nasty shit, but some transitive dependency still pinning a vulnerable version. I contrast this with Go that has a rich enough standard library so you rarely need a dependency, and you don't need any for mundane stuff, and it makes you think hard about every dependency you introduce. And they got burned early enough so vendoring is a Good Thing now. And then I compare it with Perl, in that on one hand, it has lots of core modules, but then its standard library is lacking in weird places making you reach for CPAN in 100% of the cases writing a non-trivial program (HTTP/TLS comes to mind), and it would be similar to Rust except that CPAN is very slow-paced these days. I'm curious to see how it all evolves.