11 ms·

I think the dependency situation is pretty rough, and very few folks want to admit it. An example I recently stumbled upon: the cargo-watch[0] crate. At its co

I think the dependency situation is pretty rough, and very few folks want to admit it. An example I recently stumbled upon: the cargo-watch[0] crate. At its core its a pretty simple app. I watches for file changes, and re-runs the compiler. The implementation is less than 1000 lines of code. But what happens if I vendor the dependencies? It turns out, the deps add up to almost 4 million lines of Rust code, spread across 8000+ files. For a simple file-watcher. [0] https://crates.io/crates/cargo-watch https://crates.io/crates/cargo-watch

iforgotpassword 2y ago
Maybe they can learn from the Javascript folks, I heard they're very good at this.
- teaearlgraycold 2y ago
  Not sure if you're serious and talking about tree-shaking - or joking and talking about left-pad.
- wiseowise 2y ago
  Yes, unironically they’re now. Node has improved greatly in last two years. They always had native JSON support. Now have native test runner, watch, fetch, working on permission system à la deno, added WebSockets and working on native SQLite driver. All of this makes it a really attractive platform for prototyping which scales from hello world without any dependencies to production. Good luck experimenting with Rust without pulling half the internet with it. E: and they’re working on native TS support
  - Denvercoder9 2y ago
    > without any dependencies Nah, you still have those dependencies, they're just integrated in your interpreter. That has advantages (you're now only trusting a single source) and disadvantages (you always get all the goodies and the associated risks with that, even if you don't need them).
    - wiseowise 2y ago
      You’re being pedantic for the sake of being pedantic.
- mseepgood 2y ago
  No, they are the worst perpetrators re dependency hell.
- pjmlp 2y ago
  I think the interaction between both communities is exactly the reason of the current state.
- M4Luc 2y ago
  The Javascript folks are at least aware and self critical of this. In the Rust community it's sold as a great idea.
wiseowise 2y ago
What do you propose? To include it as part of std? Are you insane? That would bloat your binaries! (Still don’t understand how the smart compiler isn’t smart enough to remove dead code) And imagine if there’s an update that makes cargo-watch not BlAzInGlY fAsT™ but uLtRa BlAzInGlY fAsT™? /s
- willvarfar 2y ago
  How does Go compare? I'm curious as I don't know Go but it often gets mentioned here on HN as very lightweight. (A quick googling finds https://pkg.go.dev/search?q=watch https://pkg.go.dev/search?q=watch which makes me think that it's not any different?)
  - wiseowise 2y ago
    https://pkg.go.dev/std https://pkg.go.dev/std They’re much better.
    - lifthrasiir 2y ago
      I recall that I was very surprised to hear that Go standard library has extensive cryptographic stuffs. Generally that would be very unwise because they will become much harder to change or remove in spite of security issues. Turns out that this particular portion would be maintained by other maintainers who are actually trained in cryptography and security---something almost any other languages wouldn't be able to do with their resources.
conradludgate 2y ago
I bet most of those lines are from the generated windows api crates. They are notoriously monstrous
- dist1ll 2y ago
  You're right, the windows crate alone contributes 2.2M. I wonder if there's a way to deal with this issue.
  - Flex247A 2y ago
    Enabling FAT LTO reduces the final binary size but it isn't a permanent fix.
    - actionfromafar 2y ago
      Not permanent how?
      - Flex247A 2y ago
        Because the compiler ends up looking at all the functions anyway, only for the linker to discard them all.
  - lifthrasiir 2y ago
    The exact size of the `windows` crate depends on feature flags, because parsing 2.2M lines of code is always going to be very expensive even when you immediately discard them.
    - JoshTriplett 2y ago
      The parser is shockingly fast. The slow parts come after parsing, where we process all those function definitions and structure definitions, only to end up throwing 98% of them away. A challenging architectural problem that several of us are trying to get someone nerdsniped into: inverting the dependency tree, such that you first check what symbols exist in a large crate like windows, then go to all the crates depending on it and see what they actually consume, then go back and only compile the bits needed for those symbols. That'd be a massive improvement to compilation time, but it's a complicated change. You'd have to either do a two-pass compilation (first to get the symbol list, then again to compile the needed symbols) or leave that instance of the compiler running and feed the list of needed symbols back into it.
DanielHB 2y ago
The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe as possible and try to support minimum-dependency projects by having a broad std library. Also I am no expert, but I think file-watchers are definitely not simple at all, especially if they are multi-platform.
- dist1ll 2y ago
  That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.
  - selfmodruntime 2y ago
    Eh. The standard library is also a gigantic dependency written entirely by volunteers.
    - dist1ll 2y ago
      I don't have a problem with dependencies in principle. There's a good reason for standard libraries to contain a decent amount of code. It is a vector for supply chain attacks, but I also have a lot of trust in the Rust maintainers. The Rust standard library is exceptionally well-written from what I've seen, so I'm not too worried about it. FWIW I checked out the nightly toolchain, and it looks like the stdlib is less than 400k SLoC. So literally 10x smaller.
    - DanielHB 2y ago
      I am not a Rust expert but the thing with the standard libraries is that it only has peer dependencies with itself and they are all synced to the same version. Meaning if you only use the std lib you: 1) Will never include two different versions of the same peer dependency because of incompatible version requirements. 2) Will usually not have two dependencies relying on two different peer-dependencies that do the same thing. This can still happen for deprecated std lib features, but tends to be a much lesser issue. These two issues are usually the ones that cause dependency size explosion in projects.
moss2 2y ago
Same problem with JavaScript's NPM. And Python's PIP.
- jwr 2y ago
  This isn't necessarily a language problem, though, more of a "culture" problem, I think. I write in Clojure and I take great pains to avoid introducing dependencies. Contrary to the popular mantra, I will sometimes implement functionality instead of using a library, when the functionality is simple, or when the intersection area with the application is large (e.g. the library doesn't bring as many benefits as just using a "black box"). I will work to reduce my dependencies, and I will also carefully check if a library isn't just simple "glue code" (for example, for underlying Java functionality). This approach can be used with any language, it just needs to be pervasive in the culture.
  - orwin 2y ago
    I think this is made easier with Clojure macro capacity. In general, if you have powerfull metaprogramming tools, you trade dependency complexity with peace of mind (I still have flashbacks of C++ templates when i talk about metaprogramming :/. Does this qualify for PTSD?).
  - josephg 2y ago
    > This isn't necessarily a language problem, though, more of a "culture" problem, I think. Author here. We could make it a language problem by having the language sandbox dependencies by default. Seems like an easy win to me. Technical solutions are almost always easier to implement than social solutions.
    - Ygg2 2y ago
      Edit: replied to wrong person.
      - josephg 2y ago
        Huh? > It's throwing the baby and bathwater into lava. Is it really so controversial to want to be able to limit the access that utility crates like humansize or serde have to make arbitrary syscalls on my computer? Seems to me like we could get pretty far with just compile-time checks - and that would have no impact whatsoever on the compiled code (or its performance). I don't understand your criticism.
armitron 2y ago
This is the main reason we have banned Rust across my Org. Every third party library needs to be audited before being introduced as a vendored dependency which is not easy to do with the bloated dependency chains that Cargo promotes.
- selfmodruntime 2y ago
  How do you solve this for other languages you use?
  - armitron 2y ago
    Our main languages are Go and OCaml. We can leverage third party libraries without easily running into transitive dependency hell as there’s an implicit understanding in these communities that large number of dependencies is not a good thing. Or, expressed differently, there is coarser granularity in what ends up being a library. This is not the case with Cargo which has decided to follow the NPM approach.
    - lifthrasiir 2y ago
      At least in my experience, Go packages and Rust crates are much coarser than NPM packages. (Look at actual direct and indirect dependencies in cargo-watch to judge it by yourself.) I think Go prefers and actually has resource to keep mostly centralized approaches, while Rust crates are heavily distributed and it takes longer for the majority to settle on a single solution.
  - hu3 2y ago
    I've seen this approach go a long way with languages that have a large standard library. Go and C# .NET comes to mind.
- simonask 2y ago
  I'm sorry, but that feels like an incredibly poorly informed decision. One thing is to decide to vendor everything - that's your prerogative - but it's very likely that pulling everything in also pulls in tons of stuff that you aren't using, because recursively vendoring dependencies means you are also pulling in dev-dependencies, optional dependencies (including default-off features), and so on. For the things you do use, is it the number of crates that is the problem, or the amount of code? Because if the alternative is to develop it in-house, then... The alternative here is to include a lot of things in the standard library that doesn't belong there, because people seem to exclude standard libraries from their auditing, which is reasonable. Why is it not just as reasonable to exclude certain widespread ecosystem crates from auditing?
nullifidian 2y ago
Some amount of the risk from the "dependency jungle" situation could be alleviated by instituting "trusted" set of crates that are selected based on some popularity threshold, and with a rolling-release linux-distro-like stabilization chain, graduating from "testing" to "stable". If the Rust Foundation raised more money from the large companies, and hired devs to work as additional maintainers for these key crates, adding their signed-offs, it would be highly beneficial. That would have been a naturally evolving and changing equivalent to an extensive standard library. Mandating at least two maintainer sign offs for such critical set of crates would have been a good policy. Instead the large companies that use rust prefer to vet the crates on their own individually, duplicating the work the other companies do. The fact that nothing has changed in the NPM and Python worlds indicates that market forces pressure the decision makers to prefer the more risky approach, which prioritizes growth and fast iteration.
TechDebtDevin 2y ago
You fuck around...
School-Cotton 2y ago
Why, concretely, does this matter? Other than people who care about relatively obscure concerns like distro packaging, nobody is impeded in their work in any practical way by crates having a lot of transitive dependencies.
- zifpanachr23 2y ago
  Because for a lot of companies, especially ones in industries that Rust is supposedly hoping to displace C and C++ in, dependencies are a much larger concern than memory safety. They slow down velocity way more than running massive amounts of static and dynamic analysis tools to detect memory issues does in C. Every dependency is going to need explicit approval. And frankly, most crates would never receive that approval given the typical quality of a lot of the small utility crates and other transitive dependencies. Not to mention, the amount of transitive dependencies and their size in a lot of popular crates makes them functionally unauditable. This more than any other issue is I think what prevents Rust adoption outside of more liberal w.r.t dependencies companies in big tech and web parts of the economy. This is actually one positive in my view behind the rather unwieldy process of using dependencies and building C/C++ projects. There's a much bigger culture of care and minimalism w.r.t. choosing to take on a dependency in open source projects. Fwiw, the capabilities feature described in the post would go a very long way towards alleviating this issue.
  - anon-3988 2y ago
    Does C++ codebases with similar features parity somehow requires less code?
    - leoedin 2y ago
      There's probably a similar amount of code in the execution path, but the Rust ecosystem reliance on dependencies means that you're pulling in vast amounts of code that doesn't make it to your final application. A C++ library author is much more likely to just implement a small feature themselves rather than look for another 3rd party library for it. Adding dependencies to your library is a more involved and manual process, so most authors would do it very selectively. Saying that - a C++ library might depend on Boost and its 14 million LOC. Obviously it's not all being included in the final binary.
lifthrasiir 2y ago
I consciously remove and rewrite various dependencies at work, but I feel it's only a half of the whole story because either 1K or 4M lines of code seem to be equally inaccurate estimates for the appropriate number of LoC for this project. It seems that most dependencies of cargo-watch are pulled from three direct requirements: clap, cargo_metadata and watchexec. Clap would pull lots of CLI things that would be naturally platform-dependent, while cargo_metadata will surely pull most serde stuffs. Watchexec does have a room for improvement though, because it depends on command-group (maintained in the same org) which unconditionally requires Tokio! Who would have expected that? Once watchexec got improved on that aspect however, I think these requirements are indeed necessary for the project's goal and any further dependency removal will probably come with some downsides. A bigger problem here is that you can't easily fix other crates' excessive dependencies. Watchexec can be surely improved, but what if other crates are stuck at the older version of watchexec? There are some cases where you can just tweak Cargo.lock to get things aligned, but generally you can't do that. You have to live with excessive and/or duplicate dependencies (not a huge problem by itself, so it's default for most people) or work around with `[patch]` sections. (Cargo is actually in a better shape given that the second option is even possible at all!) In my opinion there should be some easy way to define a "stand-in" for given version of crate, so that such dependency issues can be more systematically worked around. But any such solution would be a huge research problem for any existing package manager.
- cmrdporcupine 2y ago
  It's frustrating because the grand-daddy of build systems with automatic transitive dependency management -- Maven -- already had tools from day one to handle this kind of thing through excluded dependencies (a blunt instrument, but sometimes necessary). In my experience, [patch] doesn't cut it or compare. That, and the maven repository is moderated. Unlike crates.io. Crates.io is a real problem. No namespaces, basically unmoderated, tons of abandoned stuff. Version hell like you're talking about. I have a hard time taking it at all seriously as a professional tool. And it's only going to get worse. If I were starting a Rust project from scratch inside a commercial company at this point, I'd use Bazel or Buck or GN/Ninja and vendored dependencies. No Cargo, no crates.io.
alexvitkov 2y ago
That's what inevitably happens when you make transitive dependencies easy and you have a culture of "if there's a library for it you must use it!" C/C++ are the only widely used languages without a popular npm-style package manager, and as a result most libraries are self-contained or have minimal, and often optional dependencies. efsw [1] is a 7000 lines (wc -l on the src directory) C++ FS watcher without dependencies. The single-header libraries that are popular in the game programming space (stb_* [2], cgltf [3], etc) as well as of course Dear ImGui [4] have been some of the most pleasant ones I've ever worked with. At this point I'm convinced that new package managers forbidding transitive dependencies would be an overall net gain. The biggest issue are large libraries that other ones justifiably depend on - OpenSSL, zlib, HTTP servers/clients, maybe even async runtimes. It's by no means an unsolvable problem, e.g. instead of having zlib as a transitive dependency, it could: 1. a library can still hard-depend on zlib, and just force the user to install it manually. 2. a library can provide generic compress/decompress callbacks, that the user can implement with whatever. 3. the compress/decompress functionality can be make standard [1] https://github.com/SpartanJ/efsw https://github.com/SpartanJ/efsw [2] https://github.com/nothings/stb https://github.com/nothings/stb [3] https://github.com/jkuhlmann/cgltf https://github.com/jkuhlmann/cgltf [4] https://github.com/ocornut/imgui https://github.com/ocornut/imgui
- lifthrasiir 2y ago
  > The single-header libraries that are popular in the game programming space (stb_* [2], cgltf [3], etc) as well as of course Dear ImGui have been some of the most pleasant ones I've ever worked with. The mainstream game programming doesn't use C at all. (Source: I had been a gamedev for almost a decade, and I mostly dealt with C# and sometimes C++ for low-level stuffs.) Even C++ is now out of fashion for at least a decade, anyone claiming that C++ is necessary for game programming is likely either an engine developer---a required, but very small portion of all gamedevs---or whoever haven't done significant game programming recently. Also, the reason that single-header libraries are rather popular in C is that otherwise they will be so, SO painful to use by the modern standard. As a result, those libraries have to be much more carefully designed than normal libraries either in C or other languages and contribute to their seemingly higher qualities. (Source: Again, I have written sizable single-header libraries in C and am aware of many issues from doing so.) I don't think this approach is scalable in general.
tbillington 2y ago
vendor + linecount unfortunately doesn't represent an accurate number of what cargo-watch would actually use. It includes all platform specific code behind compile time toggles even though only one would be used at any particular time, and doesn't account for the code not included because the feature wasn't enabled. https://doc.rust-lang.org/cargo/reference/features.html https://doc.rust-lang.org/cargo/reference/features.html whether those factors impact how you view the result of linecount is subjective also as one of the other commenters mentioned, cargo watch does more than just file watching
olalonde 2y ago
Why do you care how many lines of code the dependencies are? Compile time? Lack of disk space?
- cmrdporcupine 2y ago
  Some of us like to understand what's happening in the software we work on, and don't appreciate unnecessary complexity or unknown paths in the codebase that come through third party transitive dependencies. Some of us have licensing restrictions we have to adhere to. Some of us are very concerned about security and the potential problems of unaudited or unmoderated code that comes in through a long dependency chain. Hard learned lessons through years of dealing with this kind of thing: good software projects try to minimize the size of their impact crater.
- M4Luc 2y ago
  Security and maintenance. That's what's so compelling about Go. The std lib is not a pleasure to use. Or esp. fast and featureful. But you can rely on it. You don't depend on 1000 strangers on the internet that might have abandoned their Rust crate for 3 years and nobody noticed.
- ptsneves 2y ago
  Think of the problem as a bill of materials. Knowing the origin and that all the components of a part are fit for purpose is important for some applications. If I am making a small greenhouse i can buy steel profiles and not care about what steel are they from. If I am building a house I actually want a specific standardized profile because my structure's calculations rely on that. My house will collapse if they dont. If I am building a jet engine part I want a specific alloy and all the component metals and foundry details, and will reject if the provenance is not known or suitable[1]. If i am doing my own small script for personal purposes I dont care much about packaging and libraries, just that it accomplishes my immediate task on my environment. If I have a small tetris application I also dont care much about libraries, or their reliability. If I have a business selling my application and I am liable for its performance and security I damn sure want to know all about my potential liabilities and mitigate them. [1] https://www.usatoday.com/story/travel/airline-news/2024/06/14/faa-boeing-airbus-titanium-investigation/74101769007/ https://www.usatoday.com/story/travel/airline-news/2024/06/1...
cogman10 2y ago
This is a natural and not really scary thing. All code is built on mountains of dependencies that by their nature will do more than what you are using them for. For example, part of cargo watch is to bring in a win32 API wrapper library (which is just autogenerated bindings for win32 calls). Of course that thing is going to be massive while watch is using only a sliver of it in the case it's built for windows. The standard library for pretty much any language will have millions of lines of code, that's not scary even though your apps likely only use a fraction of what's offered. And have you ever glanced at C++'s boost library? That thing is monstrously big yet most devs using it are going to really only grab a few of the extensions. The alternative is the npm hellscape where you have a package for "isOdd" and a package for "is even" that can break the entire ecosystem if the owner is disgruntled because everything depends on them. Having fewer larger dependencies maintained and relied on by multiple people is much more ideal and where rust mostly finds itself.
- preommr 2y ago
  > The alternative is the npm hellscape where you have a package for "isOdd" and a package for "is even" that can break the entire ecosystem if the owner is disgruntled because everything depends on them. This used to be true 5-10 years ago. The js ecosystem moves fast and much has been done to fix the dependency sprawl.
  - dartos 2y ago
    I… don’t think that’s true. Just look at how many downloads some of those packages have today. Look at the dependency tree for a next or nuxt app. What the js world did is make their build systems somewhat sane, whatwith not needing babel in every project anymore.
    - throwitaway1123 2y ago
      > Look at the dependency tree for a next Looks ok to me: https://npmgraph.js.org/?q=next https://npmgraph.js.org/?q=next Ironically, most of the dependencies are actually Rust crates used by swc and turbopack [1][2]. Try running `cargo tree` on either of those crates, it's enlightening to say the least. And of course, Node has a built in file watcher, and even the most popular third party package for file watching (Chokidar) has a single dependency [3]. [1] https://github.com/vercel/next.js/blob/07a55e03a31b16da1d0855f0309e2d8cf32e6113/Cargo.toml https://github.com/vercel/next.js/blob/07a55e03a31b16da1d085... [2] https://github.com/swc-project/swc/blob/b94a0e1fd2b900b05c5f18d3d993a74ff9cc6e7d/Cargo.toml https://github.com/swc-project/swc/blob/b94a0e1fd2b900b05c5f... [3] https://npmgraph.js.org/?q=chokidar https://npmgraph.js.org/?q=chokidar
hawski 2y ago
The friction in C and C++ library ecosystem is sometimes a feature for this sole reason. Many libraries try to pull as little as possible and other things as optional.
M4Luc 2y ago
Another example is Axum. Using Go, C#, Deno or Node you don't even need any third party provided more or less secure and maintained lib. It all comes from the core teams.
dathinab 2y ago
> It turns out, the deps add up to almost 4 million lines of Rust code, spread across 8000+ files (Putting aside the question weather or not that pulls in dev dependencies and that watchin files can easily have OS specific aspecects so you might have different dependencies on different OSes and that neither lines and even less files are a good measurement of complexity and that this dependencies involve a lot of code from features of dependencies which aren't used and due to rust being complied in a reasonable way are reliable not included in the final binary in most cases. Also ignoring that cargo-watch isn't implementing file watching itself it's in many aspects a wrapper around watchexec which makes it much "thiner" then it would be otherwise.) What if that is needed for a reliable robust ecosystem? I mean, I know, it sound absurd but give it some thought. I wouldn't want every library to reinvent the wheel again and again for all kinds of things, so I would want them to use dependencies, I also would want them to use robust, tested, mature and maintained dependencies. Naturally this applies transitively. But what libraries become "robust, tested, mature and maintained" such which just provide a small for you good enough subset of a functionality or such which support the full functionality making it usable for a wider range of use-case? And with that in mind let's look at cargo-watch. First it's a CLI tool, so with the points above in mind you would need a good choice of a CLI parser, so you use e.g. clap. But at this point you already are pulling in a _huge_ number of lines of code from which the majority will be dead code eliminated. Through you don't have much choice, you don't want to reinvent the wheel and for a CLI libary to be widely successful (often needed it to be long term tested, maintained and e.g. forked if the maintainers disappear etc.) it needs to cover all widely needed CLI libary features, not just the subset you use. Then you need to handle configs, so you include dotenvy. You have a desktop notification sending feature again not reason to reinvent that so you pull in rust-notify. Handling path in a cross platform manner has tricky edge cases so camino and shell-escape get pulled in. You do log warnings so log+stderrlog get pulled in, which for message coloring and similar pull in atty and termcolor even through they probably just need a small subset of atty. But again no reason to reinvent the wheel especially for things so iffy/bug prone as reliably tty handling across many different ttys. Lastly watching files is harder then it seems and the notify library already implements it so we use that, wait it's quite low level and there is watchexec which provides exactly the interface we need so we use that (and if we would not we still would use most or all of watchexecs dependencies). And ignoring watchexec (around which the discussion would become more complex) with the standards above you wouldn't want to reimplement the functionality of any of this libraries yourself it's not even about implementation effort but stuff like overlooking edge cases, maintainability etc. And while you definitely can make a point that in some aspects you can and maybe should reduce some dependnecies etc. this isn't IMHO changing the general conclusion: You need most of this dependencies if you want to conform with standards pointed out above. And tbh. I have seen way way way to many cases of projects shaving of dependencies, adding "more compact wheel reinventions" for their subset and then ran into all kinds of bugs half a year later. Sometimes leading to the partial reimplementations becoming bigger and bigger until they weren't much smaller then the original project. Don't get me wrong there definitely are cases of (things you use from) dependencies being too small to make it worth it (e.g. left pad) or more common it takes more time (short term) to find a good library and review it then to reimplement it yourself (but long term it's quite often a bad idea). So idk. the issue is transitive dependencies or too many dependencies like at all. BUT I think there are issues wrt. handling software supply chain aspects. But that is a different kind of problem with different solutions. And sure not having dependencies avoid that problem, somewhat, but it's just replacing it IMHO with a different as bad problem.
the_clarence 2y ago
Agree. That was always my major gripe with Rust: it's not battery included. The big selling point of golang was the battery included part and I think that's really what is missing in Rust. I hope that with time more stuff can't get into the rust stdlib

iforgotpassword 2y ago

Maybe they can learn from the Javascript folks, I heard they're very good at this.

teaearlgraycold 2y ago
Not sure if you're serious and talking about tree-shaking - or joking and talking about left-pad.
wiseowise 2y ago
Yes, unironically they’re now. Node has improved greatly in last two years. They always had native JSON support. Now have native test runner, watch, fetch, working on permission system à la deno, added WebSockets and working on native SQLite driver. All of this makes it a really attractive platform for prototyping which scales from hello world without any dependencies to production. Good luck experimenting with Rust without pulling half the internet with it. E: and they’re working on native TS support
- Denvercoder9 2y ago
  > without any dependencies Nah, you still have those dependencies, they're just integrated in your interpreter. That has advantages (you're now only trusting a single source) and disadvantages (you always get all the goodies and the associated risks with that, even if you don't need them).
  - wiseowise 2y ago
    You’re being pedantic for the sake of being pedantic.
mseepgood 2y ago
No, they are the worst perpetrators re dependency hell.
pjmlp 2y ago
I think the interaction between both communities is exactly the reason of the current state.
M4Luc 2y ago
The Javascript folks are at least aware and self critical of this. In the Rust community it's sold as a great idea.

wiseowise 2y ago

What do you propose? To include it as part of std? Are you insane? That would bloat your binaries! (Still don’t understand how the smart compiler isn’t smart enough to remove dead code) And imagine if there’s an update that makes cargo-watch not BlAzInGlY fAsT™ but uLtRa BlAzInGlY fAsT™? /s

willvarfar 2y ago
How does Go compare? I'm curious as I don't know Go but it often gets mentioned here on HN as very lightweight. (A quick googling finds https://pkg.go.dev/search?q=watch https://pkg.go.dev/search?q=watch which makes me think that it's not any different?)
- wiseowise 2y ago
  https://pkg.go.dev/std https://pkg.go.dev/std They’re much better.
  - lifthrasiir 2y ago
    I recall that I was very surprised to hear that Go standard library has extensive cryptographic stuffs. Generally that would be very unwise because they will become much harder to change or remove in spite of security issues. Turns out that this particular portion would be maintained by other maintainers who are actually trained in cryptography and security---something almost any other languages wouldn't be able to do with their resources.

conradludgate 2y ago

I bet most of those lines are from the generated windows api crates. They are notoriously monstrous

dist1ll 2y ago
You're right, the windows crate alone contributes 2.2M. I wonder if there's a way to deal with this issue.
- Flex247A 2y ago
  Enabling FAT LTO reduces the final binary size but it isn't a permanent fix.
  - actionfromafar 2y ago
    Not permanent how?
    - Flex247A 2y ago
      Because the compiler ends up looking at all the functions anyway, only for the linker to discard them all.
- lifthrasiir 2y ago
  The exact size of the `windows` crate depends on feature flags, because parsing 2.2M lines of code is always going to be very expensive even when you immediately discard them.
  - JoshTriplett 2y ago
    The parser is shockingly fast. The slow parts come after parsing, where we process all those function definitions and structure definitions, only to end up throwing 98% of them away. A challenging architectural problem that several of us are trying to get someone nerdsniped into: inverting the dependency tree, such that you first check what symbols exist in a large crate like windows, then go to all the crates depending on it and see what they actually consume, then go back and only compile the bits needed for those symbols. That'd be a massive improvement to compilation time, but it's a complicated change. You'd have to either do a two-pass compilation (first to get the symbol list, then again to compile the needed symbols) or leave that instance of the compiler running and feed the list of needed symbols back into it.

DanielHB 2y ago

The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe as possible and try to support minimum-dependency projects by having a broad std library. Also I am no expert, but I think file-watchers are definitely not simple at all, especially if they are multi-platform.

dist1ll 2y ago
That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.
- selfmodruntime 2y ago
  Eh. The standard library is also a gigantic dependency written entirely by volunteers.
  - dist1ll 2y ago
    I don't have a problem with dependencies in principle. There's a good reason for standard libraries to contain a decent amount of code. It is a vector for supply chain attacks, but I also have a lot of trust in the Rust maintainers. The Rust standard library is exceptionally well-written from what I've seen, so I'm not too worried about it. FWIW I checked out the nightly toolchain, and it looks like the stdlib is less than 400k SLoC. So literally 10x smaller.
  - DanielHB 2y ago
    I am not a Rust expert but the thing with the standard libraries is that it only has peer dependencies with itself and they are all synced to the same version. Meaning if you only use the std lib you: 1) Will never include two different versions of the same peer dependency because of incompatible version requirements. 2) Will usually not have two dependencies relying on two different peer-dependencies that do the same thing. This can still happen for deprecated std lib features, but tends to be a much lesser issue. These two issues are usually the ones that cause dependency size explosion in projects.

moss2 2y ago

Same problem with JavaScript's NPM. And Python's PIP.

jwr 2y ago
This isn't necessarily a language problem, though, more of a "culture" problem, I think. I write in Clojure and I take great pains to avoid introducing dependencies. Contrary to the popular mantra, I will sometimes implement functionality instead of using a library, when the functionality is simple, or when the intersection area with the application is large (e.g. the library doesn't bring as many benefits as just using a "black box"). I will work to reduce my dependencies, and I will also carefully check if a library isn't just simple "glue code" (for example, for underlying Java functionality). This approach can be used with any language, it just needs to be pervasive in the culture.
- orwin 2y ago
  I think this is made easier with Clojure macro capacity. In general, if you have powerfull metaprogramming tools, you trade dependency complexity with peace of mind (I still have flashbacks of C++ templates when i talk about metaprogramming :/. Does this qualify for PTSD?).
- josephg 2y ago
  > This isn't necessarily a language problem, though, more of a "culture" problem, I think. Author here. We could make it a language problem by having the language sandbox dependencies by default. Seems like an easy win to me. Technical solutions are almost always easier to implement than social solutions.
  - Ygg2 2y ago
    Edit: replied to wrong person.
    - josephg 2y ago
      Huh? > It's throwing the baby and bathwater into lava. Is it really so controversial to want to be able to limit the access that utility crates like humansize or serde have to make arbitrary syscalls on my computer? Seems to me like we could get pretty far with just compile-time checks - and that would have no impact whatsoever on the compiled code (or its performance). I don't understand your criticism.

armitron 2y ago

This is the main reason we have banned Rust across my Org. Every third party library needs to be audited before being introduced as a vendored dependency which is not easy to do with the bloated dependency chains that Cargo promotes.

selfmodruntime 2y ago
How do you solve this for other languages you use?
- armitron 2y ago
  Our main languages are Go and OCaml. We can leverage third party libraries without easily running into transitive dependency hell as there’s an implicit understanding in these communities that large number of dependencies is not a good thing. Or, expressed differently, there is coarser granularity in what ends up being a library. This is not the case with Cargo which has decided to follow the NPM approach.
  - lifthrasiir 2y ago
    At least in my experience, Go packages and Rust crates are much coarser than NPM packages. (Look at actual direct and indirect dependencies in cargo-watch to judge it by yourself.) I think Go prefers and actually has resource to keep mostly centralized approaches, while Rust crates are heavily distributed and it takes longer for the majority to settle on a single solution.
- hu3 2y ago
  I've seen this approach go a long way with languages that have a large standard library. Go and C# .NET comes to mind.
simonask 2y ago
I'm sorry, but that feels like an incredibly poorly informed decision. One thing is to decide to vendor everything - that's your prerogative - but it's very likely that pulling everything in also pulls in tons of stuff that you aren't using, because recursively vendoring dependencies means you are also pulling in dev-dependencies, optional dependencies (including default-off features), and so on. For the things you do use, is it the number of crates that is the problem, or the amount of code? Because if the alternative is to develop it in-house, then... The alternative here is to include a lot of things in the standard library that doesn't belong there, because people seem to exclude standard libraries from their auditing, which is reasonable. Why is it not just as reasonable to exclude certain widespread ecosystem crates from auditing?

nullifidian 2y ago

Some amount of the risk from the "dependency jungle" situation could be alleviated by instituting "trusted" set of crates that are selected based on some popularity threshold, and with a rolling-release linux-distro-like stabilization chain, graduating from "testing" to "stable". If the Rust Foundation raised more money from the large companies, and hired devs to work as additional maintainers for these key crates, adding their signed-offs, it would be highly beneficial. That would have been a naturally evolving and changing equivalent to an extensive standard library. Mandating at least two maintainer sign offs for such critical set of crates would have been a good policy. Instead the large companies that use rust prefer to vet the crates on their own individually, duplicating the work the other companies do. The fact that nothing has changed in the NPM and Python worlds indicates that market forces pressure the decision makers to prefer the more risky approach, which prioritizes growth and fast iteration.

TechDebtDevin 2y ago

You fuck around...

School-Cotton 2y ago

Why, concretely, does this matter? Other than people who care about relatively obscure concerns like distro packaging, nobody is impeded in their work in any practical way by crates having a lot of transitive dependencies.

zifpanachr23 2y ago
Because for a lot of companies, especially ones in industries that Rust is supposedly hoping to displace C and C++ in, dependencies are a much larger concern than memory safety. They slow down velocity way more than running massive amounts of static and dynamic analysis tools to detect memory issues does in C. Every dependency is going to need explicit approval. And frankly, most crates would never receive that approval given the typical quality of a lot of the small utility crates and other transitive dependencies. Not to mention, the amount of transitive dependencies and their size in a lot of popular crates makes them functionally unauditable. This more than any other issue is I think what prevents Rust adoption outside of more liberal w.r.t dependencies companies in big tech and web parts of the economy. This is actually one positive in my view behind the rather unwieldy process of using dependencies and building C/C++ projects. There's a much bigger culture of care and minimalism w.r.t. choosing to take on a dependency in open source projects. Fwiw, the capabilities feature described in the post would go a very long way towards alleviating this issue.
- anon-3988 2y ago
  Does C++ codebases with similar features parity somehow requires less code?
  - leoedin 2y ago
    There's probably a similar amount of code in the execution path, but the Rust ecosystem reliance on dependencies means that you're pulling in vast amounts of code that doesn't make it to your final application. A C++ library author is much more likely to just implement a small feature themselves rather than look for another 3rd party library for it. Adding dependencies to your library is a more involved and manual process, so most authors would do it very selectively. Saying that - a C++ library might depend on Boost and its 14 million LOC. Obviously it's not all being included in the final binary.

lifthrasiir 2y ago

I consciously remove and rewrite various dependencies at work, but I feel it's only a half of the whole story because either 1K or 4M lines of code seem to be equally inaccurate estimates for the appropriate number of LoC for this project. It seems that most dependencies of cargo-watch are pulled from three direct requirements: clap, cargo_metadata and watchexec. Clap would pull lots of CLI things that would be naturally platform-dependent, while cargo_metadata will surely pull most serde stuffs. Watchexec does have a room for improvement though, because it depends on command-group (maintained in the same org) which unconditionally requires Tokio! Who would have expected that? Once watchexec got improved on that aspect however, I think these requirements are indeed necessary for the project's goal and any further dependency removal will probably come with some downsides. A bigger problem here is that you can't easily fix other crates' excessive dependencies. Watchexec can be surely improved, but what if other crates are stuck at the older version of watchexec? There are some cases where you can just tweak Cargo.lock to get things aligned, but generally you can't do that. You have to live with excessive and/or duplicate dependencies (not a huge problem by itself, so it's default for most people) or work around with `[patch]` sections. (Cargo is actually in a better shape given that the second option is even possible at all!) In my opinion there should be some easy way to define a "stand-in" for given version of crate, so that such dependency issues can be more systematically worked around. But any such solution would be a huge research problem for any existing package manager.

cmrdporcupine 2y ago
It's frustrating because the grand-daddy of build systems with automatic transitive dependency management -- Maven -- already had tools from day one to handle this kind of thing through excluded dependencies (a blunt instrument, but sometimes necessary). In my experience, [patch] doesn't cut it or compare. That, and the maven repository is moderated. Unlike crates.io. Crates.io is a real problem. No namespaces, basically unmoderated, tons of abandoned stuff. Version hell like you're talking about. I have a hard time taking it at all seriously as a professional tool. And it's only going to get worse. If I were starting a Rust project from scratch inside a commercial company at this point, I'd use Bazel or Buck or GN/Ninja and vendored dependencies. No Cargo, no crates.io.

alexvitkov 2y ago

That's what inevitably happens when you make transitive dependencies easy and you have a culture of "if there's a library for it you must use it!" C/C++ are the only widely used languages without a popular npm-style package manager, and as a result most libraries are self-contained or have minimal, and often optional dependencies. efsw [1] is a 7000 lines (wc -l on the src directory) C++ FS watcher without dependencies. The single-header libraries that are popular in the game programming space (stb_* [2], cgltf [3], etc) as well as of course Dear ImGui [4] have been some of the most pleasant ones I've ever worked with. At this point I'm convinced that new package managers forbidding transitive dependencies would be an overall net gain. The biggest issue are large libraries that other ones justifiably depend on - OpenSSL, zlib, HTTP servers/clients, maybe even async runtimes. It's by no means an unsolvable problem, e.g. instead of having zlib as a transitive dependency, it could: 1. a library can still hard-depend on zlib, and just force the user to install it manually. 2. a library can provide generic compress/decompress callbacks, that the user can implement with whatever. 3. the compress/decompress functionality can be make standard [1] https://github.com/SpartanJ/efsw https://github.com/SpartanJ/efsw [2] https://github.com/nothings/stb https://github.com/nothings/stb [3] https://github.com/jkuhlmann/cgltf https://github.com/jkuhlmann/cgltf [4] https://github.com/ocornut/imgui https://github.com/ocornut/imgui

lifthrasiir 2y ago
> The single-header libraries that are popular in the game programming space (stb_* [2], cgltf [3], etc) as well as of course Dear ImGui have been some of the most pleasant ones I've ever worked with. The mainstream game programming doesn't use C at all. (Source: I had been a gamedev for almost a decade, and I mostly dealt with C# and sometimes C++ for low-level stuffs.) Even C++ is now out of fashion for at least a decade, anyone claiming that C++ is necessary for game programming is likely either an engine developer---a required, but very small portion of all gamedevs---or whoever haven't done significant game programming recently. Also, the reason that single-header libraries are rather popular in C is that otherwise they will be so, SO painful to use by the modern standard. As a result, those libraries have to be much more carefully designed than normal libraries either in C or other languages and contribute to their seemingly higher qualities. (Source: Again, I have written sizable single-header libraries in C and am aware of many issues from doing so.) I don't think this approach is scalable in general.

tbillington 2y ago

vendor + linecount unfortunately doesn't represent an accurate number of what cargo-watch would actually use. It includes all platform specific code behind compile time toggles even though only one would be used at any particular time, and doesn't account for the code not included because the feature wasn't enabled. https://doc.rust-lang.org/cargo/reference/features.html https://doc.rust-lang.org/cargo/reference/features.html whether those factors impact how you view the result of linecount is subjective also as one of the other commenters mentioned, cargo watch does more than just file watching

olalonde 2y ago

Why do you care how many lines of code the dependencies are? Compile time? Lack of disk space?

cmrdporcupine 2y ago
Some of us like to understand what's happening in the software we work on, and don't appreciate unnecessary complexity or unknown paths in the codebase that come through third party transitive dependencies. Some of us have licensing restrictions we have to adhere to. Some of us are very concerned about security and the potential problems of unaudited or unmoderated code that comes in through a long dependency chain. Hard learned lessons through years of dealing with this kind of thing: good software projects try to minimize the size of their impact crater.
M4Luc 2y ago
Security and maintenance. That's what's so compelling about Go. The std lib is not a pleasure to use. Or esp. fast and featureful. But you can rely on it. You don't depend on 1000 strangers on the internet that might have abandoned their Rust crate for 3 years and nobody noticed.
ptsneves 2y ago
Think of the problem as a bill of materials. Knowing the origin and that all the components of a part are fit for purpose is important for some applications. If I am making a small greenhouse i can buy steel profiles and not care about what steel are they from. If I am building a house I actually want a specific standardized profile because my structure's calculations rely on that. My house will collapse if they dont. If I am building a jet engine part I want a specific alloy and all the component metals and foundry details, and will reject if the provenance is not known or suitable[1]. If i am doing my own small script for personal purposes I dont care much about packaging and libraries, just that it accomplishes my immediate task on my environment. If I have a small tetris application I also dont care much about libraries, or their reliability. If I have a business selling my application and I am liable for its performance and security I damn sure want to know all about my potential liabilities and mitigate them. [1] https://www.usatoday.com/story/travel/airline-news/2024/06/14/faa-boeing-airbus-titanium-investigation/74101769007/ https://www.usatoday.com/story/travel/airline-news/2024/06/1...

cogman10 2y ago

This is a natural and not really scary thing. All code is built on mountains of dependencies that by their nature will do more than what you are using them for. For example, part of cargo watch is to bring in a win32 API wrapper library (which is just autogenerated bindings for win32 calls). Of course that thing is going to be massive while watch is using only a sliver of it in the case it's built for windows. The standard library for pretty much any language will have millions of lines of code, that's not scary even though your apps likely only use a fraction of what's offered. And have you ever glanced at C++'s boost library? That thing is monstrously big yet most devs using it are going to really only grab a few of the extensions. The alternative is the npm hellscape where you have a package for "isOdd" and a package for "is even" that can break the entire ecosystem if the owner is disgruntled because everything depends on them. Having fewer larger dependencies maintained and relied on by multiple people is much more ideal and where rust mostly finds itself.

preommr 2y ago
> The alternative is the npm hellscape where you have a package for "isOdd" and a package for "is even" that can break the entire ecosystem if the owner is disgruntled because everything depends on them. This used to be true 5-10 years ago. The js ecosystem moves fast and much has been done to fix the dependency sprawl.
- dartos 2y ago
  I… don’t think that’s true. Just look at how many downloads some of those packages have today. Look at the dependency tree for a next or nuxt app. What the js world did is make their build systems somewhat sane, whatwith not needing babel in every project anymore.
  - throwitaway1123 2y ago
    > Look at the dependency tree for a next Looks ok to me: https://npmgraph.js.org/?q=next https://npmgraph.js.org/?q=next Ironically, most of the dependencies are actually Rust crates used by swc and turbopack [1][2]. Try running `cargo tree` on either of those crates, it's enlightening to say the least. And of course, Node has a built in file watcher, and even the most popular third party package for file watching (Chokidar) has a single dependency [3]. [1] https://github.com/vercel/next.js/blob/07a55e03a31b16da1d0855f0309e2d8cf32e6113/Cargo.toml https://github.com/vercel/next.js/blob/07a55e03a31b16da1d085... [2] https://github.com/swc-project/swc/blob/b94a0e1fd2b900b05c5f18d3d993a74ff9cc6e7d/Cargo.toml https://github.com/swc-project/swc/blob/b94a0e1fd2b900b05c5f... [3] https://npmgraph.js.org/?q=chokidar https://npmgraph.js.org/?q=chokidar

hawski 2y ago

The friction in C and C++ library ecosystem is sometimes a feature for this sole reason. Many libraries try to pull as little as possible and other things as optional.

M4Luc 2y ago

Another example is Axum. Using Go, C#, Deno or Node you don't even need any third party provided more or less secure and maintained lib. It all comes from the core teams.

dathinab 2y ago

> It turns out, the deps add up to almost 4 million lines of Rust code, spread across 8000+ files (Putting aside the question weather or not that pulls in dev dependencies and that watchin files can easily have OS specific aspecects so you might have different dependencies on different OSes and that neither lines and even less files are a good measurement of complexity and that this dependencies involve a lot of code from features of dependencies which aren't used and due to rust being complied in a reasonable way are reliable not included in the final binary in most cases. Also ignoring that cargo-watch isn't implementing file watching itself it's in many aspects a wrapper around watchexec which makes it much "thiner" then it would be otherwise.) What if that is needed for a reliable robust ecosystem? I mean, I know, it sound absurd but give it some thought. I wouldn't want every library to reinvent the wheel again and again for all kinds of things, so I would want them to use dependencies, I also would want them to use robust, tested, mature and maintained dependencies. Naturally this applies transitively. But what libraries become "robust, tested, mature and maintained" such which just provide a small for you good enough subset of a functionality or such which support the full functionality making it usable for a wider range of use-case? And with that in mind let's look at cargo-watch. First it's a CLI tool, so with the points above in mind you would need a good choice of a CLI parser, so you use e.g. clap. But at this point you already are pulling in a _huge_ number of lines of code from which the majority will be dead code eliminated. Through you don't have much choice, you don't want to reinvent the wheel and for a CLI libary to be widely successful (often needed it to be long term tested, maintained and e.g. forked if the maintainers disappear etc.) it needs to cover all widely needed CLI libary features, not just the subset you use. Then you need to handle configs, so you include dotenvy. You have a desktop notification sending feature again not reason to reinvent that so you pull in rust-notify. Handling path in a cross platform manner has tricky edge cases so camino and shell-escape get pulled in. You do log warnings so log+stderrlog get pulled in, which for message coloring and similar pull in atty and termcolor even through they probably just need a small subset of atty. But again no reason to reinvent the wheel especially for things so iffy/bug prone as reliably tty handling across many different ttys. Lastly watching files is harder then it seems and the notify library already implements it so we use that, wait it's quite low level and there is watchexec which provides exactly the interface we need so we use that (and if we would not we still would use most or all of watchexecs dependencies). And ignoring watchexec (around which the discussion would become more complex) with the standards above you wouldn't want to reimplement the functionality of any of this libraries yourself it's not even about implementation effort but stuff like overlooking edge cases, maintainability etc. And while you definitely can make a point that in some aspects you can and maybe should reduce some dependnecies etc. this isn't IMHO changing the general conclusion: You need most of this dependencies if you want to conform with standards pointed out above. And tbh. I have seen way way way to many cases of projects shaving of dependencies, adding "more compact wheel reinventions" for their subset and then ran into all kinds of bugs half a year later. Sometimes leading to the partial reimplementations becoming bigger and bigger until they weren't much smaller then the original project. Don't get me wrong there definitely are cases of (things you use from) dependencies being too small to make it worth it (e.g. left pad) or more common it takes more time (short term) to find a good library and review it then to reimplement it yourself (but long term it's quite often a bad idea). So idk. the issue is transitive dependencies or too many dependencies like at all. BUT I think there are issues wrt. handling software supply chain aspects. But that is a different kind of problem with different solutions. And sure not having dependencies avoid that problem, somewhat, but it's just replacing it IMHO with a different as bad problem.

the_clarence 2y ago

Agree. That was always my major gripe with Rust: it's not battery included. The big selling point of golang was the battery included part and I think that's really what is missing in Rust. I hope that with time more stuff can't get into the rust stdlib