10 ms·

Rust’s dependencies are starting to worry me

• oldpersonintx 1y ago
  [dead]
• palata 1y ago
  Similar feeling here. Cargo makes it so simple to add tons of dependencies that it is really hard not to do it. But that does not stop here: even if I try to be careful with adding dependencies, a couple dependencies are likely to pull tens of transitive dependencies each. "Then don't depend on them", you say. Sure, but that means I won't write my project, because I won't write those things from scratch. I could probably audit the dependency (if it wasn't pulling 50 packages itself), but I can't reasonably write it myself. It is different with C++: I can often find dependencies that don't pull tens of transitive dependencies in C++. Maybe because it's harder to add dependencies, maybe because the ecosystem is more mature, I don't know. But it feels like the philosophy in Rust is to pull many small packages, so it doesn't seem like it will change. And that's a pity, because I like Rust-the-language better than C++-the-language. It just feels like I trade "it's not memory-safe" for "you have to pull tons of random code from the Internet".
  • jampekka 1y ago
    I take bit less unstable dependencies over the total mess of C++ dependencies with CMake, shared libraries, version conflicts etc any time. There's probably also a bit of an illusion about C++ transitive dependencies due to them usually being precompiled (because compiling them is such pain).
    • ChocolateGod 1y ago
      The whole pkgconfig, cmake, autotools etc ecosystem is insane compared to how Rust and Go do things. It's part of the reason why software distribution on Linux has been pushed to using containers, removing the point of having shared libraries. I think Google with it's C++ replacement (Carbon) plans on doing it's own system.
      • skydhash 1y ago
        From my point of view, the issue stems from developers wanting to control distribution. Fine if it's for your own usage, not really if you're planning for others to use it. You will find the most convoluted build system just because they have a pet platform they want to specially support making it hell to do anything on others. It could be better, but the current solutions (npm, go, python,...) favor only the developers, not the maintainers and packagers.
        • ChocolateGod 1y ago
          There's examples of maintainers/packagers effectively sabotaging other peoples projects when making packages for distros, whether that's shipping them broken, ancient versions etc. e.g. Bottles, WebkitGTK (distros liked keeping this one held back even though doing so is a security risk) IMHO it shouldn't be the responsibility of the OS vendor to package third party applications.
          • jampekka 1y ago
            Distro maintainers/packagers are who keep the current software stacks running. It's rather amazing how they manage to keep the billion or so lines of separately written code working in unison. That said, the labor needed to keep the stuff together could be reduced a lot by the more ergonomical and universal packaging and distribution methods like Cargo (and, dare I say, npm). I think some kind of a better bridge between developers and distros could be found here.
            • skydhash 1y ago
              > I think some kind of a better bridge between developers and distros could be found here. I don’t think there’s a need to do so. Only discipline is needed, by using stable and mature dependencies, and documenting the building process. And maybe some guides/scripts for the most popular distros.
            • ChocolateGod 1y ago
              > > I think some kind of a better bridge between developers and distros could be found here. Every tom dick and harry is making their own distro these days (even if they're just respins of Arch with Calamares and some questionable theme settings), why add more work onto developers? We have things like Flatpak and Docker now that let application developers ignore the distros and stop them breaking things, unless you're Ubuntu whom is constantly begging to get purchased by Microsoft.
  • antonvs 1y ago
    > Sure, but that means I won't write my project, because I won't write those things from scratch. You need to think a bit harder about that, to help you decide whether your position is rational.
    • MeetingsBrowser 1y ago
      This confuses me as well. Is the implied solution to choose a language where you are forced to write those things from scratch?
      • palata 1y ago
        My point is that if, in the language, everybody is incentivise to use fewer dependencies, then a random library that I would not write myself (because it is an entire project in itself) would have fewer dependencies. Because it is not the case, either I take that library and accept its transitive dependencies, or I don't have a library at all. In Rust, I'm sometimes actually tempted to wrap a C/C++ library (and its few dependencies) instead of getting the Rust alternative (and its gazillion dependencies).
    • palata 1y ago
      And you need to think a bit about that (probably not very hard), to help you decide whether I'm irrational or whether you may not have totally understood my point.
  • perrygeo 1y ago
    > the philosophy in Rust is to pull many small package I'm not sure it's a philosophy, more a pragmatic consideration for compilation speeds. Anyone who's done a non-trivial amount of Rust knows that moment when the project gets too big and needs to split into separate crates. It's kinda sad that you can't organize code according to proper abstractions, many times I feel forced to refactor for compiler performance.
  • X0Refraction 1y ago
    This was linked from the top comment on the Rust subreddit: https://wiki.alopex.li/LetsBeRealAboutDependencies https://wiki.alopex.li/LetsBeRealAboutDependencies I think it makes a good point that some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded. To some degree that is a plus though as you likely trust the maintainers of your OS distribution to provide stable, supported libraries. As other commenters have said, perhaps this is an area where the Rust maintainers could provide some kind of extended standard library where they don't guarantee backwards compatibility forever, but do provide guarantees about ongoing fixes for security issues.
    • palata 1y ago
      > some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded. Not in my case. I manually compile all the dependencies (either because I need to cross-compile, or because I may need to patch them, etc). So I clearly see all the transitive dependencies I need in C++. And I need a lot less than in Rust, by a long shot.
      • elabajaba 1y ago
        Part of the rust dependency issue is that the compiler only multithreads at the crate level currently (slowly being improved on nightly, but there's still some bugs before they can roll out the parallel compiler), so most libraries split themselves up into a ton of small crates because otherwise they just take too long to compile. edit: Also, `cargo-vet` is useful for distributed auditing of crates. There's also `cargo-crev`, but afaik it doesn't have buy in from the megacorps like cargo-vet and last I checked didn't have as many/as consistent reviews. https://github.com/mozilla/cargo-vet https://github.com/mozilla/cargo-vet https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev
        • zahlman 1y ago
          > so most libraries split themselves up into a ton of small crates because otherwise they just take too long to compile. In practice, does this make it feasible to pick and choose the pieces you actually need?
  • imtringued 1y ago
    I have been wasting 6 hours yesterday on getting the bullet examples to compile outside of bullet itself with no success. It's more likely that a lot of software simply doesn't get written because C++ and CMake are a pain in the ass.
    • palata 1y ago
      I find CMake pretty easy, and I only use a few core features from it. Usually the pain comes from completely wrong setups by people who didn't learn the basic. But it's true of everything, I think.
• stefanos82 1y ago
  Existing discussion in https://news.ycombinator.com/item?id=43930640 https://news.ycombinator.com/item?id=43930640
• demarq 1y ago
  > Many call for adding more to the rust standard library much like Go This is the way.
  • dralley 1y ago
    There should be a second stdlib with relaxed stability guarantees. Don't fill the normal stdlib full of cruft that can never be changed again.
    • demarq 1y ago
      Yeah, I agree. Something like the Boost lib for C++
      • jerf 1y ago
        A strong advantage of that approach is that you don't need to be the core Rust team to do it. Anyone who wants to do this can just start doing it now.
        • shepmaster 1y ago
          I agree. Unfortunately, I think that a lot of the people who ask for a bigger standard library really just want (a) someone else to do the work (b) someone they can trust. The people working on Rust are a finite (probably overextended!) set of people and you can't just add more work to their plate. "Just" making the standard library bigger is probably a non-starter. I think it'd be great if some group of people took up the very hard work to curate a set of crates that everyone would use and provide a nice façade to them, completely outside of the Rust team umbrella. Then people can start using this Katamari crate to prove out the usefulness of it. However, many people wouldn't use it. I wouldn't because I simply don't care and am happy adding my dependencies one-by-one with minimal feature sets. Others wouldn't because it doesn't have the mystical blessing/seal-of-approval of the Rust team.
          • demarq 1y ago
• klooney 1y ago
  > dotenv is unmaintained. How much maintenance could you possibly need to load secrets from .env into the environment.
  • iammrpayments 1y ago
    I find hilarious when people judge the quality of a repository by how many commits it has, as if 10.000 commits means the code is better.
  • shepmaster 1y ago
    I agree with your general point, but for this specific functionality, I’ll point out that setting environment variables of the current process is unsafe. It took us a long time to realize it so the function wasn’t actually marked as unsafe until the Rust 2024 edition. What this means in practice is that the call to invoke dotenv should also be marked as unsafe so that the invoker can ensure safety by placing it at the right place. If no one is maintaining the crate, that won’t happen and someone might try to load environment variables at a bad time.
    • andy_xor_andrew 1y ago
      ok, I'm hooked - how is setting an env var in the current process unsafe? My gut says it's not unsafe in a memory-ownership sense, but rather in a race condition sense? whatever the issue is, "setting an env var is unsafe" is so interesting to me that I'm now craving a blog post explaining this
      • Orangeair 1y ago
        It's a long standing bug, setenv and unsetenv are not thread-safe https://www.evanjones.ca/setenv-is-not-thread-safe.html https://www.evanjones.ca/setenv-is-not-thread-safe.html
        • robertlagrant 1y ago
          I honestly think using setenv is just a terrible idea.
• XxiXx 1y ago
  I think it's a "cultural" thing. With Go you often find developers/projects proudly mentioning that any or just a few non-std dependencies are used. Coming from Go it really feels strange when you see pages of dependencies scrolling over your screen when you build a Rust project.
  • deleted 1y ago
    [deleted]
  • sophacles 1y ago
    I have yet to come across a go project that doesn't pull in tons of 3rd party code as well. It seems like maybe you're over-stating the "culture" a bit.
    • meling 1y ago
      Yeah, while I’ve seen some great libraries that follow the practice of minimizing their dependencies, I’m a bit annoyed with the amount of dependencies that docker will bring along [1]. I’ve been on the lookout for alternatives for my docker needs, but the state of podman, buildah and some others that I checked is similar. They all bring in roughly the same number of dependencies… if anyone knows of a stripped down Go lib that can be used to build from a Dockerfile, pull, and run a container, I would be grateful for any suggestions. Heck docker / moby isn’t even using go.mod proper. [1] https://github.com/moby/moby/blob/master/vendor.mod https://github.com/moby/moby/blob/master/vendor.mod
      • lantastic 1y ago
        Wow, that's massive. I guess it's inevitable that a popular piece of open-source software for end-users will be compelled to accrue dependencies due to popular demand for features that require them. I feel Telegraf made a good compromise: out of the box, it comes with a _ton_ of stuff[1] to monitor everything, but they make it possible to build only with pieces that you need via build tags, and even provide a tool to extract said tags from your telegraf config[2]. But lots of supply-chain security stuff assume everything in go.mod is used, so that can results in a lot of noise. [1] https://github.com/influxdata/telegraf/blob/master/go.mod https://github.com/influxdata/telegraf/blob/master/go.mod [2] https://github.com/influxdata/telegraf/tree/master/tools/custom_builder https://github.com/influxdata/telegraf/tree/master/tools/cus...
• neilv 1y ago
  In the past (not in Rust, but other languages), for important systems, I've instituted policies of minimizing dependencies from these language-specific package repositories, and for the ones you do use, having to copy it to our own repos and audit each update before use. But that's not practical for all situations. For example, Web frontend developer culture might be the worst environment, to the point you often can't get many things done in feasible time, if you don't adopt the same reckless practices. I'm also seeing it now with the cargo-culting of opaque self-hosted AI tools and models. For learning and experimenting, I'd spend more time sufficiently compartmentalizing an individual tool than with using it. This weekend, I'm dusting off my Rust skills, for a small open source employability project (so I can't invest in expensive dependency management on this one). The main thing thing bothering me isn't allocation management, but the sinking feeling when I watch the cast-of-thousands explosion of transitive dependencies for the UI and async libraries that I want to use. It's only a matter of time before one of those is compromised, if not already, and one is all it takes.
  • wofo 1y ago
    There are some voices trying to address this security risk (e.g. the proponents of this new RFC: https://github.com/rust-lang/rfcs/pull/3810 https://github.com/rust-lang/rfcs/pull/3810). However, for some reason (probably culture) there isn't much momentum yet to change the status quo.
    • cogman10 1y ago
      The rust RFC process has, frankly, become somewhat of a CF. There's literally 1000s of RFCs for rust with only a small handful that are integrated. Having this forest, IMO, makes it hard for any given proposal to really stand out. Further, it makes duplicate effort almost inevitable. Rust's RFC process is effectively a dead letter box for most.
      • geodel 1y ago
        I think they can constitute committee for RFC review process(in case there is none today) and based on recommendation multiple domain specific teams/ groups can be created to review RFCs in timely manner.
• thrance 1y ago
  I had the same concerns when I started using Rust, but then I eventually embraced it, for better or worse. Cargo makes it so your build almost never breaks (it's happened maybe twice for the 8 years I've been doing Rust). Plus there are still way less vulnerabilities with Rust projects than non-Rust projects, in spite of the crazy number of dependencies. If I was to design a Rust 2.0, I'd make it so dependencies need permissions to access IO, or unsafe code, etc.
• csomar 1y ago
  > when checking a rust security advisory mentioning that dotenv is unmaintained This is a problem with all languages and actually an area where Rust shines (due to editions). Your pulled in packages will compile as they previously did. This is not true for garbage collected languages (pun intended). > Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust .... How could I ever audit all of that code? Again, another area where Rust shines. You can audit and most importantly modify the code. This is not that easy if you were using Nodejs where the runtimes are behind node/v8 or whatever. You compile these things (including TLS) yourself and have full control over them. That's why Tokio is huge.
  • lolinder 1y ago
    > This is not true for garbage collected languages JavaScript is backwards compatible going back effectively forever, as is Java. Rust's unique system is having a way to make breaking changes to the language without breaking old code, not that they prioritize supporting old code indefinitely. The libraries are a different story—you're likely to have things break under you that rely on older versions of libraries when you update—but I don't see Rust actually having solved that. > You can audit and most importantly modify the code. This is not that easy if you were using Nodejs where the runtimes are behind node/v8 or whatever. Node and V8 are open source, which makes the code just as auditable and modifiable as the 3.6 million lines of Rust. Which is to say, both are equally unapproachable.
    • csomar 1y ago
      > The libraries are a different story—you're likely to have things break under you that rely on older versions of libraries when you update—but I don't see Rust actually having solved that. No language can fix that. However, I've lost count of the times my Python/JavaScript interpretation fails because of something in one of the dependencies. Usually, it's not a JS/Python problem but rather has to do with a Node/Python version update. It always boils down to the "core" issue which is the runtime. That's why I like that Rust give me a "fixed" runtime that I download/compile/package with my program. > Node and V8 are open source, which makes the code just as auditable and modifiable as the 3.6 million lines of Rust. Which is to say, both are equally unapproachable. I've recently patched a weird bug under Tokio/Otel and can't imagine doing that with Node/V8 without it being a major hassle. It is relatively straightforward in Rust though requires maintaining your own fork of only the dependency/branch in question.
• jerf 1y ago
  A true enough statement, but "Rust" is unnecessarily specific. Dependencies are getting scary in general. Supply chain attacks are no longer hypothetical, they're here and have been for a while. If I were designing a new language I think I'd be very interested in putting some sort of capability system in so I can confine entire library trees safely, and libraries can volunteer somehow what capabilities they need/offer. I think it would need to be a new language if for no other reason than ecosystems will need to be written with the concept in them from the beginning. For instance, consider an "image loading library". In most modern languages such libraries almost invariably support loading images from a file, directly, for convenience if nothing else. In a language that supported this concept of capabilities it would be necessary to support loading them from a stream, so either the image library would need you to supply it a stream unconditionally, or if the capability support is more rich, you could say "I don't want you to be able to load files" in your manifest or something and the compiler would block the "LoadFromFile(filename)" function at compile time. Multiply that out over an entire ecosystem and I think this would be hard to retrofit. It's hugely backwards incompatible if it is done correctly, it would be a de facto fork of the entire ecosystem. I honestly don't see any other solution to this in the long term, except to create a world where the vast majority of libraries become untargetable in supply chain attacks because they can't open sockets or read files and are thus useless to attackers, and we can reduce our attack surface to just the libraries that truly need the deep access. And I think if a language came out with this design, you'd be surprised at how few things need the dangerous permissions. Even a culture of minimizing dependencies is just delaying the inevitable. We've been seeing Go packages getting supply-chain-attacked and it getting into people's real code bases, and that community is about as hostile to large dependency trees as any can be and still function. It's not good enough.
  • assassinator42 1y ago
    Java and the .NET Framework had partial trust/capabilities mechanisms decades ago. No one really used them and they were deprecated/removed.
    • pjmlp 1y ago
      It was more like no one used them correctly.
• srikanth767 1y ago
  True
• aliceryhl 1y ago
  I'm quite careful to tightly control the dependencies of Tokio. All dependencies are under control by members of the Tokio team or others that I trust.
• schmichael 1y ago
  We need a term like “Mature” or similar for dependencies that are done. Mature dependencies have two characteristics: 1. Well defined scope 2. Infrequent changes Nomad has many of these (msgpack, envparse, cli, etc). These dependencies go years without changing so the dependency management burden rapidly approaches zero. This is an especially useful property for “leaf” dependencies with no dependencies of their own. I wish libraries could advertise their intent to be Mature. I’d choose a Mature protobuf library over one that constantly tweaked its ergonomics and performance. Continual iterative improvement is often a boon, but sometimes it’s not worth the cost.
  • delusional 1y ago
    I have a lot of sympathy for this viewpoint, but I also ask that we try to remind ourselves. We are asking for professionalism from hobby projects. If you want a mature protobuf implementation you should probably buy one. Expecting some guy/gal on the internet to maintain one for your for free seems ill advised.
    • schmichael 1y ago
      A great point! All of the libraries I mentioned are created and maintained by corporations. Hobbyists, as always, are free to do as they please without judgement from me. :) I will say I get great satisfaction from the little envparse library I wrote needing near-0 maintenance. It’s a rare treat to be able to consider any project truly done.
    • procaryote 1y ago
      Isn't that an argument _for_ having a "mature" label? To avoid the hobbyists who have no intention to maintain their thing? Also there are lots of lovely projects maintained at high levels by hobbyists, and plenty of abandonware that was at some point paid for
      • deleted 1y ago
        [deleted]
      • 1y ago
• nemothekid 1y ago
  I feel like leftpad has given package managers a very bad name. I understand the OP's hesitation, but it feels a little ridiculous to me. tokio is a work-stealing, asynchronous runtime. This is a feature that would be an entire language. Does OP consider it reasonable to audit the entire Go language? or the V8 engine for Node? v8 is ~10x more lines than tokio. If Cloudflare uses Node, would you expect Cloudflare to audit v8 quarterly?
  • timewizard 1y ago
    If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default? This is something I've only ever seen cargo do.
    • metaltyphoon 1y ago
      > If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default? No, cargo will resolve using sem ver compatibility and pick the best version. Nuget, for C# does something very similar.
    • rcxdude 1y ago
      It'll do that if there isn't a single version that meets both requirements. Which is a great thing, because most other languages will just fail the build in that case (well, there are still cases where it won't even work in rust, if types from those sub-dependencies are passed in between the two closer dependencies)
    • pas 1y ago
      > This is something I've only ever seen cargo do. npm does this (which causes [caused?] the node_modules directory to have a megazillion of files usually, but sometimes "hoisting" common dependencies helps, and there's Yarn's PnP [which hooks into Node's require() and keeps packages as ZIPs], and pnpm uses symlinks/hardlinks)
  • conradludgate 1y ago
    And for what it's worth, people do audit tokio. I have audited tokio. Many times in fact. Sure, not everyone will, but someone will :)
• deleted 1y ago
  [deleted]
• righthand 1y ago
  Everyone is in such a rush to get their project out the door, no one has time to generate a key and properly code sign releases and begin developing a more secure chain. Now we have JS package "whatever code" ecosystem but for Rust. As if we haven't watched NPM get hacked many times over the last decade or so.
  • mcflubbins 1y ago
    > Everyone is in such a rush to get their project out the door This is the cause of so many issues. And its not like we're at war or trying to cure the next pandemic, we're writing CRUD apps and trying to convince people to click on adds for crap they don't need.
  • arp242 1y ago
    > As if we haven't watched NPM get hacked many times over the last decade or so. When has this happened? The only one I remember is the event-stream thing, and that was what, over five years ago? Doesn't seem all that common from what I can see?
• gxt 1y ago
  You can audit your dependencies for crates with security vulnerabilities reported to the RustSec Advisory Database, also block unmaintained crates, and enforce your license requirements using SPDX expressions with cargo-audit and cargo-deny. You can ensure that third-party Rust dependencies have been audited by a trusted entity with cargo-vet. And you should have taken a look at where those 3M locs come from, it's usually from Microsoft's windows-rs crates that are transitively included in your dependencies through default features and build targets of crates built to run on windows.
• 1vuio0pswjnm7 1y ago
  "Not thinking about package management careful makes me sloppy." Isn't the point of a memory safe language to allow programmers to be sloppy without repercussions, i.e., to not think about managing memory and even to not understand how memory works. Would managing dependencies be any different. Does Rust allow programmers to avoid thinking carefully about selecting dependencies.
  • sophacles 1y ago
    > Isn't the point of a memory safe language to allow programmers to be sloppy without repercussions, i.e., to not think about managing memory and even to not understand how memory works No. The point is even the best programmers of unsafe languages regularly introduce both simple and subtle bugs into codebases while being careful about handling memory correctly, and therefore we should use languages that don't even allow those bugs for most every use case. Using these languages still allows crap programmers to waste GBs of correctly allocated and handled memory, and good programmers to write tight, resouce-sipping code. Dependencies are orthogonal to this.
  • 1vuio0pswjnm7 1y ago
    If careful programmers who can manage memory should use the same language as careless ones who cannot, then does this mean both should also automatically use third party libraries by default. Are there systems languages that provide memory management but do not default to using third party libraries. If yes, then do these languages make it easier for programmers to avoid dependencies.
  • empath75 1y ago
    No, the point is to stop you from being sloppy. The code won't compile if you're sloppy with memory management. You can be _relatively_ sure that you're not introducing memory unsafety by adding a dependency, but you can't be sure that it isn't malware unless you audit it.
  • timewizard 1y ago
    > to be sloppy without repercussions It's the difference between a wet mess and a dry one. Rust creates dry messes. It's still a mess.
• rs186 1y ago
  I once wanted to contribute to the popular swc project (https://github.com/swc-project/swc https://github.com/swc-project/swc). I cloned the repo, ran build, and a whooping 20GB was gone from my disk. The parser itself (https://github.com/swc-project/swc/blob/main/crates/swc_ecma_parser/Cargo.toml https://github.com/swc-project/swc/blob/main/crates/swc_ecma...) has over a dozen dependencies, including serde. Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight. I decided that I should leave this project alone and spend my time elsewhere.
  • MeetingsBrowser 1y ago
    I agree that relying on unknown dependencies is a risk, but this misses the point IMO. Number of dependencies and disk space are kind of arbitrary. > Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight. The lightest weight javascript program relies on V8 to run, which has multiple orders of magnitude more dependencies. Most of which you have never heard of. At least cargo makes it easier to get a clearer picture of what the dependencies are for a program.
    • munificent 1y ago
      > relies on V8 to run, which has multiple orders of magnitude more dependencies. Actually, this isn't true. (Or at least wasn't a while back.) I used to work with a bunch of ex-V8 folks and they really despised third-party dependencies and didn't trust any code they didn't write. They used a few third-party libs but for them most part, they tried to own everything themselves.
      • dathinab 1y ago
        they are also Google .. as in they can afford to rewrite everything .. can afford to suffer from not invented here syndrome .. and are under _massive_ threat of people doing supply chain attacks compared to most other projects (as they end up running on nearly any desktop computer and half the phones out there) this just isn't viable for most projects, not just resource/time investment wise, but also reinventing/writing everything isn't exactly good to reduce bugs if you haven't to reliably access to both resources _and_ expertise. Most companies have to live with having many very average developers, and very tight resource limits.
• Orangeair 1y ago
  I think that https://blessed.rs https://blessed.rs does a pretty good job of providing recommendations for things that probably can't be crammed into the standard library, but which you'll almost certainly end up needing at one point or another. I honestly like that system a lot, it makes it so that the only packages you need to worry much about are usually doing something rather specific.
  • the__alchemist 1y ago
    Did a review; this is solid!
  • zahlman 1y ago
    Would love to see something like this for Python.
  • bjackman 1y ago
    Also shout out to cargo-vet. It lets you track what packages you "trust". Then you can choose to transitively trust the packages trusted by entities you trust. This lets you have a policy like "importing a new 3rd party package requires a signoff from our dependency tzar. But, packages that Google claim to have carefully reviewed are fine". You can also export varying definitions of "trust". E.g. Google exports statements like: - "this package has unsafe code, one of our unsafe experts audited it and thinks it looks OK" - "this package doesn't do any crypto" - "this is a crypto library, one of our crypto experts audited it and thinks it looks ok" https://github.com/google/rust-crate-audits/blob/main/auditing_standards.md#does-not-implement-crypto https://github.com/google/rust-crate-audits/blob/main/auditi... Basically it's a slightly more formal and detailed version of blessed.rs where you can easily identify all the "it's not stdlib, but, it's kinda stdlib" stuff and make it easily available to your team without going full YOLO mode. It can also give you a "semi-YOLO" approach, it supports rationales like "this package is owned by a tokio maintainer, those folks know what they're doing, it's probably fine". I think this is a nice balance for personal projects.
• kion 1y ago
  IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painstakingly included parts of into your size constrained environment (fit it on a floppy). You probably picked apart that library and pulled the bits you needed, integrating them into your builds to be as small as possible. Today we pile libraries on top of libraries on top of libraries. Its super easy to say `import foolib`, then call `foolib.do_thing()` and just start running. Who knows or cares what all 'foolib' contains. At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number. In some cases the languages make this worse. Go and Rust, for example, encourage everything for a single package/mod to go in the same file. Adding optional functionality can get ugly when it would require creating new modules, but if you only want to use a tiny part of the module, what do you do? The only real solution I can think of to deal with this long term is ultra-fine-grained symbols and dependencies. Every function, type, and other top-level language construct needs to declare the set of things it needs to run (other functions, symbols, types, etc). When you depend on that one symbol it can construct, on demand, the exact graph of symbols it needs and dump the rest for any given library. You end up with the minimal set of code for the functionality you need. Its a terrible idea and I'd hate it, but how else do you address the current setup of effectively building the whole universe of code branching from your dependencies and then dragging it around like a boat anchor of dead code.
  • nicoburns 1y ago
    As far as I'm aware, LTO completely solves this from a binary size perspective. It will optimise out anything unused. You can still get hit from a build time perspective though.
    • kion 1y ago
• conradludgate 1y ago
  As a fellow rust developer, I love our dependencies but I put a lot of effort into pruning the ones I want to use. If I see a crate using too many I might contribute to it or find a replacement. If you want to use dependencies, I wouldn't be surprised when you realise they also want to use dependencies. But you can put your money/time in the right places. Invest in the dependencies that do things well.
• zaptheimpaler 1y ago
  This is just a modern problem in all software development, regardless of language. We are doing more complex things, we have a much bigger library of existing code to draw from and there are many reasons to use it. Ultimately a dependency is untrusted code, and there's a long road to go in hardening entire systems to make running arbitrary dependencies safe (if its even possible). In the absence of a technical solution, all others basically involve someone else having to audit and constantly maintain all that code and social/legal systems of trust. If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult.
  • harha_ 1y ago
    Regardless of language, really? I highly doubt that, you don't generally see such problems with C or even C++ because dependencies are more cumbersome to add, especially in a way that's cross-platform.
    • zaptheimpaler 1y ago
      Because most dependencies are either manually installed by the user, or are dynamic libraries that are provided and audited by the distro maintainers. The dependencies are there, they're just harder to see - https://wiki.alopex.li/LetsBeRealAboutDependencies https://wiki.alopex.li/LetsBeRealAboutDependencies
      • harha_ 1y ago
        Sure, there are various dependencies, but it's nothing like "cargo install crate-name". Cargo makes it so effortless to joink the dumbest dependency for the simplest thing.
        • zaptheimpaler 1y ago
          Sure, I think software that's easy to use is a good thing and Rust dependency management is 100x nicer to work with than C++.
• QuadmasterXLII 1y ago
  I wonder how much good a “dependency depth” label on packages would do, at the crates.io level. Like, a package can only depend on a package with a lower declared dependency depth than it, and packages compete to have a low dependency depth as a badge.
  • themk 1y ago
    Everytime I daydream about my perfect language, this is one of the features I think about.
• amelius 1y ago
  I think the main problem is that you should be able to run dependencies inside their own sandbox, and the language focuses only on memory safety within a monolithic program.
  • dathinab 1y ago
    the problem is if you put library dependencies in their own sandbox you have a different kind of interface (much more limited) for libraries like e.g. if we look at sandbox boundaries we have: - some in language permission enforcement (e.g. Java Security Manage) -- this approach turned out to be a very bad idea - process boundaries, i.e. take the boundary the OS enforces and lock it down more (e.g. by stuff like pledge, cgroups etc.) -- this approach turned out okayish - VM boundaries (e.g. firecracker VMs) -- tourned out well - emulation boundaries (e.g. WASM) -- mixed history, can turn out well especially if combined with worker processes which lock themself down but what that means in practice is that wanting the reliably sand box library dependencies will most likely lead to more or less IPC boundaries between the caller and the libary what that means is practice it's unsuited for a lot of thing e.g. for most utility lib it's very unsuited e.g. for a lot (but not all) data structure libs its unsuited and might be a huge issue e.g. you can apply it to a web-server, but then you are basically reinventing CGI, AGI which okay but can quite compete with perf. e.g. but you can't apply it to some fundamental runtime engine (e.g. tokio), worse you now might have one copy of the engine running per sandbox... (but you can apply it to some sub-part of tokio internals) People have tried this a lot in various ways. But so far this always died off in the long run. Would be nice if the latest push based around WASM would have some long term success.
    • ATechGuy 1y ago
      Thanks! This is a very detailed explanation of why existing sandboxing techniques will not work as expected for dependencies (wrt to functionality or performance).
    • amelius 1y ago
      > the problem is if you put library dependencies in their own sandbox you have a different kind of interface (much more limited) for libraries Nobody said it would be easy. As an analogy, the borrow checker makes working with memory much more limited, yet some people like it because it makes things safer.
• colanderman 1y ago
  Rust at least has a partial remedy to this problem: feature flags. Many libraries use them to gate features which would otherwise pull in extra dependencies. (In fact I believe there is specific support for flags which correspond to dependency names.)
• philsnow 1y ago
  > do I even need this crate at all? 35 lines later I had the parts of dotenv I needed. I'm not saying you copy-pasted those 35 lines from dotenvy, but for the sake of argument let's say you did: now you can't automatically benefit from dotenvy patching some security issue in those lines.
  • a2128 1y ago
    To benefit you have to actually trust the current and future maintainers of the package, its dependencies, the dependencies of its dependencies, etc. You can also automatically get breached in a supply chain attack, so it's a tradeoff
  • skydhash 1y ago
    If you REALLY need such update, you can easily subscribe to updates from the mainstream project (in whatever way it allows) and patch your version when that rare situation occurs.
  • bsrkf 1y ago
    Can't benefit from them patching a security issue, but don't suffer from - them breaking something - a supply chain attack - them making a change which breaks your program - you having accidentally relied on a bug or an unintended behavior of their code (which they may fix at any moment) - many unneeded LOC in your codebase - absolution of ownership - relying on a dependency versus having written it yourself - in the latter case you'll automatically take responsibility - think much more about code's security/quality - have the knowledge to fix it and know exactly where to (in your 35-lines of code you yourself wrote) - more burdensome upgrades of your software - longer compilation speeds - having to monitor their program - is it abandoned, ownership transferred to dubious party - did the maintainer have a late night drunken stupor accepting bad pull requests - did they react to a CVE or not - did they change the license - do they have a license but added their own problematic paragraph - does the program "develop badly" (change its target scope in any problematic way) (take on more and more bloat, more unneeded functionality) - having worse of an overview of your total dependencies (since they may themselves rely on further crates you don't expect) - ... what's the trade-off now?
• vsgherzi 1y ago
  author here, thanks for reading and all of your thoughts! Here's another older HN thread with some interesting comments. https://news.ycombinator.com/item?id=43930640 https://news.ycombinator.com/item?id=43930640
• James_K 1y ago
  When I am compiling Rust applications, I must admit I'm always rather bemused at the number of dependencies pulled. Even what I'd have thought to be simple tools reach easily about 200 dependent packages. It's nightmarish. One way this becomes particularly apparent is if you're trying to create a reproducible package for Guix or Nix. You end up having to manually specify a package for every different Rust library because of how those system require reproducible builds. The process of writing Guix package for software has been extremely illuminating for me, as to just how deeply nested certain technologies are vs. others. I'd be willing to bet it's a good metric for what sticks around. If you've got 200 dependencies, I don't think your software is gonna last the test of time. It seems a recipe for endless churn.
• the__alchemist 1y ago
  To address a point near the end of the article, here is my [partial] solution that works as a baseline. Curate a collection of libraries you use and trust. This will probably involve making a number of your own. Wheel-reinvention, if you will. If done properly, even the upfront time cost will save in the long-run. I am in the minority here, but I roll my own libs whenever possible, and the 3rd party libs I use are often ones I know, have used been for, and vetted that they have a shallow tree of their own. Is this sustainable? I don't know. But It's the best I've come up with, in order to use what I see as the best programming language available for several domains. There are a lot of light-weight, excellent libs I will use without hesitation, and have wide suitability. Examples: - num_enum - num_traits - bytemuck - chrono - rand - regex - bincode - rayon - cudarc Heavier, and periodically experience mutual-version hell, but are are very useful for GUI programs: - EGUI - WGPU - Winit On a darker note, the rust web ecosystem maybe permanently lost to async and messy dependencies. Embedded is going that way too, but I have more hope there, and am doing my best to have my own tooling.
  • sitkack 1y ago
    Rust really made some unfortunate choices with async, it pollutes everything but isn't generic enough so now you are married to the runtime, this bifurcates the whole ecosystem. It is nearly phobos/demios problem from Dlang, but instead Tokio just took over. One doesn't use Rust anymore, they use Tokio.
    • infamouscow 1y ago
      Rust will thrive despite the PLT coloring debate. Async frameworks often dominate through winner-takes-all dynamics. Most blog posts on async coloring are pretentious nonsense, and I've faced heavy moderation here for calling out their intellectual bankruptcy. The completely brain dead moralizing arguments from the ignorant deserve intense derision regardless of what HN's official rules are. Real world software ecosystems evolve slowly, requiring years of debate to shift. - From HN's most outspoken Rust critic
      • 1y ago
• bsrkf 1y ago
  Excuse me for not having much to add to the discussion but two interesting references for people to check out, if so inclined of course: a) Ginger Bill (the Odin language creator, no affiliation) stated on a podcast that Odin will never have an official pkg manager, since what they're, in his opinion, mainly automating is dependency hell, and this being one of the main reasons for rising software complexity and lower software quality; see https://www.youtube.com/watch?v=fYUruq352yE&t=11m26s https://www.youtube.com/watch?v=fYUruq352yE&t=11m26s (timestamped to the correct position) (they mention Rust explicitly as an example) b) another programmer rather seriously worried about software quality/complexity is Jonathan Blow, who's talk "Preventing the Collapse of Civilization" is worth watching in my opinion: https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko (it's not talking about package managers specifically, but is on topic regarding software complexity/quality as a whole) Addendum: And sorry, I feel like almost everyone knows this xkcd by now, but since no one so far seems to have posted it; "obligatory xkcd reference": https://imgs.xkcd.com/comics/dependency_2x.png https://imgs.xkcd.com/comics/dependency_2x.png
  • mustache_kimono 1y ago
    > a)... Odin will never have an official pkg manager Perhaps this explains why Odin has found such widespread usage and popularity. /s > b)... Jonathan Blow, who's talk "Preventing the Collapse of Civilization" With such a grandiose title, before I first watched I thought it must be satire. Turns out, it is food for the credulous. I believe Jonathan Blow is less "seriously worried about software quality/complexity" than he is about marketing himself as the "last great hope". At least Blow's software has found success within its domain. However, I fear Blow's problem is the problem of all intellectuals: “An intellectual is a person knowledgeable in one field who speaks out only in others.” Blow has plenty of opinions about software outside his domain, but IMHO very little curiosity about why his domain may be different than your own. My own opinion is there is little evidence to show this is a software quality problem, and any assertion that is the case needs to compare the Rust model against the putatively "better" alternatives. Complex software, which requires many people to create, sometimes across great distances of time and space, will necessarily have and require dependencies. Can someone show me a material quality difference between ffmpeg, VLC, and Samba dependencies and any sufficiently complex Rust program (even which perhaps has many more dependencies)? ~ ldd `which ffmpeg` | wc -l 231 Now, large software dependency graphs may very well be a security problem, but it is a problem widely shared with all other software.
• vzaliva 1y ago
  They should take a look at OPAM (OCaml’s package manager). There was a really impressive talk at the OCaml Workshop at POPL or ICFP a couple of years ago about how it works. Basically, they have a huge CI infrastructure and keep all versions of every package ever published. So, once you’ve found the right set of dependencies for your project, you can be sure the exact versions will always be available via OPAM.
• socalgal2 1y ago
  I'm curious if rust has this problem. The problem I notice in npm land is many developers have no taste. Example, there's a library for globbing call glob. You'd think it would just be a function that does globbing but no, the author decided it should ALSO be a standalone commandline executable and so includes a large commandline option parser. They could have easily made a separate commandline tool that include a library that does the glob but no, this is a common and shit pattern in npm. I'd say easily 25% or more of all "your dependencies are out of date" messages are related to the argument parcing for the commandline tool in these libraries. That's just one example. Also there's arguably design. Should a 'glob' library actually read the file system and give you filenames or should it just tell you if a string matches a glob and leave the reset to you? I think it's better design to do the later, the simplest thing. This means less dependencies and more flexibility. I don't have to hack it or add option to use my own file system (like for testing). I can use it with a change monitoring system, etc... And, I'm sure there are tons of devs that like the glob is a "Do everything for me" library instead of a "do one specific thing" library which makes it worse because you get more "internet points" the more your library doesn't require the person using it to be a good dev. I can't imagine it's any different in rust land, except maybe for the executable thing. There's just too many devs and all of them, including myself, don't always make the best choices.
  • kazinator 1y ago
    > Should a 'glob' library actually read the file system and give you filenames The POSIX glob function after which these things are named traverses the filesystem and matches directory entries. The pure matching function which matches a glob pattern against a filename-like string is fnmatch. But yes, the equivalent of fnmatch should be a separate module and that could be a dependency of glob. Nobody should be trying to implement glob from scratch using a fnmatch-like function and directory traversal. It is not so trivial. glob performs a traversal that is guided by the pattern. It has to break the pattern into path components. It knows that "*/*/*" has three components and so the traversal will only go three levels deep. Also "dir/*" has a component which is a fixed match, and so it just has to open "dir" without scanning the current directory; if that fails, glob has failed. If the double star ** is supported which matches multiple components, that's also best if it likewise integrated into glob. If brace expansion is supported, that adds another difficulty because different branches of a brace can have different numbers of components, like {*/x,*/*/x,*/*/*/x}. To implement glob, it would greatly help us to have brace expansion as a separate function which expands the braces, producing multiple glob patterns, which we can then break into path components and traverse.
• mleonhard 1y ago
  > I can't rewrite the world, an async runtime and web server are just too difficult and take to long for me to justify writing for a project like this (although I should eventually just for a better understanding). I did this and it only solved half of the bloat: https://crates.io/crates/safina https://crates.io/crates/safina - Safe async runtime, 6k lines https://crates.io/crates/servlin https://crates.io/crates/servlin - Modular HTTP server library, threaded handlers and async performance, 8k lines. I use safina+servlin and 1,000 lines of Rust to run https://www.applin.dev https://www.applin.dev, on a cheap VM. It serves some static files, a simple form, receives Stripe webooks, and talks to Postgres and Postmark. It depends on some heavy crate trees: async-fs, async-net, chrono, diesel, rand (libc), serde_json, ureq, and url. 2,088,283 lines of Rust are downloaded by `cargo vendor` run in the project dir. 986,513 lines using https://github.com/coreos/cargo-vendor-filterer https://github.com/coreos/cargo-vendor-filterer to try to download only Linux deps with `cargo vendor-filterer --platform=x86_64-unknown-linux-gnu`. This still downloads the `winapi` crate and other Windows crates, but they contain only 22k lines. 976,338 lines omitting development dependencies with `cargo vendor-filterer --platform=x86_64-unknown-linux-gnu --keep-dep-kinds=normal`. 754,368 lines excluding tests with `cargo vendor-filterer --platform=aarch64-apple-darwin --exclude-crate-path='*#tests' deps.filtered`. 750k lines is a lot to support a 1k-line project. I guess I could remove the heavy deps with another 200 hours of work, and might end up with some lean crates. I've been waiting for someone to write a good threaded Rust Postgres client.
  • bsaul 1y ago
    I've come to accept that i wasn't really developing in "rust", but in "tokio-rust", and stopped worrying about async everywhere (it's not fundamentally different from what happens with other lang having async). Why the need for going back to threaded development ?
    • mleonhard 1y ago
      1. Async Rust is extra stuff for engineers to learn and maintain in their heads. 2. Async Rust has a lot of papercuts. 3. Very little code actually needs async. For example, in an API server, every request handler will need a database connection so the concurrency is limited by the database. I wrote the Servlin HTTP server in async rust, to handle slow clients, but it calls threaded request handlers.
• wpollock 1y ago
  All the comments and suggestions for improving rust dependency handling seem useful to me. To deal with dependency sprawl now, until the situation changes, I use a number of tools. To avoid having to set this up for each new project, I've made a template project that I simply unzip to create new rust projects. The tools I have found useful are: cargo outdated # check for newer versions of deps cargo deny check # check dependency licenses cargo about # generate list of used licenses cargo audit # check dependencies for known security issues cargo geiger # check deps for unsafe rust I haven't found a cargo tool I like for generating SBOMs, so I installed syft and run that. cargo install-update # keep these tools updated cargo mutants # not related to deps, but worth a mention, used when testing. Having configured all these tools once and simply unzipping a template works well for me. Suggestions for different or additional tools welcome! Disclaimer: I'm not a professional rust developer.
• vladkens 1y ago
  I agree that there are too many dependencies in Rust. I support the idea of adding some of the more popular crates to std. Many applications use something like tracing, tracing-subscriber, and basic server/client functionality. It would be great to have simple, minimal-feature implementations of these in std — similar to how Go does it. If someone needs a more complex system, they can still use an external crate, but having basic building blocks in std would really help.
• deleted 1y ago
  [deleted]
• sitta 1y ago
  I recently wrote an extremely basic Rust web service using Axum. It had 10 direct dependencies for a total of 121 resolved dependencies. I later rewrote the service in Java using Jetty. It had 3 direct dependencies for a total of 7 resolved dependencies. Absolutely nuts.
  • panstromek 1y ago
    I don't think number of dependencies is a useful comparison metric here. Java runtime already implements stuff that you have to use libraries for in Rust, and it's a design choice. Rust also has slimmer std. Both languages have different constraints for this.
• rvz 1y ago
  > What's the solution? A proper 'batteries included' standard library in the language and discouraging using too many libraries in a project. The same mistakes from the Javascript community are being repeated in front of us for Cargo (and any other project that uses too many libraries).
• hedora 1y ago
  What about cargo vet? It lets security professionals audit rust packages, and cryptographically attest to their trustworthiness.
• Charon77 1y ago
  One of the good thing in cargo packages are the feature flags. If a repo uses too much dependencies then it's time to open an issue or PR to hide them behind feature flags. I do that a lot with packages that requires std even though it could do with core and alloc. cargo tree helps a lot on viewing dependency tree. I forgot if it does LoC count or not.. > to see what lines ACTUALLY get compiled into the final binary, This doesn't really make much sense as a lot of the functions that make it to the binary get inlined so much that it often becomes part of 'main' function
  • ramon156 1y ago
    100%, I still miss feature flags in npm. Is there a package manager that can do this already? I'd love to expand our internal libs with framework-specific code
• lmm 1y ago
  A thought experiment for this writer: imagine if Tokio (and all its dependencies) were moved into the Rust standard library, so that it was more like Go. Would that make them more comfortable depending on it (not that they'd have a choice any more)? If so, why?
• wyldfire 1y ago
  > Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust. Removing the vendored packages reduces this to 11136 lines of rust. Out of those 3.6 million lines, how many are lines of test code?
• hedora 1y ago
  Has anyone had good luck with cargo vet? It lets security professionals cryptographically vouch for the trustworthiness of rust packages.
• fucker42069 1y ago
  [dead]
• NewJazz 1y ago
  Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust. Removing the vendored packages reduces this to 11136 lines of rust. Tokei hasn't had a stable release in over 4 years and misreports lines of code in some instances. The author in the past has basically said they would need to be paid to backport one line fixes with no merge conflicts that fix real accuracy issues in their software... Bad look in my book. https://github.com/XAMPPRocky/tokei/issues/875 https://github.com/XAMPPRocky/tokei/issues/875
• endorphine 1y ago
  What actually surprised me in Rust, is the amount of fragmentation and abandoned libraries. For example, serde_yaml is archived and there are two other libraries that do the same (?) thing. It seems like there's a significant effort required to search for and decide which (if at all) library to use. This is not so much pronounced in Go.
  • jlarocco 1y ago
    Maybe that's the double edged sword of making the package manager so integrated into the language.
    • jcgl 1y ago
      How is cargo more integrated into the language than Go’s? I’ve little to no experience with Rust, but Go’s package management seems pretty fully integrated to me.
  • simonask 1y ago
    Yeah, one problem in Rust is that a number of very fundamental ecosystem libraries are written by a handful of high-profile people. Often people who are also working on the standard library or the Rust compiler. Rust developers usually know their names and SoMe handles. It's a problem because those people become overworked, and eventually have to abandon things. The deprecation of `serde_yaml` was and is a huge, huge problem, especially without any functional replacement. There was no call for new maintainers, or for someone to take over the project. I can understand the reasons why (now you're suddenly auditing people, not code), but it sucks.
• arp242 1y ago
  3.6M lines of code seems so much that it sets off my "are you sure that's counted right?" alarm. I'm not very familiar with Rust, but all of Go is 1.6M lines of Go code. This includes the compiler, stdlib, tests for it all: the lot. Not that I doubt the sincerity of the author of course, but maybe some irrelevant things are counted? Or things are counted more than once? Or the download tool does the wrong thing? Or there's tons of generated code (syscalls?)? Or ... something? I just find it hard to believe that some dependencies for web stuff in Rust is twice all of Go.
  • hu3 1y ago
    This other person wrote their own async runtime and web server from scratch to reduce bloat, and their rust app still vendors 2 million lines of code: https://news.ycombinator.com/item?id=43942055 https://news.ycombinator.com/item?id=43942055
    • arp242 1y ago
      Sounds like a lot is just (generated) syscall stuff? Either way, that doesn't really explain anything.
      • hu3 1y ago
        That's one part of Rust bloat and slow compilation times. But there's still a ton of dependency code there.
• Avi-D-coder 1y ago
  The solution is strong compile time and runtime guarantees about code behavior. The author is right there's no way an individual can audit all that code. Currently all that code can run arbitrary build code at compile time on the devs machine, it can also run arbitrary unsafe code at runtime, make system calls, etc.. Software is not getting simpler, the abundance of high quality libraries is great for Rust, but there are bound to be supply chain attacks. AI and cooperative auditing can help, but ultimately the compiler must provide more guarantees. A future addition of Rust should come with an inescapable effect system. Work on effects in Rust has already started, I am not sure if security is a goal, but it needs to be.
  • deleted 1y ago
    [deleted]
• nyuriks 1y ago
  I see two points: safety - bigger supply chain attack surface, and code bloat/compiler performance. The later has been discussed in numerous posts here (the whole idea of a linker from the start was to get rid of unused functions, so not a big problem imo). The safety is a serious and legit consideration, but we also rely on Linux and build tools to build things. How do you know the compiler that was used to build Linux hasn't been compromised, perhaps several generations ago, and now your Linux has a backdoor that is not in Linux source code? There was a research paper on this IIRC. We trust the ecosystem to validate each tool we use. We just have to do the same with our own projects - only use what's relevant, and we should do dependency hygiene to check if it is coming from a reputable source...
• cryptonector 1y ago
  > What's the solution? Big things you use off-the-shelf libraries for. Small things you open-code, possibly by cribbing from suitably-licensed open source libraries. You bloat your code to some degree, but reduce your need to audit external code and reduce your exposure to supply chain attacks. Still, the big libraries are a problem, but you're not going to open code everything. This isn't just Rust. It's everything.
  • cryptonector 1y ago
    > Big things you use off-the-shelf libraries for. I should have added: "and for obvious reasons". Didn't computer science hype up code reuse for decades before it finally started happening on a massive scale? For that to actually happen we needed programming languages with nice namespaces and packaging and distribution channels. C was never going to have the library ecosystem that Java, C++, and Rust have. Now that we're there suddenly we have a very worrisome supply chain issue, with major Reflections on Trusting Trust vibes. What to do? We can't all afford to open-code everything, so we won't, but I recommend that we open-code all the _small_ things, especially in big projects and big libraries. Well, or maybe the AI revolution will save us.
• Alex_001 1y ago
  [dead]
• kirici 1y ago
  > This whole fiasco led me tho think .... do I even need this crate at all? 35 lines later I had the parts of dotenv I needed. "A little copying is better than a little dependency." - grab the parts that you need and then include the library only in a test to ensure alignment down the line, an idea I liked a lot. https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s
• CraigJPerry 1y ago
  Vendoring is a step in the right direction, you’ve constrained one side of the equation. But you’re still open to typo squatting and similar issues like crates falling unmaintained - the article mentions the now famous dotenv vs. dotenvy issue (is this solvable with a more mature governance model for the crates ecosystem? At this point dotenv should probably be reclaimed). So after vendoring a baseline set of dependencies, you need to perform comprehensive auditing. Maybe you can leverage LLMs to make that blob of vendored deps smaller / cheaper to own. Maybe you can distill out only the functionality you need (but at what cost, now you might struggle to backport fixes published upstream). Maybe LLMs can help with the auditing process itself. You need a stream of notifications of upstream fixes to those vendored deps. Unfortunately in the real world the decision making will be harder than “ooh, there’s a sec fix, I should apply that”. I always wonder why someone like JFrog don’t expand their offering to provide “trusted dependencies” or something similar. I.e. you pay to outsource that dependency governance and auditing. Xray scanning in the current product is a baby step toward the comprehensiveness I’m suggesting. Taking a step back though, I’d be really careful not to throw the baby out with the bath water here. Rust has a fairly unique capability to compose work product from across unrelated developers thanks to its type system implementation (think about what happens with a C library, who’s responsible for freeing the memory, you or me?). Composition at scale is rusts super power, at least in terms of the productivity equation for large enterprises - in this context memory safety is not the sales pitch since they already have Java or whatever.
• swoorup 1y ago
  This feels like a pointless issue. At best perhaps establish a good trust model, than blaming the tooling.
• ak_111 1y ago
  Dependency and build management is a fascinating and still unsolved problem in software engineering (in some sense it is the central problem). I am wondering if there is a good modern reference that provides a conceptual overview or comparative study of the various techniques that have been attempted. It is a hard subject to define as it cuts through several layers of the stack (all the way down to the compiler system interface layer), and most book focus on one language or build technology rather than providing a more conceptual treatment of the techniques used.
• viktorcode 1y ago
  A large removable standard library seems to be the optimal solution. It is there by default for everyone, but if needed for embedded scenario it can be removed, leaving only the core language features.
• weltensturm 1y ago
  Rust has a million ways to solve a specific problem, as it is not opinionated and gets you down to the lowest level if needed. On top of that there's a million ways to encode your types. Then there's a million ways to bind C libraries. The solution space is basically infinite, and that's a good thing for a systems programming language. It's kind of amazing how far rust reaches into higher level stuff, and I think the way too easy to use package manager and lively crate ecosystem is a big part of that. Sometimes I wish for a higher-level rust-like language though, opinionated as hell with garbage collector, generic functions without having to specify traits, and D's introspection.
  • jstanley 1y ago
    Have you tried Go?
• anothernewdude 1y ago
  Python is in the same spot now you can't easily install packages globally. I don't have the hard drive space to develop multiple projects anymore. Every single project takes up multiple gigs in just dependencies.
• morsecodist 1y ago
  I see a lot of concern like this about dependencies, mostly in node. I'm sure it's an issue I'm just not convinced it's as big of a problem as people say. We have scanners that can help keep your dependencies secure automatically. If you take a dependency and it goes unmaintained is it really that much worse than the relevant code in your own codebase going unmaintained?
• bradley13 1y ago
  This is a general problem: devs pulling in libraries instead of writing a few lines of code. Those libraries pull in more dependencies that have even more dependencies. There's no good solution...
  • nine_k 1y ago
    Why, offer a patch that introduces the few lines and removes a dependency.
  • tlb 1y ago
    LLM coding assistants are a partial solution. Recently I typed vec4 rgb2hsv(vec4 rbg) and a few tab-completes later it had filled in the body of the code with a correct color conversion routine. So that saved me searching for and pulling in some big-ass color library. Most of lodash.js can be avoided with LLMs too. Lodash's loops are easier to remember than Javascript's syntax, but if your LLM just writes the foo.forEach((value, key) => {...}) for you, you can skip the syntactic sugar library.
• GuB-42 1y ago
  The (terrible) solution that we are seeing now is generative AI. Instead of importing a library, you ask an AI to write the code for you, the AI most likely has ingested a library that implements the features you need and will essentially copy-paste that part into your code, transforming it so that it matches the rest of your code. I believe that it causes more problems than it solves, but it can be a solution to the problem of adding thousands of lines of code of dependency when you could write a 10-line function yourself. Of course, the proper thing to do is not to be the wrong kind of lazy and to understand what you are doing. I say the wrong kind of lazy because there is a right kind of lazy, and it is about not doing things you don't need to, as opposed to doing them poorly.
• ThouYS 1y ago
  I'm using fewer and fewer libs, thanks to ChatGPT. Just write me whichever function I currently need
• iainmerrick 1y ago
  No mention here of binary size (beyond linking out to a ClickHouse blog post on the topic). The total number of lines of code is relevant, sure, but for most practical purposes, compile times and binary sizes are more important. I don't know the situation in Rust, but in JS land, there's a pretty clear divide between libraries that are tree-shakable (or if you prefer, amenable to dead code elimination) and those that aren't. If you stick to tree-shakable dependencies your final bundled output will only include what you actually need and can be pretty small.
  • steveklabnik 1y ago
    Like many compiled languages, Rust does dead code elimination on everything.
    • iainmerrick 1y ago
      Right, but it depends on how the code is written, right? If you use mostly free functions things will shake out naturally, if you use lots of dynamic dispatch you'll pull in stuff that doesn't get called.
      • steveklabnik 1y ago
        Rust does static dispatch even for non-free functions. Only trait objects are dynamically dispatched, and most people argue they’re under-used in Rust, not overused. If that does become a problem, there are also techniques like https://github.com/rust-lang/rust/issues/68262 https://github.com/rust-lang/rust/issues/68262 too.
  • trehalose 1y ago
    > The total number of lines of code is relevant, sure, but for most practical purposes, compile times and binary sizes are more important. Perhaps for most practical purposes, but not for security, which the article's author seems more concerned with: > Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust... How could I ever audit all of that code? Tree-shaking can't help with that.
• yubblegum 1y ago
  So asking HN: whatever happened with OSGi? Does that architecture solve the problem, and if no, why not? https://docs.osgi.org/specification/osgi.core/7.0.0/framework.module.html#framework.module.dependencies https://docs.osgi.org/specification/osgi.core/7.0.0/framewor... "How OSGi Changed My Life" (2008) https://queue.acm.org/detail.cfm?id=1348594 https://queue.acm.org/detail.cfm?id=1348594
  • vacuity 1y ago
    This is my first encounter with OSGi. It seems to me that the "Lego hypothesis" reflects an increasing justified approach. The ACM Queue article mentions hot plugging and dependency injection, and a comment[0] in this thread brings up Sans IO. This also ties into capabilities, as a security measure but also an approach to modularity. The common thread is that programs should be written with a strong sense of boundaries: both what is included and what is not included is vital, and the boundary must allow the inside to communicate with the outside. Push dependencies to the boundary and create interfaces from them. The general principles for trivially pluggable components are all out there now. More efforts like OSGi will be needed to principles into practice. [0] https://news.ycombinator.com/item?id=43944511 https://news.ycombinator.com/item?id=43944511
    • yubblegum 1y ago
      I used to be firmly in the component oriented camp. The reality of the matter is that the conceptual (mental) model doesn't really represent the reality of composing with reusable components. All Lego components have the same simple standard mechanism: friction coupling using concave and convex surface elements of the component. Unix pipes are the closest thing we have to a Lego like approach and there the model of "hooking pipes of bytes from sources to sinks" actually represents what happens with the software. With components and APIs, unless we resort to some universal baseline (such as a small finite semantic API like REST's "verbs") that basically can marshall and unmarshall any arbitrary function call ('do (func, context, in-args, out-args, out-err)' the Lego metaphor break down very quickly. The second issue are the modalities of 'interactions' between components. So this is my first encounter with "Sans-IO" (/g) but this is just addressing the interactions issue with a fiat 'no inter-actions by components'. So Lego for software: great overall expression of desired simplicity, but not remotely effective as a generative concept and imo even possibly detrimental (as it over simplifies the problem). Now we have 2 different pieces of software tech that somewhat have managed to arrive at component orientation: using a finite set of predefined components to build general software. One is GUI components, where a small set of visual components and operational constructs ("user-events", etc.) with structural and behavioral semantics are used to create arbitrary visual interfaces for any ~domain. The other is WWW where (REST verbs of) HTTP also provide a small finite set of 'components' (here architectural) to create arbitrary services. With both, there is the tedious and painful process of mapping domain semantics to structural components. So we can get reusable component oriented software (ecosystems) but we need to understand (per lessons of GUIs and WebApps) that a great deal of (semantic) glue code and infrastructure is necessary, just as a lot of wiring (for GUIs) and code frameworks (for WebApps) are necessary. That is what something like OSGi brings to the table. This then leads to the question of component boundary and granulity. With things like DCOM and JEE you have fine grained components aggregated in process boundaries. The current approach is identifying process boundary as component boundary (docker, k8, microservices) (and doing away with 'application servers' in the process).