11 ms·
Rust without crates.io
- hobofan 3y ago> There is no mediation of any kind between when a new library/version is published and when it is consumed. > A simple time delay will allow egregious malware like malicious build.rs scripts to be caught, whether that’s the super-long Debian stable cycle or even the several days required to migrate from unstable to testing. Normal cargo + crates.io usage has the same property via the existence of lockfiles. For most people that care about stability, this will provide a natural delay (which you can also see in crates.io download graphs) to consuming the latest version. ----- The other points brought up in the article are definitely valid, but if that's the main strategy of defense you are looking for, you might already have it.
- dathinab 3y ago> if that's the main strategy of defense mostly, there can be a small attack gap when adding new dependencies to the project and at the specific point in time when you run `update` through you can also pin versions in `Cargo.toml` and then review any updates, maybe except for a few highly trusted sources (it's a bit annoying and costly (time wise) but viable) Through trying to vendor things, especially with Debian, seems like a horrible solution. And there is a lot of precedence for this causing tons of headaches, wrong bug reports and similar for developer (== time loss == time is money so we could probably be speaking about multiple millions of monetary damages). Through I have been thinking for a while that it could be interesting to have a partial crates.io "proxy" which only contains and updates crates which have gotten a "basic screening" in all of their code and dependencies probably with some tool assistance this might not find bugs but it should find many forms of supply chain attacks. Through given that this is costly to provide it probably would not be a free service.
- hobofan 3y ago> there can be a small attack gap when adding new dependencies to the project Most package managers will keep the versions of transitive dependencies as unchanged as possible when adding a new direct dependency. Of course if the only solution to satisfy the dependencies of the new direct dependency is to upgrade a transitive dependency, that will be done. (I've seen a lot of people treat dependency additions as completely unpredictable operations that regenerate the whole lockfile in the past, which is why I wanted to clear this up.)
- frankjr 3y ago> Normal cargo + crates.io usage has the same property via the existence of lockfiles. For most people that care about stability, this will provide a natural delay (which you can also see in crates.io download graphs) to consuming the latest version. Be aware however that lock files are ignored by default. You need to specify "--locked" if you want Cargo to use the "locked" versions from Cargo.lock. > By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published. https://doc.rust-lang.org/cargo/commands/cargo-install.html#dealing-with-the-lockfile https://doc.rust-lang.org/cargo/commands/cargo-install.html#...
- hobofan 3y agoThis only applies to the `cargo install` command to install binaries from crates.io. If you work in a repository on your own project and use basically any other cargo command (e.g. `cargo build`, `cargo run`), the lock file is of course taken into account (otherwise it would be near-useless).
- frankjr 3y agoRight but that's not the only use case for `cargo install`. You can also use it to install binaries system wide from cloned repositories (`cargo install --path .`). I use this often for small utilities (for which I have no intention of publishing to crates.io) or when testing before packaging for Arch.
- MrJohz 3y agoThis is installing binaries from crates.io directly (i.e. running "cargo install ripgrep" instead of "sudo apt..." or whatever equivalent). Admittedly, this is often a suggested installation method for some built-in-Rust tools, but it's usually an option of last resort, and most people should be using their usual package manager for this. Cargo does respect the package lock file when building a project from source (i.e. when you checkout a project, and run `cargo build` in that project directory). This happens, for example, when running builds in CI, or preparing a new release for each individual package manager.
- avgcorrection 3y agoUsing Debian [EDIT: APT] and indirectly `/usr/share/*` looks like a regressive solution. It’s a package manager used by a subset of Linux distributions.
- jjgreen 3y agoIf your target is Debian, then that's not really a problem? I do a similar thing with Python (target is Debian, only use apt-able dependencies, no packaging problems).
- dagw 3y agoIf you're only targeting one specific version of one specific distro on one arch, then 93% of all packaging problems tend to go away, and almost any approach will work.
- chrismorgan 3y agoThe positive side is that Debian isn’t the packager of record, so to speak: simplifying quite a bit, it’s pretending to be crates.io, just with a subset of the contents. That means that the developer doesn’t need to worry about it and can continue using Cargo and crates.io like normal, and the Debian packager, if there is one, just needs to translate Cargo.toml’s [dependencies] section into whatever Debian uses, and Debian-package any theretofore-unpackaged dependencies. All up, this makes it genuinely practicable. I’m still not convinced it’s worthwhile (and it does make life a bit harder in other ways, especially in ways that I’m sure will become more apparent over time as the system gets used increasingly), but it’s not as unreasonable as it may initially sound. Really, what’s being presented is Rust without crates.io (the domain name and content host), but with crates.io (the packages).
- ctz 3y agoYou just need to repeat the work for each linux distribution package manager, and then again for Windows, and then again Mac and iOS. Simples!
- dagw 3y ago
- jkbbwr 3y agoHaving worked with C/C++ projects. Managing dependencies is downright painful. About the best option we found was to treat everything as build from source and then git submodule our dependencies in. This is still not good but at least it gave us a consistent environment.
- palata 3y ago> Managing dependencies is downright painful. The risk when it is too easy is that you suddenly pull half of the Internet into your project. In C++, the cost of having a dependency makes me choose carefully if I want a dependency or not, and if I do, if I want this particular dependency (Who develops it? Is it maintained? If it stops being maintained, can I easily move to an alternative, or could I take over maintenance? ...). > About the best option we found was to treat everything as build from source and then git submodule our dependencies in. If you mean having a git submodule and doing something like `add_subdirectory(dependencyA)` in CMake, please don't do that. Building from source is fine, but then you can totally install the dependency locally and have your build system (be it CMake, Meson, ...) fetch it. At least I started doing it the way it is described here [1] and it works well for me. [1]: https://www.acarg.ch/posts/cmake-deps/ https://www.acarg.ch/posts/cmake-deps/
- vilunov 3y agoRequirement to "install dependencies locally" is part of the pain with C++ dep management. Not having to do it makes builds much easier to define, as you don't need to supports lots of distros and hope that everyone of them is up to date.
- palata 3y agoI don't know, I feel like it is a bit of a tradeoff. You can totally say "here is my code, it's open source, and the build system will look for those dependencies on the system" and let the users (who are developers) handle the dependencies the way they want. This way the user gets to choose if they trust the dependencies or not. Ideally the distro ships with those dependencies, and the user can just install them (better: write a package that depends on those dependencies and install that). It seems like developers don't really know how to do that anymore, and language package managers allow them to not learn about it. But at the cost of control: they don't know anymore what dependencies they are pulling, or even how many they are pulling. The modern way seems to be mostly about productivity, and therefore devs don't want to learn anything about dependencies, they want everything to "just work". But that is arguably bad for security.
- pasloc 3y agoIn my opinion this just fragments the rust ecosystem and creates more problems than it solves. What about to just have mirrors of crates.io? And I completely disagree that that someone else than the authors should distribute or manage the package. They know the best how and what there is to do in certain situation (security fixes and so on). Of course it is good to have independent reviews, but these independent people should not be in charge of updating, maintaining and updating these packages/libraries.
- chungus 3y agoOn unstable there's the "-Z no-index-update" flag which "ensures that Cargo does not attempt to update the registry index. (...) to avoid the network latency for updating the index each time" https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#no-index-update https://doc.rust-lang.org/nightly/cargo/reference/unstable.h...
- dathinab 3y agoFollowing stable options are often more then enough: `--locked` might not prevent index updates, but prevent any changes in used code, i.e. only what is in the lockfile will be used `--frozen` like locked but also requires the (download?) cache to be up to date `--offline` prevent index updates and downloads So if you have everything in the download cache `--offline` is like `-Z no-index-update` and `--locked` is most times good enough because since partial indices where added the index updates don't consume much time at all.
- chrismorgan 3y ago> What’s interesting is that this problem is largely solved for C and C++: Linux distributions like Debian package such a wide range of libraries that for many things that you want to develop or install, you don’t need any third-party libraries at all. This does not reflect my observations at all. It works if you’re an end user and the software you want has been packaged. But if either of those is not the case, you’re left in the mud. Rather, you end up with builds that are difficult to set up or reproduce except on specific platforms, require a lot more maintenance over time because bitrot occurs naturally (your code depends on libexample 1, but Debian only package version 2 now, so good luck dusting off that project you made five years ago and getting it to build again), vendoring upon vendoring, more diverse and generally lower-quality dependencies for a particular task, and other problems like these. You’ve solved one problem, but at a very significant cost that must be acknowledged, even if you decide it’s still worth it. Take an app written five years ago that depends on majorish C++ libraries, and there’s a fair chance compiling it will be a nightmare and require invasive surgery due to version incompatibilities. It’s not improbable that the most pragmatic solution may be to find and rebuild the transitive package tree for some older version of some package, and that’s generally quite painful (and can be outright incompatible). Take an app written five years ago in pure Rust, and it’ll almost certainly (p>0.99) still work fine. Take an app written five years ago in Rust that binds to some C/C++ library, and there’s a fair chance you’ll run into trouble, but it will almost certainly be a good deal easier to resolve, much more likely to work with only a version bump and no other changes than in C/C++. As a developer, I know which general mode of operation I prefer. Debian are welcome to twiddle the dependencies in the way that suits their chosen requirements, but the canonical sources are still crates.io, and I know that when they’ve stopped packaging example-1.0.0, I won’t be beholden to their packaging but can just use `cargo build` and it’ll still work (contingent upon extern bindings). I’m dubious that any meaningful curation is happening, and curious if there’s even cursory review of crate contents. But most of all, I’d be interested in a statistical comparison of the crates that Debian ships with the most-downloaded crates from crates.io. I suspect the sets would look very similar.
- josephg 3y agoYeah. The part that gives me goosebumps with that proposal is: as an author of some not very popular packages, what do I do? Is it my job to package and promote my packages on Debian, redhat, homebrew, and all the others? Do I need usage before they’ll pick it up? Sounds like a chicken and egg problem. And doing all that packaging work sounds like a boring, time consuming job that I don’t want. It’s also quite common for me to pull out bits of code I’ve written that I want to share as a dependency between different programs I write. I have dozens of packages in npm along these lines - and it’s quite elegant. I factor out the code, package it independently, write a readme and some tests and publish it. My other packages can then just depend on the brand new package directly. In this brave new world the author is proposing - what? I make a .deb, argue for it on some mailing lists I’m not on, wait who knows how long, eventually gain permission to depend on my own code, then realise I can’t install it on my Mac laptop and do the same dance in homebrew? Oh - there’s a bug. Better manually update all the package repositories? Sounds like a complete nightmare to me. I’m sympathetic to the problem. Supply chain attacks are way too easy. But maintaining up to date packages on every single Linux distribution, homebrew, ports, and whatever windows does sounds like a full time job. A boring full time job. And it would still not work as well for my users, because when a new feature is released in one of my upstream dependencies I can’t use it until all of those package repositories ship the new version of my upstream dependency too. I’d rather eat sand. “It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in. You can tell by looking at just how few dependencies the average C program uses. It’s like that because of how painful it is. Anything under a certain size tends to get either reinvented every time, badly (hash table libraries in C) or vendored (eg json parsers). Common libraries like rust’s rand don’t exist in C because it’s just too much effort to pull in dependencies. So everyone rolls their own crappy version instead. Ask GTA5 how well that went for them. And that’s not the developer ecosystem I want to work in. It’s frankly a terrible proposal.
- loftsy 3y agoWhy is a Debian developer a better gatekeeper against supply chain attacks than the developer of a popular Rust library? For me the alternative to Crates.io/npm/PyPi is a "platform release" like Android where a bunch of stuff is signed off by a corporation.
- dathinab 3y agobecause it often takes so long to do any updates that people just give up on updating in time, weather that is for bug fixes or for injecting supply chain attacks /s/j
- palata 3y agoI would assume that the Debian _maintainer_ cares about what is being shipped with Debian. And for each dependency (transitive or not), the Debian maintainer has to make sure that it is shipped by Debian, and therefore has to not only know about the dependency, but also to make a package out of it (if it is not already being shipped by Debian). Whereas the developer of a popular Rust library most likely just added some dependencies that were convenient, and doesn't know the full story of the transitive dependencies. To me it is much more likely that somebody at Debian _saw_ the dependency being shipped than an arbitrary Rust developer, who probably does not even know how many dependencies are being shipped with their program.
- throwaway322321 3y agoDebian developers go through a long step-by-step process to get in that position. Volunteering as a package maintainer first, getting Debian dev to advocate for you, cross-checking people passports, doing interview/exams to become a DD. DDs review the libraries they package. All uploads are personally signed to keep people accountable. Instead opening a github account and developing a library can be done anonymously and there's been supply chain attacks done this way.
- yjftsjthsd-h 3y agoWell if nothing else, it's the difference between one person being able to ship something vs needing at least one person to review before shipping.
- zaptheimpaler 3y agoYeah making Rust dependencies anything at all like C/C++ dependencies or tying them to the system package manager sounds just absolutely terrible to me. I don't want to get too deep here, all I know is compiling a C++ project can be a clusterfuck nightmare full of completely nonsense errors and the whole apt/deb interface is not very intuitive either. On the other hand, compiling Rust projects is easy. Works great. There are very good reasons for language toolchains to remain independent at all levels from OS toolchains.
- Dowwie 3y agoThese concerns pertain to every language using a centralized package authority. None of them-- crates.io, npm, pypi, hex -- are adequately funded by the multi-billion dollar industries they enable. So, they aren't as resilient and secure as they can be. The solution isn't to create yet another package authority but to reinforce those that the ecosystems are already relying on.
- metaltyphoon 3y agoNpm is owned by Microsoft no?
- jjice 3y agoYes https://www.zdnet.com/article/microsoft-buys-javascript-developer-platform-npm-plans-to-integrate-it-with-github/ https://www.zdnet.com/article/microsoft-buys-javascript-deve...
- tikkabhuna 3y ago> If crates.io goes down or access is otherwise disrupted then the Rust community will stop work. As someone who has worked in a big corporate their entire career this statement is bizarre. This has been solved for Java for over a decade. Builds don't access the internet directly. You host a proxy on-premise (Nexus/Artifactory/etc) and access that. If the internet location is down, you continue as normal. > You need only one author in your maybe-hundreds-of-dependencies tree to be hacked, coerced or in a malicious mood for you to have a really bad day. There are products (Nexus Firewall) that can check dependencies for vulnerabilities and either block them from entering the network or fail CI pipelines.
- richardwhiuk 3y agoEvery serious enterprise runs a Rust proxy - and indeed Artifactory / Azure Artifacts and others support this.
- jrpelkonen 3y agoSure, Artifactory, etc. have their places. But I am objecting to your "no true scotsman"[1] type of argument. Furthermore, the most bizarre setups I have seen have been in "serious" enterprises. Hardly an endorsement. 1: https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
- devman0 3y agoI have to agree with the poster you are replying to. If an entity is not doing the bare minimum to QA software supply chain, then they are not "serious". That being said, I have seen many a large enterprise that do not treat software development as a serious endeavor and it shows.
- brabel 3y agoThe main problem the author is talking about is actually about version updates, which in Maven as well as crates.io is up to each lib's author, and is not curated in any way. There's no technical solution to that, really. Do you think Nexus Firewall can pick up every exploit, or even most? How confident of that are you, and what data do you have to back that up? I don't have any myself, but would not be surprised at all if "hackers" can easily work around their scanning. However, I don't have a better approach than using scanning tools like Nexus, or as the author proposes, use a curated library repository like Debian is doing (which hopefully gets enough eyeballs to remain secure) or the https://github.com/crev-dev/cargo-crev https://github.com/crev-dev/cargo-crev project (manually reviewed code) also mentioned. It's interesting that they mention C/C++ just rely on distros providing dynamic libs instead which means you don't even control your dependencies versions, some distro does (how reliable is the distro?)... I wonder if that could work for other languages or if it's just as painful as it looks in the C world.
- kookamamie 3y ago[flagged]
- palata 3y agoI think that there is a real point in talking about the security risk from having it too easy to fetch tons of dependencies. That's literally code that gets downloaded and run on the user's machine, I would always feel safer if the dev had at least an idea about the dependencies they pull. You can also have a mixed approach, where you depend on the system libraries for those that are maintained by the distro, and build/handle the remaining dependencies manually. That is an incentive to actually use the libraries that are maintained by the distro, which is a good thing IMO: not every library gets to be distributed by Debian, there is some level of quality control there.
- josephg 3y agoMaybe if we work hard enough, we can have a big “can I use” table of rust dependencies which shows which versions of everything are available in which operating systems and what the package has been renamed to in each case. “Oh I could use rand but the latest version isn’t in gentoo so that would knock out 5% of my users.” Maintaining and stressing over that sounds like greeeaaaat fun. “Oh cool! An important new feature landed in rust stable. I can’t wait for 6 months to pass before the new rust compiler comes to Debian. Then another 6 months for my dependencies to use it. Then another 6 months before I can use it! Isn’t software great?”
- dystroy 3y agoHaving vetted repositories of libraries looks neat but it's harder in practice than what the article lets appear. A problem for such repositories like the Debian one is that it can only contain vetted versions of libraries. Many Rust libraries are still 0.x and have changing API and frequent updates, which makes it hard to vet all dependencies, especially when an update implies updates of the dependents.
- mborch 3y agoPerhaps this should be an incentive for library authors to finalize the 1.0 API and accept that there's eventually going to be a 2.x down the road, a "less is more" approach.
- ReleaseCandidat 3y ago> this problem is largely solved for C and C++: Linux distributions like Debian package such a wide range of libraries that for many things that you want to develop or install, you don’t need any third-party libraries at all. Even if it were solved on Linux (and it isn't, at least not for "our" dependencies), this doesn't help MacOS and Windows users. And no, Homebrew and Chocolatey (or whatever) don't háve all needed packages either. Crates.io does have problems, the main one being non-existent namespaces.
- baby 3y agoI love the absence of namespace personally. No name collisions!
- Aeolun 3y agoSo now you replace your trust in crates.io with trust in the debian repository (which god knows how many people have access to?). At least on crates you get to know who is maintaining the package.
- jjgreen 3y agoYour trust in crates.io extends to their authentication, which is subcontracted to GitHub (and only GitHub) so Microsoft. Debian? Microsoft? ... that's a tricky one. https://github.com/rust-lang/crates.io/issues/326 https://github.com/rust-lang/crates.io/issues/326
- Aeolun 3y agoWow, I’m more bothered by all the faffing around they do on that issue (write an RFC before developing, but discuss it on this issue more before writing an RFC) than the actual issue of a single auth provider.
- jjgreen 3y agoDebian's internal procedures are arcane for sure, I'm not involved so don't see it as issue. It's having a single authentication provider which is Microsoft that I'd object to, and that would put me off using crates.io at all.
- goodpoint 3y agoDebian requires much more strict authentication.
- selfmodruntime 3y ago> What’s interesting is that this problem is largely solved for C and C++: Linux distributions like Debian package such a wide range of libraries that for many things that you want to develop or install, you don’t need any third-party libraries at all. How is this not a third-party library, and how is this not exactly the same problem as with Crates.io?
- nullifidian 3y agoThe issue with the Rust ecosystem, and other similar "move fast and break things" npm-like ecosystems is not the lack of review per se, but the fact that most modules are microdependencies, or depend on them, each with its own maintainer, and it's very easy to end up with 100-300 dependencies(with a slightly lower number of authors) in the most trivial use cases. In the C++ world libraries tend to be larger, with multiple maintainers, or even unified into a single meta library like boost, which alleviates trust issues to a degree. Some degree of vetting at the level of distros like RedHat further lowers the risks. The current situation with the language ecosystems is basically "we must build this very lax system with as few friction moments as possible, or else our language would be outcompeted by a language with laxer requirements" and "We must make our ecosystem grow, grow, grow! Externalities down the line be damned!", i.e. it's a capitalism, careerism influenced situation imho. So we have "You must vet all the 300 modules and their updates yourself" as the result. Requiring module authors to unify into meta projects to review each other's commits is perceived as laughable, unneeded self-hindrance. Same goes for separating the ecosystem into vetted(stable)/unvetted(unstable).
- jokethrowaway 3y agoThis is an issue caused by the fact we sold software as being FREE and now businesses have an expectation that software is free to make; thank you random guy in Nebraska. I would love to have a paid curated package repository, but it would be really hard to sell. I know a handful of companies who care that much about their security not to depend on OSS libraries. Even FANG don't care. If you can raise some money and get enough sales firepower to close big contracts with the few security conscious actors, this might become reality.
- qalmakka 3y ago> Rust as we know it stops if this one particular web service goes down. True that, but that is only because we got used to not having to download dependencies manually (like with C). The option of "download all of your dependencies by hand and slap them in a directory" is always there, only instead of invoking ./configure with --with-libxyz=PATH or setting XYZ_PATH you modify a Cargo.toml. > this problem is largely solved for C and C++: Linux distributions like Debian package such a wide range of libraries that for many things that you want to develop or install, you don’t need any third-party libraries at all. That's rich. If that was the case, there would be no Docker. What about Windows? On Windows installing C/C++ libraries is a massive PITA. What about if your distro packages the wrong library version, or it's been built with a set of flags you can't use? The whole point is moot IMHO. A small part of why we ship libraries through package managers is because disk space was scarce years ago and it's arguably "safer" to build shared objects (at least for C libraries, C++ has templates so shared objects is a recipe for disaster and requires rebuilding everything anyway). The critical reason why we have a `libxyz` package in the average distro is because _C/C++ build system is a massive mess_, and that's the only sane way to distribute and allow people to build packages without wasting several hours to find and gather all dependencies. This approach has considerable downsides, so that's why we now have Flatpak and the like.
- uecker 3y agoI don't agree. IMHO Building on top of shared libraries with stable interfaces managed by a distribution package manager is the better approach. All this fast changing dependencies where you then end up having multiple installed at the same time in different containers, statically linked programs, or whatever is a complexity, security, and long-term maintenance nightmare.
- classified 3y ago> It is profoundly unresilient to have a single point of failure like this. Try telling this to GitHub customers. Half the world was unable to work in the last GitHub outage. Not to mention NPM.
- ManBeardPc 3y agoWhat I really miss is a common format for package managers with a uniform, simple and well documented API and a fully FOSS implementation. Dependency resolution and garbage collection would probably still be unique to each format, but maybe even this could be unified for some cases. This way you would only need one service for all your needs. Everyone could host their own proxy/cache/mirror/repository. BOMs would be simpler. CVEs easy to attach and query. Tools could handle multiple different package types. Auth could be shared and simplified as well. Less reliance on a single provider. Projects with multiple languages would be easier. New tools that need packages would not need to invent everything from scratch. This of course only slightly helps with the trust problem, but at least offers a common way to attach things like CVEs or if someone has reviewed a package/version and share some effort between different platforms. In the end you still need people to put in the effort to actually review the packages.
- yjftsjthsd-h 3y ago> What I really miss is a common format for package managers with a uniform, simple and well documented API and a fully FOSS implementation. RPM? You'll never get everyone actually on a single format, but even then between LSB and just its actual popularity it has support on most distros (even if only by way of converters like alien)
- ManBeardPc 3y agoRPM seems to be focused on Linux and globally managing the whole system though, but yeah, something like that. Probably would require to work more like in cargo or npm with a per-project list. Agreeing on one format is kind of the point. I know it is difficult, but I think it is possible if at least a few implementations agree on something. Maybe some kind of "backend" like LLVM which could be used by cargo, npm, maven, etc. to configure their resolution strategy. Probably only a dream though.
- defanor 3y agoSimilarly being unhappy with pulling dependencies without using a system package manager, in part because it seemed tricky to make them work with a compiler version from system repositories, this is something I wanted to try as well, and finally tried now. After having rustc installed, had to install 157 more packages in order to get argument parsing (librust-clap-dev), but it is still faster than cargo updating its package index, and then it actually builds! Would be even better if the dependency tree for a basic package was more modest, about 157 times smaller.
- defanor 3y agoFWIW, I noticed that librust-argparse-dev is also packaged for Debian, and it comes without additional dependencies. It also builds fine when pulled from crates.io directly, while using rustc version from Debian repositories, unlike clap. So whether it is crates.io or system repositories, it may be worthwhile to spend more time looking for more lightweight and stable alternatives; most commonly suggested ones are not necessarily that.
- duped 3y agoIf you used cargo, you would only have 27 dependencies pulled in for clap. I'm guessing the bulk of those dependencies are the optional dev-dependencies that are pulled in during builds, and often include things like criterion for benchmarking which has a few dependencies of its own.
- doubloon 3y agoi would rather have one source of fragile success (crates.io) than millions of points of constant failure. (Cmake, ninja, automake, etc etc) dynamic linked libraries are a way to save tiny amounts of RAM and disk space while wasting millions of hours of people's time.
- mats852 3y agoIt's like complaining about hardware stores because you can buy your wood from the lumberjack and nails from the blacksmith. I'd rather have a system devoid of emotional biases, transparent and introspectable and rely on automated tooling to constantly monitor my dependencies.
- nindalf 3y agoThe author makes one big assumption - that the packaged code in Debian is better because it has been reviewed and vetted by someone competent. If this was true, it would make this approach worthwhile, but I don't think it is. The author could have pointed to some data that demonstrates this, but didn't. I remain unconvinced. As far as I can tell, this is more about the feeling of security because of familiarity. I can completely understand why they would feel more comfortable using something they've successfully used for many years, but that feeling doesn't actually translate to improvements in security. There is a feeble attempt to suggest that slowing down the rate of change improves security, but I would prefer to have the latest versions of the language toolchain and libraries available immediately, so I can make a decision on what to use myself. If security actually needs to be addressed, it's going to be by paying maintainers and paying people to review dependencies. And that would likely be in the framework of PyPI/crates.io/npm, because none of them place constraints on where the code can be used, unlike using a single Linux distro's package manager.
- Arnavion 3y ago>The author makes one big assumption - that the packaged code in Debian is better because it has been reviewed and vetted by someone competent. They do not make this assumption, and in fact state the opposite.
- nindalf 3y agoThey state that > They might decide to give the diff at least a cursory look, which is better than nothing. So yes, they do value the cursory review, which isn’t that useful. And they talk up the security benefit of only pulling in a subset of published updates, which is even more dubious. Contrast this with the approach that Google is taking here - https://opensource.googleblog.com/2023/05/open-sourcing-our-rust-crate-audits.html?m=1 https://opensource.googleblog.com/2023/05/open-sourcing-our-.... Actual, careful review of the code that anyone can take advantage of. And no limitation of having to choose one flavour of one OS. This review helps you audit your supply chain regardless of OS you’re developing and deploying on.
- 3y ago
- weinzierl 3y ago"There is no mediation of any kind between when a new library/version is published and when it is consumed." In the corporate world this solved by using an intermediary like Nexus or JFrog.
- duped 3y agoYou are fundamentally limiting your ability to build Rust projects if you rely on the system's package manager. A Rust program `p` can depend on `q` and `v` which depend on `u`, but `q` requires `u` @ 1.0.0 and `v` requires `u` @ 2.0.0. In the Debian world, you would have to change `p` and `v` to support a mutually compatible version of `u`. If those changes introduce bugs, they're Debian's bugs, but the author of `p` is the one who gets the reports. Cargo naturally handles this case, it installs multiple incompatible versions at unique locations in its cache, and correctly invokes the system linker to link the compiled objects correctly. If `p` was a C program, we could not handle this case, because `u` would be a shared library and you cannot link two different versions of the same shared object into a single address space at runtime (if they were static libraries, a sufficiently smart build system could deal with it, just like cargo - but there is no unified build system and package manager for C, so good luck). Further, Cargo creates lock files for your projects, and authors commit and push them for executables. If you build against a lockfile. This prevents most of the cases that the author is bemoaning. People don't run Cargo update willy nilly. Frankly, the "problems" the author is claiming don't exist in practice and the "solution" makes life worse as a Rust developer. I'm getting tired of hearing people praise broken package managers like Debian. They make life worse for everyone for no benefit. How many times do people need to read testimony that "C/C++ package management sucked and Rust is great just for Cargo alone!" before they get it?
- TeeMassive 3y agoLock files are only one part of the solution. Even with a lock file dependencies need to be fetched from somewhere and if a particular version of a package was deleted by its author then you're back to case one. "People don't run Cargo update willy nilly." We don't know the same people then. Most devs like to keep their stuff up to date. Auditing the dependencies tree before updating a lock file is like reading a ToS. Most people don't even if they should.The problem of securing the chain of production is not solved with a simple lock file.
- estebank 3y ago> Even with a lock file dependencies need to be fetched from somewhere and if a particular version of a package was deleted by its author then you're back to case one. That's not quite correct. Even if a library version is yanked (which hides it in crates.io and makes it so cargo will skip that version during resolution), if it appears in a lock file, cargo will dutifully find it and use it. This is explicitly to avoid the leftpad behavior of a library disappearing breaking your build.
- sc68cal 3y agoMy experience of OS packages of python libraries was pretty awful. They are out of date and it was impossible to deal with conflicting versions of a shared library that maybe two different applications needed. Switching to VMs and Docker containers helped but really this was a band-aid to deal with the fact that you couldn't depend on an OS package manager to handle this well. Packaging up Rust libraries with the OS package manager is just going to have the same problems.
- plandis 3y ago> If crates.io goes down or access is otherwise disrupted then the Rust community will stop work. It is profoundly unresilient to have a single point of failure like this. Certainly some people will have vendored their deps and others will have a panamax mirror handy, but for most, Rust as we know it stops if this one particular web service goes down. If you can’t wait for recovery, I’m pretty sure cargo can pull source from any accessible git repo. > There is no mediation of any kind between when a new library/version is published and when it is consumed. You need only one author in your maybe-hundreds-of-dependencies tree to be hacked, coerced or in a malicious mood for you to have a really bad day. How is this prevented on Linux distros? Surely not every piece of code is being audited to prevent supply chain attacks?
- zajio1am 3y agoEven if we put away conversation about OS-provided vs. direct-from-developer libraries, why would one want centralized repositories like crates.io, instead of decentralized CVSs like Git, and just link Git URL like dependencies? Seems to me that such centralized repositories have all common disadvantages of centralized solutions, but without their advantages.
- zdimension 3y agoUsing cargo (the build tool), you can add dependencies using Git URLs. Referencing crates.io is just "the usual way" to add dependencies. Git repositories can disappear (becoming private, author changes username, etc.), crates.io is more resilient than the average OSS project repo. Most people use one of a handful of Git hosts anyway (GitHub, GitLab, ...) so Git dependencies are still like a huge centralized repo.
- pornel 3y agoCargo already supports using git repos as package sources, but it's painful to use due to slowness of the git protocol. HTTP registry updates are nearly instant in comparison. If git was the only option, most people would probably just use GitHub, which is merely changing one big host for another. When it's actually decentralized, when developers use their own git URLs on their own domain, suddenly you have to worry about things like the domains expiring. You have to worry about security practices of every single host. crates.io is a big target, but they know it and act accordingly. crates.io doesn't allow deleting of packages, but an arbitrary git URL can disappear causing a "left-pad" incident. Some of these problems could be solved with another protocol better suited for decentralization (where the data is immutable and content addressable, and identities are private keys rather than domains), but that's way more complex than "just use git".
- r2vcap 3y agoProblems of Rust: In Rust, there is a central package registry (crates.io), but there is a lack of ability to add dependencies using git URLs. Problems of Swift: Swift can add dependencies using git URLs, but there is no central package registry. Both languages should have a central package registry and the ability to add dependencies using git URLs, as it would make flexible dependency management much easier. For a central package registry, it makes it easier to find popular packages, but it has a problem of centralization.
- LawnGnome 3y agoRust does have this: you can specify dependencies from other registries or Git repositories pretty easily. (See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-other-registries https://doc.rust-lang.org/cargo/reference/specifying-depende... for the exact Cargo.toml syntax.)
- moltonel3x 3y agoRust does support adding deps using git urls, alternate repositories, filesystem paths, and even multiple sources for one dep. https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-other-registries https://doc.rust-lang.org/cargo/reference/specifying-depende...
- r2vcap 3y agoI was wrong :(
- deleted 3y ago[deleted]
- shakow 3y ago> but there is a lack of ability to add dependencies using git URLs. There is absolutely no lack of that in cargo; this is literally the first sentence in the relevant section in the doc (https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html https://doc.rust-lang.org/cargo/reference/specifying-depende...): Your crates can depend on other libraries from crates.io or other registries, *git repositories*, or subdirectories on your local file system.
- jchw 3y agoPersonally, I really like the solution that Go has come up with. Go does have the huge downside that its system needs some centralized servers (and possibly worse depending on how you look at it, defaults to Google's for them), but it has the huge upside that it avoids any particular single point of failure. If you are using a module proxy (like the default,) you can still fetch dependencies from GitHub packages when GitHub is down since it can grab through the module proxy. But if the module proxy is down, it can just fetch the package from GitHub instead. It's also possible to add multiple proxies to the GOPROXY variable, if you wanted extra redundancy. But what makes this design especially nice, in my opinion, is that it means that you can host Go modules anywhere, so as long as you follow some conventions. And because of the GOPROXY mechanism, even if your site is a little unreliable, it shouldn't generally be much of an issue. The sumdb and lockfiles additionally helps ensure that the integrity of existing published versions of packages can't be compromised even if e.g. somebody acquires the domain later on. I'm not saying this is perfect, but it's a really nice blend of properties. If you're going to have to have a centralized entity that provides package services, making it work like this prevents a hard-dependency on a centralized party to handle package management while reducing some of the security and reliability concerns of self-hosting package repositories. Admittedly, this would lose some of its luster if a lot of packages became unavailable because the GOPROXY stopped caching them and they were long removed from the Internet; but OTOH, things eventually disappearing is always going to be a concern on the Internet, as the act of keeping things online is an active effort that requires time and money. As of today, I think this remains a sensible compromise. This doesn't solve curation at all, but that's kind of the thing; if programming language package management is the wrong place to gate insecure or potentially even malicious packages packages, then they may as well not colocate packages to specific central locations, either.
- TeeMassive 3y agoWhere I work we use Artifactory as a proxy and a private package hosting service. It works great but it could be better. Especially when it comes to flat downloads with no underlying package manager like Yocto/Bitbake and some are just terrible to setup like NugGets. The fact is, most languages or frameworks were just not designed with private repositories in mind. This is made worse when you have to make these private repositories integrate with non-developer environments like a CI worker node or a BitBake recipe. BitBake has a flat cache for downloads and binary builds and you can setup remote caches online for example and it works great. There is a dire need for some kind of universal caching protocol. Downloading files addressed by content through a proxy should be trivial and universal.
- baby 3y agoI stopped reading when he said C/C++ does it better
- 0xfedbee 3y agoLot of backlash against author’s suggestions. I guess it will take a DDoS attack on crates.io to convince this crowd. Can’t wait for that to happen.
- 1vuio0pswjnm7 3y agoProgress.
- Jean-Papoulos 3y ago> If crates.io goes down [...] certainly some people will have vendored their deps and others will have a panamax mirror handy, but for most, Rust as we know it stops. So, there are solutions already. > There is no mediation of any kind between when a new library/version is published and when it is consumed. No one is forcing you to stay automatically up-to-date. I personally on changes in critical libraries before updating. > Any tampering with crates.io itself (espionage, disgruntlement, national security) could have an incredibly wide blast radius, or a incredibly wide set of targets from which to choose. Repeat of point 1. > Since crates.io is the source for crates, it is normal for both developers and CI machines to be hitting this web service all the time. Opportunities for mischief are exacerbated when clients are phoning home so frequently. Again, point 1. > So what’s the alternative? Given there's only 2 problematic points in this list and you have pointed to solutions already, I don't see any need for an alternative.