12 ms·

I like this idea, but I dont know if Hyper is the best package to go with. Hyper occupies part of the Rust ecosystem that I think suffers from package bloat, li

• danielheath 6y ago
  IMO "how many packages are the dependencies broken into" is a far less useful question than "how many maintainers have commit access to the dependency subtree". The latter is a better question because: * It's directly connected to your security posture. * It's a stable metric across languages with different norms about module size.
• sk2020 6y ago
  A quick check of cargo-Geiger shows many hundreds of unsafe invocations in the dependencies of hyper. I think it’s hard to argue that some rust HTTP library is Irrefutably safer when you’ve thrown out so many of the static guarantees of the language and replaced them with “dude, trust me”.
• stjohnswarts 6y ago
  It's fine, how often do you have to rebuild curl?
• delfinom 6y ago
  Pfft, vetting dependencies is for boomers. -- Modern software engineering
• nine_k 6y ago
  All these packages provide important pieces of functionality, which, I suppose, mostly cannot be omitted. Either you depend on other's work for that, or you roll your own. Choose your poison.
  • est31 6y ago
    A quick survey gave me: * tracing is only present for logging purposes. does curl need it? It should be configurable. * itoa is only present for performance purposes, and only seems used by the server. * It seems that a bunch of projects in the dependency tree use pin-project which is heavyweight and instead could use pin-project-lite. Some already do, which creates both being used, so you are worse off than just with pin-project alone... * hyper contains code for both http servers and clients. Even if the dependencies are the same (and as seen above they are not), having to compile the server code for the client means an increase in compile time. It would be cleaner to provide separate server and client flags to make it possible to turn one off.
    • nicoburns 6y ago
      The latter is scheduled for the next release https://github.com/hyperium/hyper/issues/2223 https://github.com/hyperium/hyper/issues/2223
  • cle 6y ago
    There’s a third option if the package is so important: put it in the standard library. Technically it’s still a dependency but the standard library is maintained with a standard that is rarely matched by third party libraries, and can dramatically simplify the ecosystems’ dependency graph.
    • nextaccountic 6y ago
      Currently the mpsc channel on the stdlib is worse than the one found on crossbeam (according to the author[0]). That's one data point. [0] https://internals.rust-lang.org/t/scaling-back-my-involvement-in-rust/11754 https://internals.rust-lang.org/t/scaling-back-my-involvemen...
• steveklabnik 6y ago
  https://wiki.alopex.li/LetsBeRealAboutDependencies https://wiki.alopex.li/LetsBeRealAboutDependencies > These complaints are valid, but my argument is that they’re also not NEW, and they’re certainly not unique to Rust ... The only thing new about it is that programmers are exposed to more of the costs of it up-front.
  • k__ 6y ago
    "It has always been that way." doesn't seem to be a helpful statement here.
    • stjohnswarts 6y ago
      Yes but people often throw out empty criticism without offering solutions or alternatives and it gets annoying after a while and is just a complaint rather than a healthy critique. As programmers we've heard them all before I guarantee it, so why add another one to the pile?
    • steveklabnik 6y ago
      I mean, in some sense you're right, but in another, the point is that it's not a regression.
      • k__ 6y ago
        True. Also, I coud imagine Rust's type-system raises the bar of dependencies that can be mangaged before everything breaks down. So a lib with 100 deps in NPM isn't the same as a lib with 100 deps in Cargo.
        • tracker1 6y ago
          If you lock to specific versions, I don't think it differs much. Edit: I mean if you don't use ~ or ^ in your nose dependencies,. Just explicit versions.
  • svnpenn 6y ago
• kingkilr 6y ago
  While I don't love the proliferation of dependencies, from a risk perspective the raw number of dependencies isn't always the right metric. Looking at the authors and publishers numbers from https://github.com/rust-secure-code/cargo-supply-chain https://github.com/rust-secure-code/cargo-supply-chain it's clear a lot of these are maintained by the same set of trusted folks.
  • count 6y ago
    This is true today, will it remain true?
    • nicoburns 6y ago
      Well yes if they're published as part of the same project as lots of these are. In C/C++ you wouldn't do this because consuming a library is a pain so you want to minimise the number of dependencies. In Rust, what would be 1 library in C often gets broken up into a few that are published together in order to allow people to depend on only the functionality they need.
    • simias 6y ago
      You could argue the same thing for many big monolithic C projects though. How many of the original authors/maintainers are left in OpenSSL or the Linux kernel? My main worry about Rust dependencies is not so much the number, it's that it's still a fairly young ecosystem that hasn't stabilized yet, packages come and go fairly quickly even for relatively basic features. For instance for a long time lazy_static (which is one of the dependencies listed here) was the de-facto standard way of dealing with global data that needed an initializer. Apparently things are changing though, I've seen many people recommend once_cell over it (I haven't had the opportunity to try it yet). Things like tokio are also moving pretty fast, I wouldn't be surprised if something else took over in the not-so-far future. It's like that even for basic things: a couple of years ago for command line parsing in an app I used "argparse". It did the job. Last week I had to implement argument parsing for a new app, at first I thought about copy/pasting my previous code, but I went on crates.io and noticed that argparse hadn't been updated in 2years and apparently the "go to" argument parsing lib was now "clap". So I used clap instead. Will it still be used and maintained two years from now? Who knows.
• timdorr 6y ago
  I count 14: https://github.com/hyperium/hyper/blob/master/Cargo.toml#L22-L37 https://github.com/hyperium/hyper/blob/master/Cargo.toml#L22... bytes, futures-core, futures-channel, futures-util, http, http-body, httpdate, httparse, h2, itoa, tracingfeatures, pin-project, tower-service, tokio, want
  • est31 6y ago
    Those are the direct dependencies. They have dependencies of their own. What counts is the entire DAG traversal including all transitive dependencies.
    • seg_lol 6y ago
      I count 67 --> 42 (after removing the dev only deps) https://gist.github.com/seg-lol/0d22cf5002f890305cfd094f9ed1094a https://gist.github.com/seg-lol/0d22cf5002f890305cfd094f9ed1... *edit, passed in -e no-dev to `cargo tree`, thanks @est31 for the suggestion
      • est31 6y ago
        The number is smaller than that. You need to pass -e no-dev to cargo tree to filter out the dev-dependencies (which are only relevant for hyper development).
        • seg_lol 6y ago
          Thanks!
  • svnpenn 6y ago
    Here is my script: https://github.com/nu8/cove/tree/master/rust-deps https://github.com/nu8/cove/tree/master/rust-deps
• dochtman 6y ago
  Some of these are tiny, and a substantial part are platform-specific, so you will never need all of them for a single build.
• johncolanduoni 6y ago
  Most of these are maintained as sets of crates under the same project/maintainers. For example, everything starting with futures comes from one repo, everything starting with tokio plus mio (plus some others) are under the tokio-rs GitHub organization, all the windows bindings packages are from the same repo, etc. Plus some of the dependencies are also dependencies of the standard library (hashbrown, cfg-if, libc).
• nicoburns 6y ago
  Part of this is just crates being broken up more in Rust. For example the `http` crate only contains trait (interface) definitions. They break down like so: Platform integration: libc, winapi, winapi-build, winapi-i686-pc-windows-gnu, winapi-x86_64-pc-windows-gnu, ws2_32-sys, fuchsia-zircon, fuchsia-zircon-sys, kernel32-sys, redox_syscall Primitive algorithms: itoa, memchr, unicode-xid Proc macro / pinning utilities: proc-macro2, autocfg, cfg-if, lazy_static, quote, syn, pin-project, pin-project-internal, pin-project-lite, pin-utils Data structures: bitflags, bytes, fnv, hashbrown, indexmap, slab Core Rust asyncio crates: mio, miow, iovec, tokio, tokio-util, futures-channel, futures-core, futures-sink, futures-task, futures-util, Logging: log, tracing, tracing-core The following are effectively sub-crates of the project: http, http-body, httparse, httpdate, tower-service, h2 Not sure what these are for: net2, socket2, try-lock, want
  • nindalf 6y ago
    Why does hyper pull in hashbrown? Isn't it identical to std::collections::HashMap?
    • steveklabnik 6y ago
      I imagine that it's there because it is older than that happening. HashBrown's README says > Since Rust 1.36, this is now the HashMap implementation for the Rust standard library. However you may still want to use this crate instead since it works in environments without std, such as embedded systems and kernels. I don't know if that use case is important to Hyper or not.
    • CUViper 6y ago
      The dependency goes hyper -> h2 -> indexmap -> hashbrown, and indexmap is built on hashbrown::raw::RawTable.
      • scottlamb 6y ago
        Looks like the indexmap -> hashbrown dependency was added recently: https://github.com/bluss/indexmap/pull/131 https://github.com/bluss/indexmap/pull/131
• no_wizard 6y ago
  I've recently taken a different stance on this. the issue isn't number of dependencies, its the reliability of said dependencies to be the following - Secure. Does this contain malicious code, exploits or otherwise known bugs that could be fixed but aren't? This of course is hard, and will never be perfect. There are static security scans though, espcially in a language like rust, that should be able to verify what the code does before its being published for consumption via carog. NPM is trying to do something similiar. This isn't foolproof and more sophisicated exploits always will exist, but getting low hanging and mid-level fruit should be within reach, which is a net win - Is it quality? This is beyond just secure, but does it provide real utility value? One thing we always hear is DRY, which may mean sometimes you consume alot of dependencies, since the problem space you work in involves a lot of things, so why re-invent every single fix if something exists and you can glue together to start making impact in your problem domain? I don't think this is an issue espcially if #1 is true So I don't know, I think its fine to have a lot of dependencies, I think the justification for those dependencies is often related to the complexity of the work involved. I'd expect a cURL replacement to have quite a few dependencies, since its complicated software with lots of edge cases, so for instance not re-inventing a HTTP parser is a great idea. Now if only we all shared as developers this sentiment in terms of upstreaming contributions back too. The more we contribute and share with each other the more productive we can be. Of course, sometimes package managers do terrible jobs at ensuring some sort of base quality, around security or otherwise, and thats never great. So its important to be aware of the trade offs you make when you source your dependencies. By no means does this excuse developers from not understanding their dependency tree either. Its really the opposite.
• LockAndLol 6y ago
  C/C++ suffers from severe wheel reinvention due to a lack of package management, but do you see people cautioning the use of programs written in that language for that reason? The STL package contains lots of things, but most of them are pretty ugly to use and quite a few of them are quite complicated to use because they are generalized for as many use cases as possible. There are so many authors out there that decide they can do memory management manually, write their own thread or process pools, implement sorting in a different way, rewrite basic algorithms for list operations like mapping, filtering, zipping, reducing... Simply counting the number of dependencies isn't a great indicator for dependency bloat. There are extremes on both ends : no deps --> I know everything better and reimplemented the world, and thousands of deps --> I put a one liner in a package ma! One should not judge too quickly.
• skohan 6y ago
  I also wonder if Hyper is really the right tool for the job here. When I was looking into HTTP/S crates, I decided against Hyper because it seemed to require bringing in a runtime for HTTPS, and the async nature of hyper did not seem necessary for my totally synchronous CLI tool. For something like CURL it seems like you would just want the leanest, simplest synchronous HTTP implementation possible
  • tele_ski 6y ago
    The curl easy api perform call is just a blocking wrapper around the curl multi async interface afaik, so I actually think it makes sense.
  • ordu 6y ago
    I think that it is the right tool. 1. CURL without https seems insufficient nowadays. 2. CURL could be improved by running multiple downloads at once. I'm not sure that curl command line utility could do it, but certainly libcurl.so has this ability, it allows client code to work with multiple connections. 3. Any application having UI could benefit from async: input/output and main task are async by nature. For example, curl might want to show progress/status to a terminal despite of a stalled connection. So maybe Hyper is too much of a code for a curl, but it is arguable that it is not.
    • skohan 6y ago
      1. So as to your first point, I totally agree CURL needs to support HTTPS. My point is that Hyper needs a runtime for HTTPS, and it doesn't necessarily make sense for CURL to have a runtime. 2. I'm not sure that CURL should necessarily support multiple concurrent downloads. It could also be argued it's more UNIX-y to make it just do one thing and allow the caller to run multiple CURL processes at the same time 3. You wouldn't necessarily need async to achieve these goals. You could easily have a synchronous http implementation which allows for printing to the console between receiving chunks of data from the network. And if you really didn't want blocking to have a "spinning" activity indicator or something, you could still achieve it with threads. I think ultimately you'd have to decide based on the relative cost of including an entire runtime vs. just launching a second thread.