8 ms·

Funny thing is, Rust and JS are both given that comment. Seeing how they are different languages, there's not a single language feature that explains this depe

Funny thing is, Rust and JS are both given that comment. Seeing how they are different languages, there's not a single language feature that explains this dependency gluttony. I'd like to open up the option that it is a necessary side-effect of large ecosystems: more devs using the language, more packages published, more quality in small packages. And then, when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. And so, driven by nothing but good will and best practices, since you have a lot of prebuilt "assets" at your disposal ,your project ends up with 100s of deps and everybody complains.

ReactiveJelly 5y ago
> when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. Yes, for instance one of `reqwest`'s dependencies is `base64`. We can argue both that re-inventing base64 in every big library is pointless, and also that a language doesn't need base64 in its stdlib because C and C++ don't have it. There's no right way to cut this cake - Many a language has included something in the standard library, and then struggled to fix a broken interface once everything depended on it. (Like JS picking UTF-16 for strings) Many a language has also failed to include something in the stdlib and caused every application to suffer and re-invent it. (Like C++ not really saying anything about Unicode strings, which means Qt and everyone else has their own little take on it.) And the suffering is only bigger if you make dependencies artificially hard to add. Rust's approach is more flexible, at the cost of building the universe from scratch every time you make a new workspace. Personally I don't mind it. At least it's not like the problems I've heard from Haskell land. Moving from futures to async in Rust was fairly smooth.
- tptacek 5y ago
  Rust should include Base64 in the standard library, and languages should continue to standardize on UTF-8. Doesn't seem hard to work out.
  - burntsushi 5y ago
    You're nitpicking one example. Putting base64 into std makes the higher level comment go from complaining about 100 dependencies to complaining about 99 dependencies. So maybe it's harder to work out than you think.
    - tptacek 5y ago
      I don't have an opinion about the broader dependency problem in Rust. The comment to which I responded was hosting an internal debate about Base64 and UTF-8. I think that debate is easy to resolve: use UTF-8, put Base64 in the stdlib. Do you disagree?
      - burntsushi 5y ago
        The comment you responded to was using base64 as an example. That's why your comment is a nitpick in this conversation. And yeah, I doubt I would support including base64 in std unless there were a compelling reason to do so. I can't think of one. And yes, use UTF-8. Not contesting that.
        tptacek 5y ago
        I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. There might be an upside to that (I have not detected it), but there's an obvious downside. Rust gets a bunch of stuff right that other languages missed. I don't think this one of them. I think Go's strategy of having a pretty-ok version of most of the table-stakes things developers do is probably the right one, just like I think sum types and option/result are better than error types and `nil`. I share your irritation at people who drop into the middle of discussions like these to nitpick random language war arguments, but this is a thread that is actually about Rust dependency hell, and the comment I responded to, I think, sort of diagnoses why Rust has that "problem" (if you think it's a problem --- I do, but whatever). For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
- awsthro00945 5y ago
  >and also that a language doesn't need base64 in its stdlib because C and C++ don't have it I don't follow this logic. If we just wanted to do everything the way C/C++ does it, we'd be using C/C++, and Rust wouldn't exist.
  - pjmlp 5y ago
    True, although my only complain with C++ is the subculture that to this day pushes for C types and unsafe by default. For anything else I don't have any major complaint.
    - secondcoming 5y ago
      who is pushing for 'unsafe by default'?
      - pjmlp 5y ago
        Anyone that uses char * instead of std::array for example.
        yakubin 5y ago
        std::array will instantiate function templates for all used array sizes, lengthen compilation time, and be hard to use across separately-compiled code without putting code into headers. It has its uses, but more often I'd prefer T[N] with bounds-checking, and, in another world, a builtin variably-sized, non-resizable array type; basically a sort of absl::FixedArray[1], but with the semantics of operator[] and at() reversed, so that bounds-checking is the default. Sadly, nothing like that is included in the standard, so won't be a lingua-franca. [1]: <https://github.com/abseil/abseil-cpp/blob/master/absl/container/fixed_array.h https://github.com/abseil/abseil-cpp/blob/master/absl/contai...>
        pjmlp 5y ago
        On Visual Studio it is relatively easy to enable bounds checking for operator[](). https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-use-native-run-time-checks?view=vs-2019 https://docs.microsoft.com/en-us/visualstudio/debugger/how-t... That is the thing, the standard doesn't require bounds checking for operator()[], while it does require for at(), but it doesn't forbid it either, it is up to the implementations.
- hardwaregeek 5y ago
  Yeah in the debate about dependencies, people often frame it as more dependencies == bad and less dependencies == good. Except more/less dependencies is actually more or less explicit dependencies. Fact of the matter is that you need a crazy amount of dependencies to write most modern software. You need ways to handle Unicode, to connect to databases, to write async code, to handle dates, etc. The question is whether that code is going to come from a monolithic Swiss Army Knife of a library (aka a standard library), a bunch of packages, or some code that everybody copies into their codebase. In which case the packages option doesn't seem so bad. At least they're explicitly versioned, tracked for vulnerabilities and can be updated at a granular level.
  - pjmlp 5y ago
    Because when working in large corporations one ends up having to write license checker tools and having humans go through those packages, alongside legal, to validate what is allowed to appear on the final product, and to vendor them on the internal package repository.
mirekrusin 5y ago
True in js/ts world this is a problem. Since couple of years I've been experimenting with more radical fat trimming on dependencies. It works very well on backend code. I belive shallow or no dependencies is the way to go. What seems to be missing is standard library as constellation of well designed modules/packages. I decided to just do it for myself (and anybody who finds it interesting). I don't think it's going to fly high but if somebody is interested the link is here [0]. I'm adding modules every few days. The aim is to have something worthwile to show around the time node v16 goes lts. I have some experience taking care of high stake business critical systems. This is fun side project as a hobby atm. Modules being dropped there are reflection of problems I'm facing during my day-to-day work. You'll see there some non conventional code. Large parts are inspired by functional programming, ocalm and simplicitly in general. Personally I find it exciting how well this kind of approach fits into production projects. The nice thing about this approach is that you can have niche leaf packages/modules as part of standard library (constellation) - if you don't want to use it, it doesn't matter, it wont incur any cost. For example order book module or exchange matching engine - those can simply live there, be useful to people who care and have zero impact to people who don't care at all. I find it to be an interesting difference from builtin, shipped library. Large package ecosystem is a sign of popularity. With time libraries should converge onto shared approaches. Interestingly practice shows wide spread of duplication. I think more effort from big players with wide reach could reduce it. Ie. not me doing stuff but Microsoft or others picking up the ball. [0] https://github.com/preludejs https://github.com/preludejs
codefined 5y ago
Rust and JS both have a very small standard library. This is what unites them in a 'dependency hell' of having to handle thousands of packages for most projects. Other languages that don't face this, e.g. Python, have very large standard libraries which already include a dizzying array of features. E.g. take a look at some of the things you get with standard Python[0]. - sqlite3 (DB-API 2.0 interface for SQLite databases) - bz2 (Support for bzip2 compression) - sunau (Read and write Sun AU files) - netrc (netrc file processing) - curses (terminal handling for character-cell displays) - mailbox (manipulate mailboxes in various formats) [0] https://docs.python.org/3/library/ https://docs.python.org/3/library/
- mrec 5y ago
  There's a downside to Python's batteries-included approach though: https://leancrew.com/all-this/2012/04/where-modules-go-to-die/ https://leancrew.com/all-this/2012/04/where-modules-go-to-di... HN discussion: https://news.ycombinator.com/item?id=3913182 https://news.ycombinator.com/item?id=3913182 The obvious compromise would be to let libraries bake as third-party projects for a good long while until there's a fair amount of consensus that they're the Right Thing To Do, then pull them into the standard library. Java's Joda-Time would be one example.
  - smoldesu 5y ago
    Oracle doesn't own Rust though, there's no real push to onboard these features besides the general curb-cutting efforts for newcomers. For the most part, 'integral' crates like serde and tokio have been community-managed without issue so far, so it doesn't make much sense to pull one open-source developers passion project from them and pass it to another.
_tom_ 5y ago
I don’t think this is a characteristic of languages or a large ecosystems. It’s a side effect of dependency managers making it seem like zero cost to include another library. Including one dependency can end up pulling in an entire ecosystem of connected dependencies, when the perhaps one function in the original dependency. One recent build I did, a small library included a dependency to build its documentation. This was a full-fledged documentation project, which pulled in hundreds more dependencies. It would be interesting to see a display of “cost” in projects indicating how much additional dead weight each individual dependency pulls in. This might lead to slight better decisions on dependencies to choose.
- mrec 5y ago
  > It would be interesting to see a display of “cost” in projects cargo-bloat is kind of along these lines, although it only measures the absolute size of each dependency (as opposed to "dead weight") and has some platform limitations. https://crates.io/crates/cargo-bloat/ https://crates.io/crates/cargo-bloat/
jollybean 5y ago
" it is a necessary side-effect of large ecosystems" It depends a lot. Java has a huge ecosystem, but almost all of the core functionality is provided for you. Like everything there's a 90/10 rule in that 90% of the 'things we need' really amount to only 10% of the total packages out there, so it actually makes sense that the people backing the platform provide for it. Language, compiler, debugger, documentation, tutorials, standard libraries etc. - it's all part of the platform. Looking at it that way, it becomes a little more obvious why even some cool languages don't quite breakthrough.
- xcambar 5y ago
  Java is a good example. As another commenter pointed out, it may relate more to the size of the standard library than to the size of the ecosystem. Which your comment seems to emphasize as well.

ReactiveJelly 5y ago

> when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel. Yes, for instance one of `reqwest`'s dependencies is `base64`. We can argue both that re-inventing base64 in every big library is pointless, and also that a language doesn't need base64 in its stdlib because C and C++ don't have it. There's no right way to cut this cake - Many a language has included something in the standard library, and then struggled to fix a broken interface once everything depended on it. (Like JS picking UTF-16 for strings) Many a language has also failed to include something in the stdlib and caused every application to suffer and re-invent it. (Like C++ not really saying anything about Unicode strings, which means Qt and everyone else has their own little take on it.) And the suffering is only bigger if you make dependencies artificially hard to add. Rust's approach is more flexible, at the cost of building the universe from scratch every time you make a new workspace. Personally I don't mind it. At least it's not like the problems I've heard from Haskell land. Moving from futures to async in Rust was fairly smooth.

tptacek 5y ago
Rust should include Base64 in the standard library, and languages should continue to standardize on UTF-8. Doesn't seem hard to work out.
- burntsushi 5y ago
  You're nitpicking one example. Putting base64 into std makes the higher level comment go from complaining about 100 dependencies to complaining about 99 dependencies. So maybe it's harder to work out than you think.
  - tptacek 5y ago
    I don't have an opinion about the broader dependency problem in Rust. The comment to which I responded was hosting an internal debate about Base64 and UTF-8. I think that debate is easy to resolve: use UTF-8, put Base64 in the stdlib. Do you disagree?
    - burntsushi 5y ago
      The comment you responded to was using base64 as an example. That's why your comment is a nitpick in this conversation. And yeah, I doubt I would support including base64 in std unless there were a compelling reason to do so. I can't think of one. And yes, use UTF-8. Not contesting that.
      - tptacek 5y ago
        I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. There might be an upside to that (I have not detected it), but there's an obvious downside. Rust gets a bunch of stuff right that other languages missed. I don't think this one of them. I think Go's strategy of having a pretty-ok version of most of the table-stakes things developers do is probably the right one, just like I think sum types and option/result are better than error types and `nil`. I share your irritation at people who drop into the middle of discussions like these to nitpick random language war arguments, but this is a thread that is actually about Rust dependency hell, and the comment I responded to, I think, sort of diagnoses why Rust has that "problem" (if you think it's a problem --- I do, but whatever). For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
awsthro00945 5y ago
>and also that a language doesn't need base64 in its stdlib because C and C++ don't have it I don't follow this logic. If we just wanted to do everything the way C/C++ does it, we'd be using C/C++, and Rust wouldn't exist.
- pjmlp 5y ago
  True, although my only complain with C++ is the subculture that to this day pushes for C types and unsafe by default. For anything else I don't have any major complaint.
  - secondcoming 5y ago
    who is pushing for 'unsafe by default'?
    - pjmlp 5y ago
      Anyone that uses char * instead of std::array for example.
      - yakubin 5y ago
        std::array will instantiate function templates for all used array sizes, lengthen compilation time, and be hard to use across separately-compiled code without putting code into headers. It has its uses, but more often I'd prefer T[N] with bounds-checking, and, in another world, a builtin variably-sized, non-resizable array type; basically a sort of absl::FixedArray[1], but with the semantics of operator[] and at() reversed, so that bounds-checking is the default. Sadly, nothing like that is included in the standard, so won't be a lingua-franca. [1]: <https://github.com/abseil/abseil-cpp/blob/master/absl/container/fixed_array.h https://github.com/abseil/abseil-cpp/blob/master/absl/contai...>
        pjmlp 5y ago
        On Visual Studio it is relatively easy to enable bounds checking for operator[](). https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-use-native-run-time-checks?view=vs-2019 https://docs.microsoft.com/en-us/visualstudio/debugger/how-t... That is the thing, the standard doesn't require bounds checking for operator()[], while it does require for at(), but it doesn't forbid it either, it is up to the implementations.
hardwaregeek 5y ago
Yeah in the debate about dependencies, people often frame it as more dependencies == bad and less dependencies == good. Except more/less dependencies is actually more or less explicit dependencies. Fact of the matter is that you need a crazy amount of dependencies to write most modern software. You need ways to handle Unicode, to connect to databases, to write async code, to handle dates, etc. The question is whether that code is going to come from a monolithic Swiss Army Knife of a library (aka a standard library), a bunch of packages, or some code that everybody copies into their codebase. In which case the packages option doesn't seem so bad. At least they're explicitly versioned, tracked for vulnerabilities and can be updated at a granular level.
- pjmlp 5y ago
  Because when working in large corporations one ends up having to write license checker tools and having humans go through those packages, alongside legal, to validate what is allowed to appear on the final product, and to vendor them on the internal package repository.

mirekrusin 5y ago

True in js/ts world this is a problem. Since couple of years I've been experimenting with more radical fat trimming on dependencies. It works very well on backend code. I belive shallow or no dependencies is the way to go. What seems to be missing is standard library as constellation of well designed modules/packages. I decided to just do it for myself (and anybody who finds it interesting). I don't think it's going to fly high but if somebody is interested the link is here [0]. I'm adding modules every few days. The aim is to have something worthwile to show around the time node v16 goes lts. I have some experience taking care of high stake business critical systems. This is fun side project as a hobby atm. Modules being dropped there are reflection of problems I'm facing during my day-to-day work. You'll see there some non conventional code. Large parts are inspired by functional programming, ocalm and simplicitly in general. Personally I find it exciting how well this kind of approach fits into production projects. The nice thing about this approach is that you can have niche leaf packages/modules as part of standard library (constellation) - if you don't want to use it, it doesn't matter, it wont incur any cost. For example order book module or exchange matching engine - those can simply live there, be useful to people who care and have zero impact to people who don't care at all. I find it to be an interesting difference from builtin, shipped library. Large package ecosystem is a sign of popularity. With time libraries should converge onto shared approaches. Interestingly practice shows wide spread of duplication. I think more effort from big players with wide reach could reduce it. Ie. not me doing stuff but Microsoft or others picking up the ball. [0] https://github.com/preludejs https://github.com/preludejs

codefined 5y ago

Rust and JS both have a very small standard library. This is what unites them in a 'dependency hell' of having to handle thousands of packages for most projects. Other languages that don't face this, e.g. Python, have very large standard libraries which already include a dizzying array of features. E.g. take a look at some of the things you get with standard Python[0]. - sqlite3 (DB-API 2.0 interface for SQLite databases) - bz2 (Support for bzip2 compression) - sunau (Read and write Sun AU files) - netrc (netrc file processing) - curses (terminal handling for character-cell displays) - mailbox (manipulate mailboxes in various formats) [0] https://docs.python.org/3/library/ https://docs.python.org/3/library/

mrec 5y ago
There's a downside to Python's batteries-included approach though: https://leancrew.com/all-this/2012/04/where-modules-go-to-die/ https://leancrew.com/all-this/2012/04/where-modules-go-to-di... HN discussion: https://news.ycombinator.com/item?id=3913182 https://news.ycombinator.com/item?id=3913182 The obvious compromise would be to let libraries bake as third-party projects for a good long while until there's a fair amount of consensus that they're the Right Thing To Do, then pull them into the standard library. Java's Joda-Time would be one example.
- smoldesu 5y ago
  Oracle doesn't own Rust though, there's no real push to onboard these features besides the general curb-cutting efforts for newcomers. For the most part, 'integral' crates like serde and tokio have been community-managed without issue so far, so it doesn't make much sense to pull one open-source developers passion project from them and pass it to another.

_tom_ 5y ago

I don’t think this is a characteristic of languages or a large ecosystems. It’s a side effect of dependency managers making it seem like zero cost to include another library. Including one dependency can end up pulling in an entire ecosystem of connected dependencies, when the perhaps one function in the original dependency. One recent build I did, a small library included a dependency to build its documentation. This was a full-fledged documentation project, which pulled in hundreds more dependencies. It would be interesting to see a display of “cost” in projects indicating how much additional dead weight each individual dependency pulls in. This might lead to slight better decisions on dependencies to choose.

mrec 5y ago
> It would be interesting to see a display of “cost” in projects cargo-bloat is kind of along these lines, although it only measures the absolute size of each dependency (as opposed to "dead weight") and has some platform limitations. https://crates.io/crates/cargo-bloat/ https://crates.io/crates/cargo-bloat/

jollybean 5y ago

" it is a necessary side-effect of large ecosystems" It depends a lot. Java has a huge ecosystem, but almost all of the core functionality is provided for you. Like everything there's a 90/10 rule in that 90% of the 'things we need' really amount to only 10% of the total packages out there, so it actually makes sense that the people backing the platform provide for it. Language, compiler, debugger, documentation, tutorials, standard libraries etc. - it's all part of the platform. Looking at it that way, it becomes a little more obvious why even some cool languages don't quite breakthrough.

xcambar 5y ago
Java is a good example. As another commenter pointed out, it may relate more to the size of the standard library than to the size of the ecosystem. Which your comment seems to emphasize as well.