3 ms·

> I said code should be auditable for end users. Hundreds (or thousands) of packages isn’t auditable. And what is your idea to fix this? > Deep transitive dep

by Ygg2 3y ago

> I said code should be auditable for end users. Hundreds (or thousands) of packages isn’t auditable. And what is your idea to fix this? > Deep transitive dependencies are problematic and... Gonna stop you right here. Deep transitive dependencies are nothing more than huh, other library find this functionality useful, let's pull it into a library. By not having deep dependencies, you are in no uncertain terms claiming everyone should write their own. Or God forbid copying code into your own project. Imagine for a moment we write a set of perfect auditable libraries. Fuzzed, written to spec, proven correct & fast in theory and practice. Guess what? Everyone and their mother will use it, leading to deep transitive dependencies. The complexity of consumer hardware and software requires that even trivial libraries carry with them many dependencies. The alternative is basically Ludditism. Where do you draw the line at what is or isn't bloat? Everyone only uses 20% of the features but in aggregate, they use 110% (10% from Hyrum's Law) of the features.

yukkuri 3y ago
Yes, even the original link admits their "simple is better" solution is only better in limited use cases. "Our software will be so much better quality if it won't do the things you're paying me to make it do" is a non-starter in the real world.
- whstl 3y ago
  You two are talking with a straw man. Nobody ever said anything of the sort.
whstl 3y ago
Fuzzed, proven correct libraries clearly aren't the problem here. But this is really far away from the bulk of Rust or NPM libraries, and it's very naive to pretend that the situation isn't like that. I already mentioned two examples of things that could have been separate from the main libraries (or at least require flags or separate compilation) and caused global problems: heartbeat extensions from OpenSSL and LDAP/JDNI from Log4J. EDIT: I can mention two other examples that I already talked about in this thread: I often come across simple GraphQL-API-consuming apps or libraries that have dependencies on large frontend libraries, while a simple fetch would suffice. The library is needed for introspection, or for building autocomplete clients, but for pure consumption this is unnecessary. Another example are libraries from authors that are splitting code only to bump their NPM or Cargo metrics, like is-odd, or the ora package that has 15 dependencies and a lot are covered by stdlib.
- Ygg2 3y ago
  > Fuzzed, proven correct libraries clearly aren't the problem here. They aren't. I'm steelmanning your position. "What if there exists a set of well audited libraries". And "what if we have package managers everyone uses". The logical answer is that everyone reuses them, leading to deeply nested transitive dependencies. Hence a contradiction. Deeply nested libraries aren't a problem. > caused global problems: heartbeat extensions from OpenSSL and LDAP/JDNI from Log4J. Already covered that. How do you disambiguate what is a crucial feature and what is overhead? On surface it seems simple, but constraints, call for features and backwards compatibility guarantees makes that something that only seems obvious in hindsight.
  - Ygg2 3y ago
    Here are my underlying assumptions: - Some set of libraries are fully audited. - Some libraries can't be fully specced and audited because their specs are in flux (e.g. games, GUIs, webapps) - Those libraries/apps will limit their maintenance burden by using audited ones - Package managers are useful. As long as the set of audited libraries is large or tends to grow large, or the audited libraries build upon each other, or the non-audited libraries depend on many other libraries, the "deeply nested" (I don't think it's deeply nested as much as many dependencies) hierarchy will be replicated.
    - whstl 3y ago
      Naturally I'm fine with people using even single-function libraries if they're audited, but the problem is that those are the ones who historically are not cared for that much :/ All those assumptions can be all true, but that still doesn't make the overarching assumption right. I would also appreciate if you took a look at my messages with a bit more empathy. I state several times that I'm not advocating for the opposite "NIH syndrome" point of view... this has been implied by you and by the other poster several times, and it's not cool :/
  - whstl 3y ago
    > They aren't. I'm steelmanning your position. Sure, but nevertheless: if this were a reality, sure, it would be perfectly fine! But we don't live in the fantasy world so we have problems. This is why people don't complain about this kind of code being moved to standard libraries: stdlibs they are often heavily audited (with exceptions), So maybe that's a suggestion from me: moving SOME things (NOT ALL) that are traditionally in stdlibs in other languages to the stdlib of your own language. Or another: perhaps force yourself to use primarily audited libraries (parent commenter mentioned NewtonSoft.JSON... this is a good one that is an industry standard). I however still won't bulge that depending on too many mystery libraries is not a good thing :/ > How do you disambiguate what is a crucial feature and what is overhead To keep using the same two examples: if you don't need something for 99% of regular use cases, then it can definitely be optional. But optional doesn't mean the same as "non existent". I'm not advocating removing features or not making it, like another person said. I'm just advocating building software a bit differently. Instead of having 100 features and a lot of dependencies in one root package, have instead 1 base package (with fewer features and fewer dependencies) + N plugin packages (with the remaining features and dependencies). This way you can reduce the blast radius of problematic dependencies without removing features. Naturally you need good DX, the core package must implement something, it can't be just an empty package that doesn't do anything and requires 200 other packages to work. But there are again no silver bullets here.

by Ygg2 3y ago

yukkuri 3y ago

Yes, even the original link admits their "simple is better" solution is only better in limited use cases. "Our software will be so much better quality if it won't do the things you're paying me to make it do" is a non-starter in the real world.

whstl 3y ago
You two are talking with a straw man. Nobody ever said anything of the sort.

whstl 3y ago

Fuzzed, proven correct libraries clearly aren't the problem here. But this is really far away from the bulk of Rust or NPM libraries, and it's very naive to pretend that the situation isn't like that. I already mentioned two examples of things that could have been separate from the main libraries (or at least require flags or separate compilation) and caused global problems: heartbeat extensions from OpenSSL and LDAP/JDNI from Log4J. EDIT: I can mention two other examples that I already talked about in this thread: I often come across simple GraphQL-API-consuming apps or libraries that have dependencies on large frontend libraries, while a simple fetch would suffice. The library is needed for introspection, or for building autocomplete clients, but for pure consumption this is unnecessary. Another example are libraries from authors that are splitting code only to bump their NPM or Cargo metrics, like is-odd, or the ora package that has 15 dependencies and a lot are covered by stdlib.

Ygg2 3y ago
> Fuzzed, proven correct libraries clearly aren't the problem here. They aren't. I'm steelmanning your position. "What if there exists a set of well audited libraries". And "what if we have package managers everyone uses". The logical answer is that everyone reuses them, leading to deeply nested transitive dependencies. Hence a contradiction. Deeply nested libraries aren't a problem. > caused global problems: heartbeat extensions from OpenSSL and LDAP/JDNI from Log4J. Already covered that. How do you disambiguate what is a crucial feature and what is overhead? On surface it seems simple, but constraints, call for features and backwards compatibility guarantees makes that something that only seems obvious in hindsight.
- Ygg2 3y ago
  Here are my underlying assumptions: - Some set of libraries are fully audited. - Some libraries can't be fully specced and audited because their specs are in flux (e.g. games, GUIs, webapps) - Those libraries/apps will limit their maintenance burden by using audited ones - Package managers are useful. As long as the set of audited libraries is large or tends to grow large, or the audited libraries build upon each other, or the non-audited libraries depend on many other libraries, the "deeply nested" (I don't think it's deeply nested as much as many dependencies) hierarchy will be replicated.
  - whstl 3y ago
    Naturally I'm fine with people using even single-function libraries if they're audited, but the problem is that those are the ones who historically are not cared for that much :/ All those assumptions can be all true, but that still doesn't make the overarching assumption right. I would also appreciate if you took a look at my messages with a bit more empathy. I state several times that I'm not advocating for the opposite "NIH syndrome" point of view... this has been implied by you and by the other poster several times, and it's not cool :/
- whstl 3y ago
  > They aren't. I'm steelmanning your position. Sure, but nevertheless: if this were a reality, sure, it would be perfectly fine! But we don't live in the fantasy world so we have problems. This is why people don't complain about this kind of code being moved to standard libraries: stdlibs they are often heavily audited (with exceptions), So maybe that's a suggestion from me: moving SOME things (NOT ALL) that are traditionally in stdlibs in other languages to the stdlib of your own language. Or another: perhaps force yourself to use primarily audited libraries (parent commenter mentioned NewtonSoft.JSON... this is a good one that is an industry standard). I however still won't bulge that depending on too many mystery libraries is not a good thing :/ > How do you disambiguate what is a crucial feature and what is overhead To keep using the same two examples: if you don't need something for 99% of regular use cases, then it can definitely be optional. But optional doesn't mean the same as "non existent". I'm not advocating removing features or not making it, like another person said. I'm just advocating building software a bit differently. Instead of having 100 features and a lot of dependencies in one root package, have instead 1 base package (with fewer features and fewer dependencies) + N plugin packages (with the remaining features and dependencies). This way you can reduce the blast radius of problematic dependencies without removing features. Naturally you need good DX, the core package must implement something, it can't be just an empty package that doesn't do anything and requires 200 other packages to work. But there are again no silver bullets here.