4 ms·

> And yet we do want all these packages, because they solve specific problems in an optimized manner, and we do want anyone to be able to publish packages. Do

by klabb3 4y ago

> And yet we do want all these packages, because they solve specific problems in an optimized manner, and we do want anyone to be able to publish packages. Do we want all those packages or do we want the functionality of them? Dependency hell happens for reasons of deficient first-party support. Notably, languages that lack a sufficient standard library and a blessed toolchain should be considered a pre-existing condition to this decease. I get that language maintainers have legit reasons to exclude something like HTTP from std, but there has to be some middle ground here. For instance, Golang provides experimental packages that have high quality but a lower level of support. To me, this is a win. It centers the community in a common direction, and delivers real world value in the meantime, with minimal maintenance upkeep compared to the 100s of packages we see in some ecosystems like JS and to a lesser extent Rust.

p-e-w 4y ago
Python has the best or second-best standard library in existence, Go being its only competitor for that title. It still contains only a tiny fraction of the functionality needed by any large project. The world is just too complex for a standard library to ever realistically cover a meaningful portion of the problem space. We all need tensor algebra, video decoding, cutting-edge network protocols, compression, cryptography, dozens of data exchange formats, syscall bindings for three or more platforms, fuzzing, containers, and I don't know what else. This isn't going to all fit into a standard library. "Dependency hell" is here to stay.
- TylerE 4y ago
  Java?
  - v3ss0n 4y ago
    Log4j?
    - richbell 4y ago
      That's a third-party dependency, not a part of the standard library.
    - fbdab103 4y ago
      Never thought I would shrill for Java, but that vulnerability could exist in any language. That it was such a widespread issue only goes to show how widely deployed Java is for server software.
      - GrayShade 4y ago
        No, it wouldn't. Java is the only ecosystem where dynamically loading code by default from random web servers is considered to be a feature. Probably a legacy of its early decade, when everybody was dreaming of this.
      - rfoo 4y ago
        Java is unique as the culture encourages doing magic convention over configuration by reflection AND it-just-works serialization, which leads to a new bug class: unsafe deserialization except that the deserialization happens in the library by default (e.g. fastjson, jackson, tons of struts2 bugs, etc) and propagates along the supply chain. Libraries refuse to shift away from this culture thing, and decided to just blacklist the known exploit path while keeping the vulnerability intact. It's not that they do this for backward-compatibility, it's new libraries still being designed this way. The log4j 2 bug is slightly different, but IMO as a design issue it has root in the same culture thing described above.
        josefx 4y ago
        > Libraries refuse to shift away from this culture thing, and decided to just blacklist the known exploit path while keeping the vulnerability intact. Log4j 2 refused to shift away from it since they are paid to add that kind of bloat, yes this wasn't an accident caused by a bad culture, this was a feature someone requested. As far as I understood they are completely separate from the original log4j and it is just another, mostly compatible logging implementation. > Java is unique as the culture encourages doing magic convention over configuration by reflection AND it-just-works serialization As opposed to what? Python? The language that culturally refuses to fix the GIL and as workaround provides multiprocessing, which requires that everything is serialized? Also I may not have seen many Python code bases, but from what I have seen ClassLoader/Import abuse is alive in both languages.
  - speedgoose 4y ago
    Let’s start the year with a programming language debate! In my opinion, Java’s standard library is good, but not as good as the one in Python and then Golang. I also prefer the one in C#.
- josefx 4y ago
  > This isn't going to all fit into a standard library. There is a size limit for standard libraries? I think Java introduced modules and its dependency model so you could package only what your software needed.
  - TuringTest 4y ago
    > There is a size limit for standard libraries? There is a size limit to what language curators are able to maintain, yes. The idea of the standard library is that it will select components that are general enough to cover a wide range of use cases; therefore the language builders ned time to review every new proposal, and their time is limited. You may dump everything under the sun in the standard library, but then it won't be better than what you get from a library marketplace.
dotancohen 4y ago
I think that Debian and most other Linux distros handle this extremely well. There are thousands, if not tens of thousands, of packages available to install via apt or yum. But most of those packages are packaged by a dedicated maintainer - not any rando. The bar isn't much higher, but there _is_ a bar. Python's (well, pip) practice of letting any vermin with no prior vetting publish is root of the problem in my opinion.
- henearkr 4y ago
  Staging the packages in experimental then testing, sometimes for months, before going to stable, is doing also a lot against this kind of threats. Which leads me to wonder: do we really always want the very latest version or the packages? Any slightly older version will be immune to dependency-poisoning, thanks of the scrutiny of users over several weeks.
  - usrnm 4y ago
    > Which leads me to wonder: do we really always want the very latest version or the packages? Not really, but then someone needs to decide what version to use and when/how this version is updated. For every single dependency. I agree that keeping track of your dependencies and actually managing them may be better engineering but it's a lot of work not many people want to do

by klabb3 4y ago

p-e-w 4y ago

Python has the best or second-best standard library in existence, Go being its only competitor for that title. It still contains only a tiny fraction of the functionality needed by any large project. The world is just too complex for a standard library to ever realistically cover a meaningful portion of the problem space. We all need tensor algebra, video decoding, cutting-edge network protocols, compression, cryptography, dozens of data exchange formats, syscall bindings for three or more platforms, fuzzing, containers, and I don't know what else. This isn't going to all fit into a standard library. "Dependency hell" is here to stay.

TylerE 4y ago
Java?
- v3ss0n 4y ago
  Log4j?
  - richbell 4y ago
    That's a third-party dependency, not a part of the standard library.
  - fbdab103 4y ago
    Never thought I would shrill for Java, but that vulnerability could exist in any language. That it was such a widespread issue only goes to show how widely deployed Java is for server software.
    - GrayShade 4y ago
      No, it wouldn't. Java is the only ecosystem where dynamically loading code by default from random web servers is considered to be a feature. Probably a legacy of its early decade, when everybody was dreaming of this.
    - rfoo 4y ago
      Java is unique as the culture encourages doing magic convention over configuration by reflection AND it-just-works serialization, which leads to a new bug class: unsafe deserialization except that the deserialization happens in the library by default (e.g. fastjson, jackson, tons of struts2 bugs, etc) and propagates along the supply chain. Libraries refuse to shift away from this culture thing, and decided to just blacklist the known exploit path while keeping the vulnerability intact. It's not that they do this for backward-compatibility, it's new libraries still being designed this way. The log4j 2 bug is slightly different, but IMO as a design issue it has root in the same culture thing described above.
      - josefx 4y ago
        > Libraries refuse to shift away from this culture thing, and decided to just blacklist the known exploit path while keeping the vulnerability intact. Log4j 2 refused to shift away from it since they are paid to add that kind of bloat, yes this wasn't an accident caused by a bad culture, this was a feature someone requested. As far as I understood they are completely separate from the original log4j and it is just another, mostly compatible logging implementation. > Java is unique as the culture encourages doing magic convention over configuration by reflection AND it-just-works serialization As opposed to what? Python? The language that culturally refuses to fix the GIL and as workaround provides multiprocessing, which requires that everything is serialized? Also I may not have seen many Python code bases, but from what I have seen ClassLoader/Import abuse is alive in both languages.
- speedgoose 4y ago
  Let’s start the year with a programming language debate! In my opinion, Java’s standard library is good, but not as good as the one in Python and then Golang. I also prefer the one in C#.
josefx 4y ago
> This isn't going to all fit into a standard library. There is a size limit for standard libraries? I think Java introduced modules and its dependency model so you could package only what your software needed.
- TuringTest 4y ago
  > There is a size limit for standard libraries? There is a size limit to what language curators are able to maintain, yes. The idea of the standard library is that it will select components that are general enough to cover a wide range of use cases; therefore the language builders ned time to review every new proposal, and their time is limited. You may dump everything under the sun in the standard library, but then it won't be better than what you get from a library marketplace.

dotancohen 4y ago

I think that Debian and most other Linux distros handle this extremely well. There are thousands, if not tens of thousands, of packages available to install via apt or yum. But most of those packages are packaged by a dedicated maintainer - not any rando. The bar isn't much higher, but there _is_ a bar. Python's (well, pip) practice of letting any vermin with no prior vetting publish is root of the problem in my opinion.

henearkr 4y ago
Staging the packages in experimental then testing, sometimes for months, before going to stable, is doing also a lot against this kind of threats. Which leads me to wonder: do we really always want the very latest version or the packages? Any slightly older version will be immune to dependency-poisoning, thanks of the scrutiny of users over several weeks.
- usrnm 4y ago
  > Which leads me to wonder: do we really always want the very latest version or the packages? Not really, but then someone needs to decide what version to use and when/how this version is updated. For every single dependency. I agree that keeping track of your dependencies and actually managing them may be better engineering but it's a lot of work not many people want to do