4 ms·

"only uses ~800 dependencies" slightly horrifies me. I was horrified to see how much time I started spending fussing with dependency hell after I moved from .N

"only uses ~800 dependencies" slightly horrifies me. I was horrified to see how much time I started spending fussing with dependency hell after I moved from .NET to Java about 10 years back. And I am currently horrified by how much time I have to spend doing vulnerability updates and fussing with dependency hell in both Java and Python projects nowadays. I think maybe the reason I didn't have this problem to nearly the same extent in .NET is that .NET was relatively late to the automated package management scene. NuGet is relatively young, and, as of the last time I got paid to do .NET work, very few of the projects I worked on had actually adopted it yet. So, at least back then, .NET had a stronger culture of well-focused projects that didn't take on enormous transitive dependency trees. I would also compare this to the recent news about Boeing. Theories abound about why it's gone down the tubes. The one that I find most compelling, though, is that, over the past couple decades, they have focused on moving more of their production out to third-party suppliers, and also cost optimizing their outside supply chain. And that has made their supply chain increasingly difficult to actually manage. The details are different, but in broad strokes it looks a lot like modern software engineering culture regarding supply chain - and some have even argued that this is where Boeing got the idea. Meanwhile, the place I've worked where I found dependency management to be the least annoying - and where we had the fewest problems with quality - was a financial firm that had banned package managers for supply chain security reasons. There's something to be said for code that absolutely will not change unless you explicitly change it. I've heard similar sentiments expressed by friends and acquaintances who work at Google. We did write a lot of stuff for ourselves where others would just import a package, and that was good, too. The in-house implementation would do just what we need, and be held to a higher coding standard. So it was easier to understand, easier to debug, and easier to modify as requirements change. And here's the thing: writing it in the first place is a one-time cost, and one-time costs have good amortization characteristics. The recurring costs of dealing with code that's trying to be everything to everybody can easily be greater in the long run. They generally don't amortize; they compound. Rich Hickey really got me to see how this kind of phenomenon works in his talk "Simple made Easy." Long story short, simple is different from easy. The simpler option tends to look harder up front. But it also tends to be easier in the long run, after you give second-order effects some time to take their toll.

DrScientist 3y ago
Totally agree - I think package managers that make it very easy to pull in huge transitive dependency trees are fundamentally making the wrong thing too easy. First all those dependences you pull in aren't necessarily dependencies because they are done at the package level. If I use one class/function from package A, I may not need any of the package A sub-dependencies - yet these package managers will pull them in recursively down the tree. Second you are trading control for having somebody else manage version dependencies - not sure that saves you time in the end - especially if you took an approach that didn't pull in unncessary dependencies in the first place.
- bunderbunder 3y ago
  And that, in turn, enables a lot of really troublesome feature creep. Last year one of the dependency hell hassles I had to deal with stemmed from MLFlow, a Python package for organizing and collecting results from machine learning experiments, had a hard dependency on LLVM. Why? Because Numba, a JIT compiler for accelerating calculations in Python, uses LLVM. Numba, in turn, is required by SHAP, a model explainability tool. Producing SHAP explanations, in turn, is baked into MLFlow as a kitchen sink feature that is not needed by typical users and could easily be supplied manually or have been included in a separate add-on library. Probably the most upsetting version of this that I encounter is that Apache Spark has all sorts of known vulnerabilities in all the transitive dependencies it pulls in to support every imaginable feature. The Spark project has declined to fix a whole heap of them, on the grounds that Spark doesn't call into the code that has the vulnerability. For a while they even wontfixed @$#% Log4Shell. This is a huge ticking time bomb in my book. Because Java dependency management is such that your Java process typically only gets one version of each JAR, and, if you're using Maven for builds, which version you get is unpredictable. So Spark can cause applications that use both Spark and the affected library, and thought they were using a patched version, to instead get whatever old vulnerable version the Spark project has decided to stick with. Yes, there are lots of clever things you can do to mitigate this problem. But they don't happen by default, and require extra effort and no small measure of specialized Java ecosystem expertise to get right.
  - 3y ago

DrScientist 3y ago

Totally agree - I think package managers that make it very easy to pull in huge transitive dependency trees are fundamentally making the wrong thing too easy. First all those dependences you pull in aren't necessarily dependencies because they are done at the package level. If I use one class/function from package A, I may not need any of the package A sub-dependencies - yet these package managers will pull them in recursively down the tree. Second you are trading control for having somebody else manage version dependencies - not sure that saves you time in the end - especially if you took an approach that didn't pull in unncessary dependencies in the first place.

bunderbunder 3y ago
And that, in turn, enables a lot of really troublesome feature creep. Last year one of the dependency hell hassles I had to deal with stemmed from MLFlow, a Python package for organizing and collecting results from machine learning experiments, had a hard dependency on LLVM. Why? Because Numba, a JIT compiler for accelerating calculations in Python, uses LLVM. Numba, in turn, is required by SHAP, a model explainability tool. Producing SHAP explanations, in turn, is baked into MLFlow as a kitchen sink feature that is not needed by typical users and could easily be supplied manually or have been included in a separate add-on library. Probably the most upsetting version of this that I encounter is that Apache Spark has all sorts of known vulnerabilities in all the transitive dependencies it pulls in to support every imaginable feature. The Spark project has declined to fix a whole heap of them, on the grounds that Spark doesn't call into the code that has the vulnerability. For a while they even wontfixed @$#% Log4Shell. This is a huge ticking time bomb in my book. Because Java dependency management is such that your Java process typically only gets one version of each JAR, and, if you're using Maven for builds, which version you get is unpredictable. So Spark can cause applications that use both Spark and the affected library, and thought they were using a patched version, to instead get whatever old vulnerable version the Spark project has decided to stick with. Yes, there are lots of clever things you can do to mitigate this problem. But they don't happen by default, and require extra effort and no small measure of specialized Java ecosystem expertise to get right.
- 3y ago