3 ms·
And that, in turn, enables a lot of really troublesome feature creep. Last year one of the dependency hell hassles I had to deal with stemmed from MLFlow, a Py
by bunderbunder 3y ago
And that, in turn, enables a lot of really troublesome feature creep.
Last year one of the dependency hell hassles I had to deal with stemmed from MLFlow, a Python package for organizing and collecting results from machine learning experiments, had a hard dependency on LLVM. Why? Because Numba, a JIT compiler for accelerating calculations in Python, uses LLVM. Numba, in turn, is required by SHAP, a model explainability tool. Producing SHAP explanations, in turn, is baked into MLFlow as a kitchen sink feature that is not needed by typical users and could easily be supplied manually or have been included in a separate add-on library.
Probably the most upsetting version of this that I encounter is that Apache Spark has all sorts of known vulnerabilities in all the transitive dependencies it pulls in to support every imaginable feature. The Spark project has declined to fix a whole heap of them, on the grounds that Spark doesn't call into the code that has the vulnerability. For a while they even wontfixed @$#% Log4Shell. This is a huge ticking time bomb in my book. Because Java dependency management is such that your Java process typically only gets one version of each JAR, and, if you're using Maven for builds, which version you get is unpredictable. So Spark can cause applications that use both Spark and the affected library, and thought they were using a patched version, to instead get whatever old vulnerable version the Spark project has decided to stick with.
Yes, there are lots of clever things you can do to mitigate this problem. But they don't happen by default, and require extra effort and no small measure of specialized Java ecosystem expertise to get right.
- cchance 3y agoGotta say this is one thing i noticed when i started playing with Rust, sure there are creates for everything but there seems to be a pretty heavy effort to minimize deep dependency trees unless really needed, and when it's not the case people tend to avoid the packages like the plague i feel like.
- bunderbunder 3y agoIt's a rare day that I feel sad about doing something the way systems programmers like to do it.