15 ms·

We're Drowning

187 points by mgrayson 4y ago

lmm 4y ago
Funnily enough this becomes much less of a problem once you use a real type system instead of having to write manual tests. Both because whether your code compiles becomes a pretty good (not perfect, just as your test suite is not perfect) indication of whether this dependency upgrade is breaking for you or not, and because those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place.
- gyulai 4y ago
  I don't think a type system will save you. The problem is that, since the introduction of dependencies has become so easy, people have become dependency hogs. Every time you introduce a dependency, you need to weigh the benefits against the cost. But to the person introducing a dependency, at that moment when they make that decision, the cost is just "pip install x". So they think the cost is negligible, but, of course, the true cost doesn't start to show until much later, when it's too late to undo the decision. One slighly masochistic thing I've done for myself is that I don't allow myself to ever do "pip install x". I force myself to (1) download a source distribution for the package, or wheel if it's pure (2) cut the internet connection (3) do "pip install --no-deps" or "setup.py", building from source from the local package (4) see if it works, or if the absence of dependencies causes actual breakage (5) if there's breakage, go to 1, repeating the process with the missing dependency -- when I've spent more time on this process than I think the package is worth, given the benefit it brings to my project, then stop the process and try to live without the dependency. Interestingly: Forcing yourself to build everything from source also tells you a lot about the state of health of a package you're about to introduce as a dependency. Like: Does it depend on something that's just a wrapper around a perl module that hasn't been touched in 15 years; that kind of thing.
  - picsao 4y ago
  - Smaug123 4y ago
    You may be duplicating a lot of work that's already been done in Nixpkgs, by the way.
- reidrac 4y ago
  I disagree. I work with Scala and I can see myself in that post 100%. If things compile and your test pass you might be OK (sure), but the part where "those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place" is not there, unfortunately. If you are using Scala Steward, you'll have a good number of PRs almost every week.
- drewcoo 4y ago
  > instead of having to write manual tests Write automated tests instead! Welcome to the 21st century, human.
keyle 4y ago
It seems there is one of those posts on HN every 6 weeks. And yet there is no solutions besides yolo-live-with-it. I thought it sucks too, I'm in the same boat, but I considered the opposite for a while: what if we didn't have all the code reuse and boatloads of libraries at our fingertips? That would be a TON of code added to our applications and overall - shock full of bugs and lack of optimizations. These libraries, when popular are battle tested, cover much more grounds than we do today (but might need tomorrow) and have been looked at if slow. This ecosystem relies mostly on everyone acting like adults; Libraries to respect semver and not making large, breaking API changes on minor versions; and an ecosystem that is "safe", where bad actors can't pretend to be someone else. Ideally there would be an open "group" of people running penetration tests and security overview of most packages where the collective could donate against their work. Large donations would allow specifying which packages you'd want them to review, and fixes would flow money to the maintainers... But I'm dreaming. We're all building on a gigantic pile of hacks written for someone's portfolio to get their next job, which won't be in the same language and the thing will become abandoned.
- mellavora 4y ago
  > That would be a TON of code added to our applications wait, doesn't using a library also add that code to your application? And if the library has to cover more than your immediate needs, isn't is much much more code added?
  - mojzu 4y ago
    I think they mean that it's code they're not having to write/test/maintain themselves because presumably if you pick a good enough library the quality of code/features are better then what one individual or team could do While there are issues with this approach like the article outlines, personally I believe it to be better then the alternative of countless developers reimplementing the same feature sets over and over because honestly that just seems like a waste of human time and talent
    - deleted 4y ago
      [deleted]
irrational 4y ago
> You’ve probably seen run-of-the-mill web applications with hundreds of direct dependencies This is one of the reasons I like working on legacy web apps. Other than maybe jquery, they typically don’t have any dependencies. It is so easy to follow the code logic because it is all right there and nothing is hidden 10 layers deep in dependency hell.
- Semaphor 4y ago
  I work on a legacy webapp (created 1996, updated to a more modern codebase in 2001), we even have closed source dependencies.
throwawaaarrgh 4y ago
The solution is to do more with less. Devs are taking advantage of a wealth of abstraction and being buried under it. But they actually don't need it. You do not need some 3rd party library to write a command line interface for your python code. You do not need three different projects just to maintain your list of installed packages or virtual environment. And you don't need a package to wrap around loading environment variables. Just learn how to do these things in a simpler way, without the extra deps. Will there be some more boilerplate code? Yes. But it will be your code, and it will be so simple that you will learn to write it very well. In addition: make your code loosely coupled and composeable, and overload existing code with additional functions when feasible, rather than starting a new project from scratch. Press your company to let you contribute back to OSS, as this lets somebody else do more of the maintenance, and has the side effect of reducing the total number of software projects. (It immediately saves you money and time. Kind of stupid that more of us don't do this already.) Simplifying is hard. But when you get good at it, you find you're much more buoyant.
- chasd00 4y ago
  > Simplifying is hard. But when you get good at it, you find you're much more buoyant. it's not finished when there's nothing left to add, it's finished when there's nothing left to take away.
  - Supermancho 4y ago
    Code can always be improved. How you evaluate tradeoffs both varies between individuals and your own views from day to day. Code is never finished, until it's forgotten or feared.
- drewcoo 4y ago
  > You do not need some 3rd party library to write a command line interface for your python code. > Just learn how to do these things in a simpler way, without the extra deps. That's known as reinventing the wheel. Commonly espoused by people with NIH syndrome. https://www.techopedia.com/definition/3848/not-invented-here-syndrome-nihs https://www.techopedia.com/definition/3848/not-invented-here...
Katie69 4y ago
[dead]
jmillikin 4y ago
In the "old" days ("old" being .. what, ten years ago?) packages were released as libraries, which bundled large amounts of functionality into a single atomic version. You could depend on OpenSSL for all your crypto code, GLib for argument parsing or atomics or UUIDs, libiconv for character encoding. There was some expectation (usually met) that the developers of these libraries would have some sort of basic testing strategy, and that they would try to maintain API compatibility over long periods of time. A large program might have less than a dozen external dependencies, and they released every couples months at most, so keeping up to date was easy. And most of the time you didn't even need to update unless someone discovered a security vulnerability, which was not that common despite all the libraries being written in C or C++. The approach of NPM (etc) is fundamentally different in that it's considered normal for a project to have hundreds or thousands of dependencies. To manage the update cadence, the users of these package repositories try to invent new schemes for version numbers -- or write quirky gif-laden blog posts about how they burn the monthly budget of a small country on testing in CI. But the core problem is that they've locked themselves into a stupid and unsustainable paradigm. You can't do the sort of software development that involves writing code if you've decided that your job is to haphazardly glue together other people's code, especially when your're willing to depend on packages written by anything and anyone. -- I honestly think a lot of the problem of over-dependency comes from the new package managers that make it too easy to depend on stuff recursively. If NPM required you to type out `npm add-project-dependency left-pad/1.0.0 --checksum sha256:a1b2c3...` then that would have been a strong forcing function against a huge number of tiny packages, because at some point your fingers get tired. For example, in my personal projects I use Bazel as a build system, which makes me a bit more aware of dependency hell because each dependency needs to be registered. There are projects like QEMU that are straightforward to build in Bazel because their dependency tree is bounded, but some of the stuff coming out of JavaScript land is just unavailable to me because I don't have the patience to chase down hundreds of packages to run a "hello world". I've found I avoid certain parts of the Rust ecosystem because they're JS-ish, but there's other parts that have a more C/C++-ish style and those packages work fine for me. Similarly for Python. Go has largely avoided the issue, though I don't have an intuition as to why (different culture?).
- juped 4y ago
gernb 4y ago
I went through dependency hell today. I wanted to start a new react project and someone who I was going to collaborate with said they wanted try storybook. Ok, follow the tutorial. It immediately spits out all these warnings warning " > @testing-library/user-event@13.5.0" has unmet peer dependency "@testing-library/dom@>=7.21.4". warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-syntax-flow@^7.14.5". warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-transform-react-jsx@^7.14.9". warning "@storybook/builder-webpack5 > fork-ts-checker-webpack-plugin@6.5.2" has unmet peer dependency "typescript@>= 2.7". warning "@storybook/addon-essentials > @storybook/addon-docs > @babel/plugin-transform-react-jsx@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0". warning "react-scripts > eslint-config-react-app > @typescript-eslint/eslint-plugin > tsutils@3.21.0" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta". warning "@storybook/addon-actions > react-inspector@5.1.1" has incorrect peer dependency "react@^16.8.4 || ^17.0.0". warning " > @storybook/addon-essentials@6.5.7" has unmet peer dependency "@babel/core@^7.9.6". warning "@storybook/addon-essentials > @storybook/addon-docs > @mdx-js/react@1.6.22" has incorrect peer dependency "react@^16.13.1 || ^17.0.0". warning "@storybook/addon-interactions > @devtools-ds/object-inspector > @devtools-ds/themes > @design-systems/utils@2.12.0" has unmet peer dependency "@types/react@". warning " > @storybook/preset-create-react-app@4.1.2" has unmet peer dependency "@babel/core@". warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin@1.0.2-canary.6.9d540b91e815f8fc2f8829189deb00553559ff63.0" has unmet peer dependency "typescript@>= 3.x". warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin > react-docgen-typescript@2.2.2" has unmet peer dependency "typescript@>= 4.3.x". warning " > @storybook/react@6.5.7" has unmet peer dependency "require-from-string@^2.0.2". warning "@storybook/react > @babel/preset-flow@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0". warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1". warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react-dom@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1". So that failed IMO. There should be zero warnings! A pile of warnings has no meaning because no will notice when it goes from 18 to 19 warnings. I ended up aborting that and trying other things. It's been 10hrs now and I still have nothing working. Every path has lead to some kind of out dated article and/or dependency issue.
dvh 4y ago
The thing I hate the most is when some JavaScript library uses some obscure build system (or any build system really) but on GitHub they don't publish the built version. What could have been simple download now turns into several hours exercise in trying to understand and make work yet another build system, downloading intermediate bullshit from random websites. Just publish the damn library, thanks.
- cudgy 4y ago
  True, but then how do you know the library was built with the code in the repository? This could introduce a security risk.
  - rini17 4y ago
    All these build dependencies don't introduce security risk?
    - cudgy 4y ago
      Yes, they do. However, accepting pre-built binaries introduces even more risk.
      - rini17 4y ago
        Any basis for your assertion please? Binary infected with malicious payload is more likely to be detected by antivirus or by manual checking of the signature/checksum if user cares to. Infected build system? In case of linux distributions, there are maintainers and packagers responsible for their source and binaries. In case of javascript, does anyone care?
sinuhe69 4y ago
For simple, once used projects, I don’t concern much with these aspects. But for a big projects, choosing your framework and dependency is always a big concern and should do it carefully. Investigate all your dependency is a must, not just for the security aspect. And when you do it, you will convince yourself less is more and your team will re-create many things just to reduce the dependency.
amelius 4y ago
I'm personally amazed that we see so few security issues with all the dependency chains that we build on.
- walderf 4y ago
  that doesn't mean they aren't there.
fmajid 4y ago
Read also this excellent article by Russ Cox (of Go fame) on the issue: https://research.swtch.com/deps https://research.swtch.com/deps This is at heart an economics problem: how to provide a public good (software security) in the presence of free riders. The way society handles this kind of risk is through insurance, but there are a number of things that need to happen to enable a sustainable economic basis to fund the necessary dependency-vetting to happen. Edit: I wrote up my thoughts on the way forward here: https://blog.majid.info/supply-chain-vetting/ https://blog.majid.info/supply-chain-vetting/
bob1029 4y ago
If you are really struggling, I'd suggest trying out the latest .NET before abandoning all hope. I was recently able to build a 3d software rasterizer and streaming web service using .NET6 and I did not have to consume a single 3rd party dependency. Even things like SIMD-optimized projection matrix calculations can be found in the box (System.Numerics). There is obviously the Microsoft aspect to this equation, but I feel like you should at least try the porridge before throwing it away. The empire has a hell of a Death Star these days.
- ajuc 4y ago
  Most languages were like that when they were new. Famously Python. It doesn't last.
  - Deukhoofd 4y ago
    C# is 22 years old.
  - andyp-kw 4y ago
    C# is old enough to drive. I don't think any languages are feature-full when they're new. It takes time for the ecosystem to develop.
- trinovantes 4y ago
  Did you need to build a GUI? It feels like Microsoft invents some new Windows UI library every 5 years or so and deprecates the previous one. The last I used .NET, it was WPF but I hear WinUI is the hot thing now.
  - pjc50 4y ago
    Yes, I was wondering where they were rasterizing to. Although of course the most long-lived way is to just P/Invoke straight into user32.dll to bring up a window. (The last time I wrote a 3D rasterizer was in 1996, and I had to type my dependencies in from paper into Borland Turbo C)
black_13 4y ago
tedk-42 4y ago
We're not drowning. The boat was not made to be waterproof and yet we think it is. Any software you build, like everything else needs maintenance. It's a fallacy to think that just because math is correct (boolean algebra) that your code will tick along fine indefinitely.
- andyp-kw 4y ago
  This is a good point. However some ecosystems and languages might require less maintenance than others.
openfuture 4y ago
How about building on solid ground for once? Datalisp.is the thing I keep posting here and it keeps getting overlooked but a solid foundation means longer lasting structures, just look at the 3-4-5 triangle; it still has a right angle.
prepend 4y ago
Who’s drowning? I feel better than ever. This is a great problem compared to the old way of stuff being out of date and it being a giant pain to use libraries and stuff. Also (knock wood) I haven’t experienced any dependency related outages and have been able to run Java, ruby, Python, and javascript stuff for many years without problems. When I need reproducibility I pin to specific versions. And I vet what packages I use and don’t just randomly grab whatever is in the stackoverflow copy and paste.
imwillofficial 4y ago
This type of overwhelming amounts of info has made me focus my efforts more and narrow my focus
joeyguerra 4y ago
Not me.
tobyhinloopen 4y ago
Stop using NodeJS. Solved.
- mindaslab 4y ago
  agreed 100%
- codegeek 4y ago
  Not that simple. You also have to stop using some popular frameworks/libraries even in CSS. For example, Tailwind is the hotness these days. I like it but in production, they recommend installing using npm and not CDN. Yes it is for performance and size reasons but still.
spion 4y ago
We're really bad at code reuse.
commandlinefan 4y ago
Another downside of the deluge of dependencies that I don't see discussed often enough is bloat. If you import something that imports something else that imports something else ad nauseam, you're importing a lot of stuff you're not actually using and won't ever actually need. Tree shaking was supposed to help with that, but it barely moves the scale. The author says that "Reusing external code is a huge advantage, and when everyone does it, it’s also a competitive necessity", but when I've taken it upon myself to "reinvent" something that there's already a moderately decent solution for, I've found that it really wasn't that much extra effort and the end result was insanely more flexible in the ways that I needed it to be than the supposedly off-the-shelf alternative.
forgotmypw17 4y ago
Every library is code I have to maintain without even knowing what's inside. I only use libraries for something I absolutely can't write myself, like PGP. I also avoid languages and technologies which introduce breaking changes. e.g. Python will never be a top-level language in my project, only an optional add-on for scripting, with failure checks.
shp0ngle 4y ago
ahh yeah the good old days of getting a library from SourceForge
fleddr 4y ago
It's basically a batteries not included problem. Say you create a new React app. The fact that you do is an indicator that the core tech (web tech in this case) does not satisfy your needs out of the box. So you grab a framework from "user land", one of many. React though is not self-contained. To even get it to build anything, it depends on various other projects that each have deep dependencies. Webpack, linters, a choice of CSS framework, etc. At this point, you app itself has zero functional dependencies. You haven't even produced a hello world, all of this stuff is needed just to get it to do anything at all. As you'll then start to build actual functionality, you'll notice that React doesn't have facilities or an opinion on the most basic of stuff. Loading data, managing forms, standardized UI, the essentials are not there. So you go for robust/popular libraries for each of those gaps, which in turn have their own dependency trees. This way even the simplest of apps require thousands of dependencies. It's not due to developers too lazy to write "leftpad". It's a tech stack issue where nothing is standardized or included out of the box.
mikkergp 4y ago
Isn't this the consequence of a large, free, open, diverse system? I remember the constant complaints about their being "too many javascript" frameworks a few years ago, and how their are always new ones. I'm less concerned with the moral or practical considerations here, but can this really be a bad thing? Or, perhaps more to the point, isn't it a sign of a healthy ecosystem? It seems as the world gets more interconnected this problem of "too much"; it's an interesting one to solve... And I'm not saying we shouldn't, but I guess I'm curious about the philosophical question of complaining about complexity in a decentralized public ecosystem. Is there a solution to the ecosystem as a whole? Or will sort of curated subgroups of the ecosystem be birthed that attempt to create a walled garden. It doesn't seem possible to create a shared walled garden without stifling innovation, becoming more closed, all the things that I think most open source communities don't want to do.
electroly 4y ago
I lightheartedly object to C# being mentioned in the same breath as the rest. I've been writing Windows Forms applications for 21 years. Most C# applications have fewer than 10 dependencies, and they tend not to have their own transitive dependencies. It's bliss. We just really don't have the same level of problems in this ecosystem.
- sdflhasjd 4y ago
  Yes, having originally been a C# developer and now diversified and had some experience in other tech, I realised that we are spoiled by one of the most robust standard library and suite of first party libraries that nothing else I've ever used has stood up to it.
  - pjmlp 4y ago
    Java is the only ecosystem on comparable level, hence why both of them are my daily tools.
- vikingerik 4y ago
  I have to disagree, C# can be as bad too. I work on a medium-sized (~500k loc) C# codebase, and we get tangled up in dependency chains frequently. It's a web app with a number of integrations and API calls for things like Google/Facebook/Bing ads that each use some library from them, and other stuff like cloud storage calls and FTP file deliveries. They all tend to bring in more dependencies, like Newtonsoft Json parsing, Excel reader/writer classes, HTTPS and certificate management libraries, log4net logging. I spend a non-trivial amount of time sorting out version conflicts among these, even with the Nuget package manager doing most of the work.
  - andyp-kw 4y ago
    I don't understand how any of the libraries you mentioned would cause dependency issues unless you're referencing them in a weird way. Nuget packages are usually compiled and self contained with their dependencies.
OmarIsmail 4y ago
I'm going through this right now. At dayjob we have a lot of internal dependencies and libraries that are created sufficiently far from me that I treat them as third party and when things go wrong (which they do a lot) it's a nightmare to debug and work through. Easily half a day gone. So for sideproject I'm going mostly bespoke and using very little 3rd party libraries. The result of that is there's sooooooo much boiler plate to write and it's very boring and it's hard to be motivated and productive to get through it. What's keeping me going is that once the boilerplate is done - it's mostly the foundational level of data queries + endpoints - then I shouldn't really have to touch that stuff again. My conclusion is that there's no great answer. Using lots of libraries is like riding a horse. It'll start moving right away but you don't have full control and you gotta find the right way to get it to do what you want, and may have to fight it if it really doesn't want to. Vs building most things yourself is like building a car. You don't go anywhere for a long time as you're building it, but once you've built it you can move very quickly and if anything goes wrong you can easily pop the hood and fix/change things up. I think for long lived projects it's better to build the car, but also be disciplined in writing very good code + documentation. The initial build out sucks hard but that pain will get amortized over a very long future and ultimately be worth it. And for absolutely required dependencies go with paid services. Specifically paid services where the company's primary focus _is_ providing that service. They are financially and existentially motivated to give good service, and are usually good about keeping backwards compatibility while usually staying on top of security/modernization upgrades without requiring work on your own end. A managed database is a good example of this kind of paid dependency.
mperham 4y ago
This path was paved when NPM and Node.js decided that lots of small dependencies are a good thing. All of this damage and JavaScript's reputation as a maintenance nightmare is a result of that terrible, terrible decision.

187 points by mgrayson 4y ago

lmm 4y ago

Funnily enough this becomes much less of a problem once you use a real type system instead of having to write manual tests. Both because whether your code compiles becomes a pretty good (not perfect, just as your test suite is not perfect) indication of whether this dependency upgrade is breaking for you or not, and because those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place.

gyulai 4y ago
I don't think a type system will save you. The problem is that, since the introduction of dependencies has become so easy, people have become dependency hogs. Every time you introduce a dependency, you need to weigh the benefits against the cost. But to the person introducing a dependency, at that moment when they make that decision, the cost is just "pip install x". So they think the cost is negligible, but, of course, the true cost doesn't start to show until much later, when it's too late to undo the decision. One slighly masochistic thing I've done for myself is that I don't allow myself to ever do "pip install x". I force myself to (1) download a source distribution for the package, or wheel if it's pure (2) cut the internet connection (3) do "pip install --no-deps" or "setup.py", building from source from the local package (4) see if it works, or if the absence of dependencies causes actual breakage (5) if there's breakage, go to 1, repeating the process with the missing dependency -- when I've spent more time on this process than I think the package is worth, given the benefit it brings to my project, then stop the process and try to live without the dependency. Interestingly: Forcing yourself to build everything from source also tells you a lot about the state of health of a package you're about to introduce as a dependency. Like: Does it depend on something that's just a wrapper around a perl module that hasn't been touched in 15 years; that kind of thing.
- picsao 4y ago
- Smaug123 4y ago
  You may be duplicating a lot of work that's already been done in Nixpkgs, by the way.
reidrac 4y ago
I disagree. I work with Scala and I can see myself in that post 100%. If things compile and your test pass you might be OK (sure), but the part where "those ecosystems tend to have a lot less need for new versions to fix things or change behaviour in the first place" is not there, unfortunately. If you are using Scala Steward, you'll have a good number of PRs almost every week.
drewcoo 4y ago
> instead of having to write manual tests Write automated tests instead! Welcome to the 21st century, human.

keyle 4y ago

It seems there is one of those posts on HN every 6 weeks. And yet there is no solutions besides yolo-live-with-it. I thought it sucks too, I'm in the same boat, but I considered the opposite for a while: what if we didn't have all the code reuse and boatloads of libraries at our fingertips? That would be a TON of code added to our applications and overall - shock full of bugs and lack of optimizations. These libraries, when popular are battle tested, cover much more grounds than we do today (but might need tomorrow) and have been looked at if slow. This ecosystem relies mostly on everyone acting like adults; Libraries to respect semver and not making large, breaking API changes on minor versions; and an ecosystem that is "safe", where bad actors can't pretend to be someone else. Ideally there would be an open "group" of people running penetration tests and security overview of most packages where the collective could donate against their work. Large donations would allow specifying which packages you'd want them to review, and fixes would flow money to the maintainers... But I'm dreaming. We're all building on a gigantic pile of hacks written for someone's portfolio to get their next job, which won't be in the same language and the thing will become abandoned.

mellavora 4y ago
> That would be a TON of code added to our applications wait, doesn't using a library also add that code to your application? And if the library has to cover more than your immediate needs, isn't is much much more code added?
- mojzu 4y ago
  I think they mean that it's code they're not having to write/test/maintain themselves because presumably if you pick a good enough library the quality of code/features are better then what one individual or team could do While there are issues with this approach like the article outlines, personally I believe it to be better then the alternative of countless developers reimplementing the same feature sets over and over because honestly that just seems like a waste of human time and talent
  - deleted 4y ago
    [deleted]

irrational 4y ago

> You’ve probably seen run-of-the-mill web applications with hundreds of direct dependencies This is one of the reasons I like working on legacy web apps. Other than maybe jquery, they typically don’t have any dependencies. It is so easy to follow the code logic because it is all right there and nothing is hidden 10 layers deep in dependency hell.

Semaphor 4y ago
I work on a legacy webapp (created 1996, updated to a more modern codebase in 2001), we even have closed source dependencies.

throwawaaarrgh 4y ago

The solution is to do more with less. Devs are taking advantage of a wealth of abstraction and being buried under it. But they actually don't need it. You do not need some 3rd party library to write a command line interface for your python code. You do not need three different projects just to maintain your list of installed packages or virtual environment. And you don't need a package to wrap around loading environment variables. Just learn how to do these things in a simpler way, without the extra deps. Will there be some more boilerplate code? Yes. But it will be your code, and it will be so simple that you will learn to write it very well. In addition: make your code loosely coupled and composeable, and overload existing code with additional functions when feasible, rather than starting a new project from scratch. Press your company to let you contribute back to OSS, as this lets somebody else do more of the maintenance, and has the side effect of reducing the total number of software projects. (It immediately saves you money and time. Kind of stupid that more of us don't do this already.) Simplifying is hard. But when you get good at it, you find you're much more buoyant.

chasd00 4y ago
> Simplifying is hard. But when you get good at it, you find you're much more buoyant. it's not finished when there's nothing left to add, it's finished when there's nothing left to take away.
- Supermancho 4y ago
  Code can always be improved. How you evaluate tradeoffs both varies between individuals and your own views from day to day. Code is never finished, until it's forgotten or feared.
drewcoo 4y ago
> You do not need some 3rd party library to write a command line interface for your python code. > Just learn how to do these things in a simpler way, without the extra deps. That's known as reinventing the wheel. Commonly espoused by people with NIH syndrome. https://www.techopedia.com/definition/3848/not-invented-here-syndrome-nihs https://www.techopedia.com/definition/3848/not-invented-here...

Katie69 4y ago

[dead]

jmillikin 4y ago

In the "old" days ("old" being .. what, ten years ago?) packages were released as libraries, which bundled large amounts of functionality into a single atomic version. You could depend on OpenSSL for all your crypto code, GLib for argument parsing or atomics or UUIDs, libiconv for character encoding. There was some expectation (usually met) that the developers of these libraries would have some sort of basic testing strategy, and that they would try to maintain API compatibility over long periods of time. A large program might have less than a dozen external dependencies, and they released every couples months at most, so keeping up to date was easy. And most of the time you didn't even need to update unless someone discovered a security vulnerability, which was not that common despite all the libraries being written in C or C++. The approach of NPM (etc) is fundamentally different in that it's considered normal for a project to have hundreds or thousands of dependencies. To manage the update cadence, the users of these package repositories try to invent new schemes for version numbers -- or write quirky gif-laden blog posts about how they burn the monthly budget of a small country on testing in CI. But the core problem is that they've locked themselves into a stupid and unsustainable paradigm. You can't do the sort of software development that involves writing code if you've decided that your job is to haphazardly glue together other people's code, especially when your're willing to depend on packages written by anything and anyone. -- I honestly think a lot of the problem of over-dependency comes from the new package managers that make it too easy to depend on stuff recursively. If NPM required you to type out `npm add-project-dependency left-pad/1.0.0 --checksum sha256:a1b2c3...` then that would have been a strong forcing function against a huge number of tiny packages, because at some point your fingers get tired. For example, in my personal projects I use Bazel as a build system, which makes me a bit more aware of dependency hell because each dependency needs to be registered. There are projects like QEMU that are straightforward to build in Bazel because their dependency tree is bounded, but some of the stuff coming out of JavaScript land is just unavailable to me because I don't have the patience to chase down hundreds of packages to run a "hello world". I've found I avoid certain parts of the Rust ecosystem because they're JS-ish, but there's other parts that have a more C/C++-ish style and those packages work fine for me. Similarly for Python. Go has largely avoided the issue, though I don't have an intuition as to why (different culture?).

juped 4y ago

gernb 4y ago

I went through dependency hell today. I wanted to start a new react project and someone who I was going to collaborate with said they wanted try storybook. Ok, follow the tutorial. It immediately spits out all these warnings warning " > @testing-library/user-event@13.5.0" has unmet peer dependency "@testing-library/dom@>=7.21.4". warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-syntax-flow@^7.14.5". warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-transform-react-jsx@^7.14.9". warning "@storybook/builder-webpack5 > fork-ts-checker-webpack-plugin@6.5.2" has unmet peer dependency "typescript@>= 2.7". warning "@storybook/addon-essentials > @storybook/addon-docs > @babel/plugin-transform-react-jsx@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0". warning "react-scripts > eslint-config-react-app > @typescript-eslint/eslint-plugin > tsutils@3.21.0" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta". warning "@storybook/addon-actions > react-inspector@5.1.1" has incorrect peer dependency "react@^16.8.4 || ^17.0.0". warning " > @storybook/addon-essentials@6.5.7" has unmet peer dependency "@babel/core@^7.9.6". warning "@storybook/addon-essentials > @storybook/addon-docs > @mdx-js/react@1.6.22" has incorrect peer dependency "react@^16.13.1 || ^17.0.0". warning "@storybook/addon-interactions > @devtools-ds/object-inspector > @devtools-ds/themes > @design-systems/utils@2.12.0" has unmet peer dependency "@types/react@". warning " > @storybook/preset-create-react-app@4.1.2" has unmet peer dependency "@babel/core@". warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin@1.0.2-canary.6.9d540b91e815f8fc2f8829189deb00553559ff63.0" has unmet peer dependency "typescript@>= 3.x". warning "@storybook/preset-create-react-app > @storybook/react-docgen-typescript-plugin > react-docgen-typescript@2.2.2" has unmet peer dependency "typescript@>= 4.3.x". warning " > @storybook/react@6.5.7" has unmet peer dependency "require-from-string@^2.0.2". warning "@storybook/react > @babel/preset-flow@7.17.12" has unmet peer dependency "@babel/core@^7.0.0-0". warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1". warning "@storybook/react > react-element-to-jsx-string@14.3.4" has incorrect peer dependency "react-dom@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1". So that failed IMO. There should be zero warnings! A pile of warnings has no meaning because no will notice when it goes from 18 to 19 warnings. I ended up aborting that and trying other things. It's been 10hrs now and I still have nothing working. Every path has lead to some kind of out dated article and/or dependency issue.

dvh 4y ago

The thing I hate the most is when some JavaScript library uses some obscure build system (or any build system really) but on GitHub they don't publish the built version. What could have been simple download now turns into several hours exercise in trying to understand and make work yet another build system, downloading intermediate bullshit from random websites. Just publish the damn library, thanks.

cudgy 4y ago
True, but then how do you know the library was built with the code in the repository? This could introduce a security risk.
- rini17 4y ago
  All these build dependencies don't introduce security risk?
  - cudgy 4y ago
    Yes, they do. However, accepting pre-built binaries introduces even more risk.
    - rini17 4y ago
      Any basis for your assertion please? Binary infected with malicious payload is more likely to be detected by antivirus or by manual checking of the signature/checksum if user cares to. Infected build system? In case of linux distributions, there are maintainers and packagers responsible for their source and binaries. In case of javascript, does anyone care?

sinuhe69 4y ago

For simple, once used projects, I don’t concern much with these aspects. But for a big projects, choosing your framework and dependency is always a big concern and should do it carefully. Investigate all your dependency is a must, not just for the security aspect. And when you do it, you will convince yourself less is more and your team will re-create many things just to reduce the dependency.

amelius 4y ago

I'm personally amazed that we see so few security issues with all the dependency chains that we build on.

walderf 4y ago
that doesn't mean they aren't there.

fmajid 4y ago

Read also this excellent article by Russ Cox (of Go fame) on the issue: https://research.swtch.com/deps https://research.swtch.com/deps This is at heart an economics problem: how to provide a public good (software security) in the presence of free riders. The way society handles this kind of risk is through insurance, but there are a number of things that need to happen to enable a sustainable economic basis to fund the necessary dependency-vetting to happen. Edit: I wrote up my thoughts on the way forward here: https://blog.majid.info/supply-chain-vetting/ https://blog.majid.info/supply-chain-vetting/

bob1029 4y ago

If you are really struggling, I'd suggest trying out the latest .NET before abandoning all hope. I was recently able to build a 3d software rasterizer and streaming web service using .NET6 and I did not have to consume a single 3rd party dependency. Even things like SIMD-optimized projection matrix calculations can be found in the box (System.Numerics). There is obviously the Microsoft aspect to this equation, but I feel like you should at least try the porridge before throwing it away. The empire has a hell of a Death Star these days.

ajuc 4y ago
Most languages were like that when they were new. Famously Python. It doesn't last.
- Deukhoofd 4y ago
  C# is 22 years old.
- andyp-kw 4y ago
  C# is old enough to drive. I don't think any languages are feature-full when they're new. It takes time for the ecosystem to develop.
trinovantes 4y ago
Did you need to build a GUI? It feels like Microsoft invents some new Windows UI library every 5 years or so and deprecates the previous one. The last I used .NET, it was WPF but I hear WinUI is the hot thing now.
- pjc50 4y ago
  Yes, I was wondering where they were rasterizing to. Although of course the most long-lived way is to just P/Invoke straight into user32.dll to bring up a window. (The last time I wrote a 3D rasterizer was in 1996, and I had to type my dependencies in from paper into Borland Turbo C)

black_13 4y ago

tedk-42 4y ago

We're not drowning. The boat was not made to be waterproof and yet we think it is. Any software you build, like everything else needs maintenance. It's a fallacy to think that just because math is correct (boolean algebra) that your code will tick along fine indefinitely.

andyp-kw 4y ago
This is a good point. However some ecosystems and languages might require less maintenance than others.

openfuture 4y ago

How about building on solid ground for once? Datalisp.is the thing I keep posting here and it keeps getting overlooked but a solid foundation means longer lasting structures, just look at the 3-4-5 triangle; it still has a right angle.

prepend 4y ago

Who’s drowning? I feel better than ever. This is a great problem compared to the old way of stuff being out of date and it being a giant pain to use libraries and stuff. Also (knock wood) I haven’t experienced any dependency related outages and have been able to run Java, ruby, Python, and javascript stuff for many years without problems. When I need reproducibility I pin to specific versions. And I vet what packages I use and don’t just randomly grab whatever is in the stackoverflow copy and paste.

imwillofficial 4y ago

This type of overwhelming amounts of info has made me focus my efforts more and narrow my focus

joeyguerra 4y ago

Not me.

tobyhinloopen 4y ago

Stop using NodeJS. Solved.

mindaslab 4y ago
agreed 100%
codegeek 4y ago
Not that simple. You also have to stop using some popular frameworks/libraries even in CSS. For example, Tailwind is the hotness these days. I like it but in production, they recommend installing using npm and not CDN. Yes it is for performance and size reasons but still.

spion 4y ago

We're really bad at code reuse.

commandlinefan 4y ago

Another downside of the deluge of dependencies that I don't see discussed often enough is bloat. If you import something that imports something else that imports something else ad nauseam, you're importing a lot of stuff you're not actually using and won't ever actually need. Tree shaking was supposed to help with that, but it barely moves the scale. The author says that "Reusing external code is a huge advantage, and when everyone does it, it’s also a competitive necessity", but when I've taken it upon myself to "reinvent" something that there's already a moderately decent solution for, I've found that it really wasn't that much extra effort and the end result was insanely more flexible in the ways that I needed it to be than the supposedly off-the-shelf alternative.

forgotmypw17 4y ago

Every library is code I have to maintain without even knowing what's inside. I only use libraries for something I absolutely can't write myself, like PGP. I also avoid languages and technologies which introduce breaking changes. e.g. Python will never be a top-level language in my project, only an optional add-on for scripting, with failure checks.

shp0ngle 4y ago

ahh yeah the good old days of getting a library from SourceForge

fleddr 4y ago

It's basically a batteries not included problem. Say you create a new React app. The fact that you do is an indicator that the core tech (web tech in this case) does not satisfy your needs out of the box. So you grab a framework from "user land", one of many. React though is not self-contained. To even get it to build anything, it depends on various other projects that each have deep dependencies. Webpack, linters, a choice of CSS framework, etc. At this point, you app itself has zero functional dependencies. You haven't even produced a hello world, all of this stuff is needed just to get it to do anything at all. As you'll then start to build actual functionality, you'll notice that React doesn't have facilities or an opinion on the most basic of stuff. Loading data, managing forms, standardized UI, the essentials are not there. So you go for robust/popular libraries for each of those gaps, which in turn have their own dependency trees. This way even the simplest of apps require thousands of dependencies. It's not due to developers too lazy to write "leftpad". It's a tech stack issue where nothing is standardized or included out of the box.

mikkergp 4y ago

Isn't this the consequence of a large, free, open, diverse system? I remember the constant complaints about their being "too many javascript" frameworks a few years ago, and how their are always new ones. I'm less concerned with the moral or practical considerations here, but can this really be a bad thing? Or, perhaps more to the point, isn't it a sign of a healthy ecosystem? It seems as the world gets more interconnected this problem of "too much"; it's an interesting one to solve... And I'm not saying we shouldn't, but I guess I'm curious about the philosophical question of complaining about complexity in a decentralized public ecosystem. Is there a solution to the ecosystem as a whole? Or will sort of curated subgroups of the ecosystem be birthed that attempt to create a walled garden. It doesn't seem possible to create a shared walled garden without stifling innovation, becoming more closed, all the things that I think most open source communities don't want to do.

electroly 4y ago

I lightheartedly object to C# being mentioned in the same breath as the rest. I've been writing Windows Forms applications for 21 years. Most C# applications have fewer than 10 dependencies, and they tend not to have their own transitive dependencies. It's bliss. We just really don't have the same level of problems in this ecosystem.

sdflhasjd 4y ago
Yes, having originally been a C# developer and now diversified and had some experience in other tech, I realised that we are spoiled by one of the most robust standard library and suite of first party libraries that nothing else I've ever used has stood up to it.
- pjmlp 4y ago
  Java is the only ecosystem on comparable level, hence why both of them are my daily tools.
vikingerik 4y ago
I have to disagree, C# can be as bad too. I work on a medium-sized (~500k loc) C# codebase, and we get tangled up in dependency chains frequently. It's a web app with a number of integrations and API calls for things like Google/Facebook/Bing ads that each use some library from them, and other stuff like cloud storage calls and FTP file deliveries. They all tend to bring in more dependencies, like Newtonsoft Json parsing, Excel reader/writer classes, HTTPS and certificate management libraries, log4net logging. I spend a non-trivial amount of time sorting out version conflicts among these, even with the Nuget package manager doing most of the work.
- andyp-kw 4y ago
  I don't understand how any of the libraries you mentioned would cause dependency issues unless you're referencing them in a weird way. Nuget packages are usually compiled and self contained with their dependencies.

OmarIsmail 4y ago

I'm going through this right now. At dayjob we have a lot of internal dependencies and libraries that are created sufficiently far from me that I treat them as third party and when things go wrong (which they do a lot) it's a nightmare to debug and work through. Easily half a day gone. So for sideproject I'm going mostly bespoke and using very little 3rd party libraries. The result of that is there's sooooooo much boiler plate to write and it's very boring and it's hard to be motivated and productive to get through it. What's keeping me going is that once the boilerplate is done - it's mostly the foundational level of data queries + endpoints - then I shouldn't really have to touch that stuff again. My conclusion is that there's no great answer. Using lots of libraries is like riding a horse. It'll start moving right away but you don't have full control and you gotta find the right way to get it to do what you want, and may have to fight it if it really doesn't want to. Vs building most things yourself is like building a car. You don't go anywhere for a long time as you're building it, but once you've built it you can move very quickly and if anything goes wrong you can easily pop the hood and fix/change things up. I think for long lived projects it's better to build the car, but also be disciplined in writing very good code + documentation. The initial build out sucks hard but that pain will get amortized over a very long future and ultimately be worth it. And for absolutely required dependencies go with paid services. Specifically paid services where the company's primary focus _is_ providing that service. They are financially and existentially motivated to give good service, and are usually good about keeping backwards compatibility while usually staying on top of security/modernization upgrades without requiring work on your own end. A managed database is a good example of this kind of paid dependency.

mperham 4y ago

This path was paved when NPM and Node.js decided that lots of small dependencies are a good thing. All of this damage and JavaScript's reputation as a maintenance nightmare is a result of that terrible, terrible decision.