6 ms·

Usually, dependency hell doesn't bite you, until it does. Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-) I recently h

by mastazi 11y ago

Usually, dependency hell doesn't bite you, until it does. Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-) I recently had to rebuild a large RoR app from circa 2011 and it took me longer to solve dependencies issues than to familiarise myself with the code base. Excessive dependencies are a huge anti-pattern and, in our respective developers communities, we should try to circulate the idea that, while it's silly to reinvent the wheel, it's even worse to add unnecessary dependencies.

emidln 11y ago
> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-) This is your fault for expecting free resources to remain free forever. If you care about build reproduction, dedicate resources to maintain a mirror for your dependencies. These are trivial to setup for any module system worth mentioning (and trivial to write if your module system is so new or esoteric that one wasn't already written for you). If you don't want to do this, you have no place to complain when your free resource disappears in the future.
- xeromal 11y ago
  Most of the time this is something you inherit. Not something you wrote.
  - mastazi 11y ago
    Yes that was my case.
  - emidln 11y ago
    A maintenance programmer should be raising to management the risk if they do not have reproducible builds. The issue isn't that the company's software has a dependency. The issue is that the company is taking for granted the generosity of others. If they did not get a reproducible build before, they should attempt to get one as soon as they are aware of the problem. If the package is no longer available, they must now accept the punishment in terms of lost staff time or dollars to work around the lack of the dependency.
    - mastazi 11y ago
      I agree with you, but s..t happens.
- mastazi 11y ago
  I agree. But I find two problems with your proposal: 1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that. 2- As a corollary to the above, sometimes the problem is not the package itself but compatibility among packages. E.g. package A requires version <=2.5 of package B, but package C requires version >= 2.8 of package B. Now I hear you asking "then how did it compile in the frist place?" probably the requirement was for package A v.2.9 and package C latest version, so while A was frozen, C got updated. This kind of problems is not solved by forking on Github, unless you mantain a different fork of each library for each of your project, but that's even more problematic than maintaining dependencies themselves. P.S. At least for once, it wasn't "my fault", I didn't build that app LOL ;-)
  - NeutronBoy 11y ago
    > 1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that. You've precisely identified the trade-off. You basically have three options. You can 1. Maintain a local repo of your dependencies (high effort) 2. No dependencies, include everything as 'first-class' code (lower upfront effort, but v. messy) 3. Rely on third-party repos (easiest, riskiest)
    - mastazi 11y ago
      Yes I agree, yours is a very good summary of the main alternatives at play. They all have pros and cons as you have rightly noted.
  - emidln 11y ago
    This problem is solved by mirror dependencies and pinning the versions. Even against a git repo, pinning to a particular sha is something that is possible. Automatically upgrading versions (i.e. not pinning versions) in a production build is an anti-pattern. These sound like problems incurred due to a previous lack of software engineering rigor. As an industry, when we encounter challenges like this we should be learning how to solve them without reinventing the wheel. Pinning versions and maintaining mirrors of dependencies (whether that's an http caching proxy for npm/pypi/maven/etc or keeping a snapshot of all dependencies in a directory or a filesystem somewhere) is something that any company requiring stability needs to take seriously.
- jestar_jokin 11y ago
  So, in the context of this discussion... you should make use of micro-modules to reduce code duplication, avoid defects, etc. However, don't expect those micro-modules to be maintained or available in the future; so you need to set up your own package cache to be maintained in perpetuity. Or, you can implement the functionality yourself (or copy/paste if the license allows) and avoid the hassle.
  - mastazi 11y ago
    Yeah, I think the takeaway here is something like: "find the right middle ground between reinventing the wheel and dependency hell"
- ksenzee 11y ago
  There's more to dependency hell than "oops, the package disappeared." Try updating one of those dependencies because of a security fix, and finding that it now depends on Gizmo 7.0 when one of your other dependencies requires Gizmo < 6.0.
  - mastazi 11y ago
    Yes that's exactly the kind of issues that I had experienced.
- lucaspiller 11y ago
  I've been in the same situation as OP many times (although in most cases I've been brought in fix someone else's code). In the Ruby ecosystem, library authors didn't really start caring about semantic versioning and backwards compatability until a few years ago. Even finding a changelog circa 2011 was a godsend. I think this was mainly caused by the language itself not caring about those either. 10 years ago upgrading between patch releases of Ruby (MRI) was likely to break something. At least this is one thing JavaScript seems to do better.
UK-AL 11y ago
Why did you have to rebuild it?
- mastazi 11y ago
  Long story... we had the git repo but lost access to any working installation. I had to rebuild a dev vagrant VM first, and later on a production-ish setup on a server.
- mastazi 11y ago
  PS I did not downvote you and I find your question legitimate...
- Bluestrike2 11y ago
  I can't speak for him, but upgrading really old Rails apps can get complicated very quickly. Especially when you're going across multiple major versions and have to deal with significant changes in Rails behavior and broken gems. "Rebuild" might not be the most accurate way to describe the slow, steady incremental approach you're forced to take (you aren't redoing huge swaths of your domain logic, for instance), but it gets the gist across. The same goes for almost any legacy app.
  - UK-AL 11y ago
    Rebuilding for most cases seems silly to me. I've been working on a .net web app, that's been around since 2008. It's been continually evolved, so it's running on the latest MVC framework, uses microservices etc As result it's build up a huge amount of automated tests. The business logic has been built up from experience and is well tested even for odd cases. You throw away a lot of stuff for a rebuild.
    - Bluestrike2 11y ago
      Oh, definitely. It's not something I'd recommend ever doing for the hell of it unless you're a real masochist. Even then, it might be easier if just broke out the floggers. All kidding aside, if the concern is just for continued security patches, there's always Rails LTS [0]. You have to pay to play, but it's cheaper than a security breach or weeks of development time. But if you're dealing with significant performance issues or looking at major feature changes/additions, it might be more effective in the long-term to consider an upgrade. You just need to be aware of just how large a project that can turn out to be. 0. https://railslts.com/ https://railslts.com/
mmanfrin 11y ago
Shouldnt the Gemfile.lock have prevented that? It takes your gemfile and makes a specific-version locked file of those gems.
- mastazi 11y ago
  The gemfile.lock must have been "gitignored" at some point, because it had much older packages than the ones in Gemfile. Background: all we had was a git repo and did not have access to any "living" installation.
overgard 11y ago
> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-) Let's be honest though, in the current trendy javascript ecosystem these people will already be two or three jobs away before the consequences of their decisions become obvious. Most of the stuff built with this is basically disposable.
- adambratt 11y ago
  Not to mention a good chunk of the companies they were writing this type of code for won't exist by then either...
- emodendroket 11y ago
  I never can believe how often frontend developers talk about "you're just going to rebuild it all in 2 years" anyway. I guess it's a good way to keep yourself employed.
lmm 11y ago
I find large dependencies like RoR itself cause a lot more dependency hell than zillions of small dependencies like this one. What kind of dependency hell could possibly happen for a module like this?

by mastazi 11y ago

emidln 11y ago

> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-) This is your fault for expecting free resources to remain free forever. If you care about build reproduction, dedicate resources to maintain a mirror for your dependencies. These are trivial to setup for any module system worth mentioning (and trivial to write if your module system is so new or esoteric that one wasn't already written for you). If you don't want to do this, you have no place to complain when your free resource disappears in the future.

xeromal 11y ago
Most of the time this is something you inherit. Not something you wrote.
- mastazi 11y ago
  Yes that was my case.
- emidln 11y ago
  A maintenance programmer should be raising to management the risk if they do not have reproducible builds. The issue isn't that the company's software has a dependency. The issue is that the company is taking for granted the generosity of others. If they did not get a reproducible build before, they should attempt to get one as soon as they are aware of the problem. If the package is no longer available, they must now accept the punishment in terms of lost staff time or dollars to work around the lack of the dependency.
  - mastazi 11y ago
    I agree with you, but s..t happens.
mastazi 11y ago
I agree. But I find two problems with your proposal: 1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that. 2- As a corollary to the above, sometimes the problem is not the package itself but compatibility among packages. E.g. package A requires version <=2.5 of package B, but package C requires version >= 2.8 of package B. Now I hear you asking "then how did it compile in the frist place?" probably the requirement was for package A v.2.9 and package C latest version, so while A was frozen, C got updated. This kind of problems is not solved by forking on Github, unless you mantain a different fork of each library for each of your project, but that's even more problematic than maintaining dependencies themselves. P.S. At least for once, it wasn't "my fault", I didn't build that app LOL ;-)
- NeutronBoy 11y ago
  > 1- Maintaining a mirror of dependencies can be a non-trivial overhead. In this app that I was working on, the previous devs had forked some gems on github, and then added that specific github repo to the requirements. But they did not do it for every dependency, probably they did not have time/resources to do that. You've precisely identified the trade-off. You basically have three options. You can 1. Maintain a local repo of your dependencies (high effort) 2. No dependencies, include everything as 'first-class' code (lower upfront effort, but v. messy) 3. Rely on third-party repos (easiest, riskiest)
  - mastazi 11y ago
    Yes I agree, yours is a very good summary of the main alternatives at play. They all have pros and cons as you have rightly noted.
- emidln 11y ago
  This problem is solved by mirror dependencies and pinning the versions. Even against a git repo, pinning to a particular sha is something that is possible. Automatically upgrading versions (i.e. not pinning versions) in a production build is an anti-pattern. These sound like problems incurred due to a previous lack of software engineering rigor. As an industry, when we encounter challenges like this we should be learning how to solve them without reinventing the wheel. Pinning versions and maintaining mirrors of dependencies (whether that's an http caching proxy for npm/pypi/maven/etc or keeping a snapshot of all dependencies in a directory or a filesystem somewhere) is something that any company requiring stability needs to take seriously.
jestar_jokin 11y ago
So, in the context of this discussion... you should make use of micro-modules to reduce code duplication, avoid defects, etc. However, don't expect those micro-modules to be maintained or available in the future; so you need to set up your own package cache to be maintained in perpetuity. Or, you can implement the functionality yourself (or copy/paste if the license allows) and avoid the hassle.
- mastazi 11y ago
  Yeah, I think the takeaway here is something like: "find the right middle ground between reinventing the wheel and dependency hell"
ksenzee 11y ago
There's more to dependency hell than "oops, the package disappeared." Try updating one of those dependencies because of a security fix, and finding that it now depends on Gizmo 7.0 when one of your other dependencies requires Gizmo < 6.0.
- mastazi 11y ago
  Yes that's exactly the kind of issues that I had experienced.
lucaspiller 11y ago
I've been in the same situation as OP many times (although in most cases I've been brought in fix someone else's code). In the Ruby ecosystem, library authors didn't really start caring about semantic versioning and backwards compatability until a few years ago. Even finding a changelog circa 2011 was a godsend. I think this was mainly caused by the language itself not caring about those either. 10 years ago upgrading between patch releases of Ruby (MRI) was likely to break something. At least this is one thing JavaScript seems to do better.

UK-AL 11y ago

Why did you have to rebuild it?

mastazi 11y ago
Long story... we had the git repo but lost access to any working installation. I had to rebuild a dev vagrant VM first, and later on a production-ish setup on a server.
mastazi 11y ago
PS I did not downvote you and I find your question legitimate...
Bluestrike2 11y ago
I can't speak for him, but upgrading really old Rails apps can get complicated very quickly. Especially when you're going across multiple major versions and have to deal with significant changes in Rails behavior and broken gems. "Rebuild" might not be the most accurate way to describe the slow, steady incremental approach you're forced to take (you aren't redoing huge swaths of your domain logic, for instance), but it gets the gist across. The same goes for almost any legacy app.
- UK-AL 11y ago
  Rebuilding for most cases seems silly to me. I've been working on a .net web app, that's been around since 2008. It's been continually evolved, so it's running on the latest MVC framework, uses microservices etc As result it's build up a huge amount of automated tests. The business logic has been built up from experience and is well tested even for odd cases. You throw away a lot of stuff for a rebuild.
  - Bluestrike2 11y ago
    Oh, definitely. It's not something I'd recommend ever doing for the hell of it unless you're a real masochist. Even then, it might be easier if just broke out the floggers. All kidding aside, if the concern is just for continued security patches, there's always Rails LTS [0]. You have to pay to play, but it's cheaper than a security breach or weeks of development time. But if you're dealing with significant performance issues or looking at major feature changes/additions, it might be more effective in the long-term to consider an upgrade. You just need to be aware of just how large a project that can turn out to be. 0. https://railslts.com/ https://railslts.com/

mmanfrin 11y ago

Shouldnt the Gemfile.lock have prevented that? It takes your gemfile and makes a specific-version locked file of those gems.

mastazi 11y ago
The gemfile.lock must have been "gitignored" at some point, because it had much older packages than the ones in Gemfile. Background: all we had was a git repo and did not have access to any "living" installation.

overgard 11y ago

> Try to rebuild that thousand-dependencies app in three years from now and you'll see ;-) Let's be honest though, in the current trendy javascript ecosystem these people will already be two or three jobs away before the consequences of their decisions become obvious. Most of the stuff built with this is basically disposable.

adambratt 11y ago
Not to mention a good chunk of the companies they were writing this type of code for won't exist by then either...
emodendroket 11y ago
I never can believe how often frontend developers talk about "you're just going to rebuild it all in 2 years" anyway. I guess it's a good way to keep yourself employed.

lmm 11y ago

I find large dependencies like RoR itself cause a lot more dependency hell than zillions of small dependencies like this one. What kind of dependency hell could possibly happen for a module like this?