5 ms·

The whole process of upgrading dependencies is annoying, mainly because stuff keeps breaking. I currently try to keep everything up-to-date, simply because then

The whole process of upgrading dependencies is annoying, mainly because stuff keeps breaking. I currently try to keep everything up-to-date, simply because then the number of breakages are spread out and easier to deal with, and when critical fixes come through, they're easier to apply. Automation does help with this. But ideally, I'd like the "it it ain't broke don't fix it" approach. It feels like such a waste of time. I would love to see Dependabot go further and track upgrade breakage. Being notified about vulnerabilities is a great service and takes the load off. But too many times, it still breaks. Recent example: Checkstyle security issue, Dependabot cuts a PR. Great. Except almost every minor release [0] has a breaking change. Some libraries are much better than this, and in fact I ended up simply ditching Checkstyle. (It's open source, they don't owe me anything. I get it.) Until we quantify the breakage issue, you don't have enough data to make a decision if upgrading dependencies is worth it outside of a security fix. And in those cases, it's hard to make an argument keeping up to date is the best choice (other than personal philosophy). [0] https://checkstyle.org/releasenotes.html https://checkstyle.org/releasenotes.html

colinbartlett 7y ago
I believe Dependabot quantifies this to some extent with a "Compatibility score": https://dependabot.com/#how-it-works https://dependabot.com/#how-it-works But it can only measure what people have test coverage for, presumably.
- guitarbill 7y ago
  I did see that, but it's pretty opaque (some percentage). And when I click through the badge, it takes me to GitHub's help page [0]. What would be helpful is to link to other public PRs so you might be able to find out how other people solved the issues. Is there any way to see historic compatibility scores to evaluate if it's worth to drop a certain dependency? Edit: For normal dependency updates, compatibility probably more useful. For security fixes, what choice to you have other than to apply it, or spend more time investigating and trying to come up with your own mitigation? [0] https://help.github.com/articles/configuring-automated-security-fixes https://help.github.com/articles/configuring-automated-secur...
sdenton4 7y ago
Automated testing, good source management and relatively short update cycles make all of this much easier... Import the new version in a branch, and see right away which (if any) tests break. Then fix and commit. If a dependency breaks with every minor version change, it's probably worth reevaluating whether it's worth the work, or if the time is better spent replacing the dep with a substitute or some local code.
- guitarbill 7y ago
  I have all that. But it takes time out of my day. I'd like to quantify that vs putting it off and spending 2 days or maybe 2 weeks doing it all at once. Also, right now we have to manually and individually keep track of breakage. It might change mentality and increase empathy in software development when you see how much breakage a change could cause. I don't believe nothing should ever break, but would like to see more awareness of the impact of backwards-incompatible changes.
  - jacques_chester 7y ago
    > I'd like to quantify that vs putting it off and spending 2 days or maybe 2 weeks doing it all at once. That would be possible if the contribution of each change was linear with some amount of fixed overheads, so that batching them was an effective way to improve efficiency. But there are just so many nonlinearities in practice. For example, changes in dependencies can compound such that time-to-upgrade is superlinear in the number of upgrades yet to be applied. Most of the time you get away with it, until one day you discover that you're so far down the road lashing yourself to v1.1.3 that it's very expensive to move to v1.1.5, which you must consume in order to address a critical bug or security vulnerability. > Also, right now we have to manually and individually keep track of breakage. It might change mentality and increase empathy in software development when you see how much breakage a change could cause. I don't believe nothing should ever break, but would like to see more awareness of the impact of backwards-incompatible changes. I agree, and it has been done at some organisations by mandating tools and dependency-management practices which enable downstream testing (eg. Blaze at Google). As a practical matter that is difficult to achieve when crossing a Conway boundary, but I expect Github will start feeding test information back upstream as time goes on.

colinbartlett 7y ago

I believe Dependabot quantifies this to some extent with a "Compatibility score": https://dependabot.com/#how-it-works https://dependabot.com/#how-it-works But it can only measure what people have test coverage for, presumably.

guitarbill 7y ago
I did see that, but it's pretty opaque (some percentage). And when I click through the badge, it takes me to GitHub's help page [0]. What would be helpful is to link to other public PRs so you might be able to find out how other people solved the issues. Is there any way to see historic compatibility scores to evaluate if it's worth to drop a certain dependency? Edit: For normal dependency updates, compatibility probably more useful. For security fixes, what choice to you have other than to apply it, or spend more time investigating and trying to come up with your own mitigation? [0] https://help.github.com/articles/configuring-automated-security-fixes https://help.github.com/articles/configuring-automated-secur...

sdenton4 7y ago

Automated testing, good source management and relatively short update cycles make all of this much easier... Import the new version in a branch, and see right away which (if any) tests break. Then fix and commit. If a dependency breaks with every minor version change, it's probably worth reevaluating whether it's worth the work, or if the time is better spent replacing the dep with a substitute or some local code.

guitarbill 7y ago
I have all that. But it takes time out of my day. I'd like to quantify that vs putting it off and spending 2 days or maybe 2 weeks doing it all at once. Also, right now we have to manually and individually keep track of breakage. It might change mentality and increase empathy in software development when you see how much breakage a change could cause. I don't believe nothing should ever break, but would like to see more awareness of the impact of backwards-incompatible changes.
- jacques_chester 7y ago
  > I'd like to quantify that vs putting it off and spending 2 days or maybe 2 weeks doing it all at once. That would be possible if the contribution of each change was linear with some amount of fixed overheads, so that batching them was an effective way to improve efficiency. But there are just so many nonlinearities in practice. For example, changes in dependencies can compound such that time-to-upgrade is superlinear in the number of upgrades yet to be applied. Most of the time you get away with it, until one day you discover that you're so far down the road lashing yourself to v1.1.3 that it's very expensive to move to v1.1.5, which you must consume in order to address a critical bug or security vulnerability. > Also, right now we have to manually and individually keep track of breakage. It might change mentality and increase empathy in software development when you see how much breakage a change could cause. I don't believe nothing should ever break, but would like to see more awareness of the impact of backwards-incompatible changes. I agree, and it has been done at some organisations by mandating tools and dependency-management practices which enable downstream testing (eg. Blaze at Google). As a practical matter that is difficult to achieve when crossing a Conway boundary, but I expect Github will start feeding test information back upstream as time goes on.