6 ms·

Dependency Drift: A Metric for Software Aging

49 points by colinbartlett 7y ago

jefffoster 7y ago
I agree this is an important metric and a kind of technical debt that's hard to capture. How do you discriminate this from RenovateBot or Dependa-Bot? Both of those will not only track the drift but they'll also generate the PR to fix it.
- colinbartlett 7y ago
  Co-author here, thanks for the input. We use Dependabot on all our projects and love it. I've not seen any kind of numeric metric in Dependabot or any of their competitors. What I really want is a graph that shows, numerically, the drift over time. You can show this to your stakeholders: "See, look! We haven't dedicated time to stack upgrades and now look how far out of date we are creeping." And then, consequently, I want to see that chart burn down to zero as you do stay on top of things with the help of Dependabot or similar.
  - k__ 7y ago
    As a freelancer, this sounds really interesting to me. It's always hard to sell people maintenance.
  - SideburnsOfDoom 7y ago
    This is the libyears metric ( https://libyear.com/ https://libyear.com/ ) tracked over time. Or are you using a different metric that you haven't defined for us?
yourpalkeith 7y ago
A variant of this idea that would be more useful to me personally would be some measure of how actively maintained my dependencies are. It would be nice to know if my long-lived project depends on something that has been abandoned. Details might be tricky, given that some very useful packages simply don't need frequent updates.
- k__ 7y ago
  I like the drift metric more, because some libraries are simply stable and don't need much updates anymore, this doesn't mean they are bad for your system.
guitarbill 7y ago
The whole process of upgrading dependencies is annoying, mainly because stuff keeps breaking. I currently try to keep everything up-to-date, simply because then the number of breakages are spread out and easier to deal with, and when critical fixes come through, they're easier to apply. Automation does help with this. But ideally, I'd like the "it it ain't broke don't fix it" approach. It feels like such a waste of time. I would love to see Dependabot go further and track upgrade breakage. Being notified about vulnerabilities is a great service and takes the load off. But too many times, it still breaks. Recent example: Checkstyle security issue, Dependabot cuts a PR. Great. Except almost every minor release [0] has a breaking change. Some libraries are much better than this, and in fact I ended up simply ditching Checkstyle. (It's open source, they don't owe me anything. I get it.) Until we quantify the breakage issue, you don't have enough data to make a decision if upgrading dependencies is worth it outside of a security fix. And in those cases, it's hard to make an argument keeping up to date is the best choice (other than personal philosophy). [0] https://checkstyle.org/releasenotes.html https://checkstyle.org/releasenotes.html
- colinbartlett 7y ago
  I believe Dependabot quantifies this to some extent with a "Compatibility score": https://dependabot.com/#how-it-works https://dependabot.com/#how-it-works But it can only measure what people have test coverage for, presumably.
  - guitarbill 7y ago
    I did see that, but it's pretty opaque (some percentage). And when I click through the badge, it takes me to GitHub's help page [0]. What would be helpful is to link to other public PRs so you might be able to find out how other people solved the issues. Is there any way to see historic compatibility scores to evaluate if it's worth to drop a certain dependency? Edit: For normal dependency updates, compatibility probably more useful. For security fixes, what choice to you have other than to apply it, or spend more time investigating and trying to come up with your own mitigation? [0] https://help.github.com/articles/configuring-automated-security-fixes https://help.github.com/articles/configuring-automated-secur...
alangibson 7y ago
I think you're on to something here. This is hot topic amongst our teams right now. Dependency maintenance looks to me to be one of those things like code duplication that correlates strongly with a multitude of sins. It's also interesting to see someone else building a tool around a new metric. I'm doing something similar (shameless plug -> https://without.fail/estimate-accuracy https://without.fail/estimate-accuracy) for determining how well estimates correlate with actual development times. I could definitely take some lessons from you because Dependency Drift is way more brandable than Estimate Accuracy.
jacques_chester 7y ago
At Pivotal (now part of VMware, but it's easier to explain this way) we have an increasing number of teams that track "vulnerability budgets" and "legacy budgets", by way of analogy to "error budgets" from SRE approaches.[0] Essentially, a vulnerability budget tracks days since last release for a dependency or system; a legacy budget tracks time left until a dependency or system falls out of support. Our CloudOps team and telemetry teams now plot these for their own systems and customer systems, producing a characteristic sawtooth pattern. I think these are good steps forward, but I see some possible evolutions. One evolution is to combine these into a more general risk framework like FAIR, so that we're able to use a single budget for all these categories of risk. Another is to consider risk exposure as a volume. At any given instant I have a wide range of risks in play, distributed over a wide range of systems. As time evolves these risks come and go. A single score (like the one sketched on the far side of the link) only gives me one axis of risk. I also need to know what systems are currently expressing that risk by the presence or absence of software, giving me a second axis. Finally, these evolve over time, giving a third axis. The sum of volume for any time slice is what ought to be budgeted. Another thing is to make these problems behave less like "big-bang control", where we switch modes drastically when crossing a threshold value. What would be preferred is a greater degree of smoothness. And finally, to make all of this tractable: automation. All the metrics and scorecards and reports and graphs in the world won't make this problem easier. They can at best make it visible and demoralising. [0] This presentation by David Laing and Jim Thomson is a good introduction: https://www.youtube.com/watch?v=lXPmh9Ap114 https://www.youtube.com/watch?v=lXPmh9Ap114 David & Tim are responsible for an ongoing multi-year systematic investigation of upgrades as a technical and organisational behaviour.
desc 7y ago
The problem is that this is a bunch of bollocks. But at the same time, that might actually resolve my problem with it as the pain gets fed back. If one is very very lucky, a library will have accurate changenotes explaining what the version increment actually means, distinguishing between security updates and wanking-over-new-subdependency-for-shiny-irrelevant-features updates. However, if people are penalised for not wasting their time chasing the latest version of leftpad-with-custom-characters-except-in-locations-which-are-prime-or-square.0.4.3-beta.butnotreally, maybe we'll see shallower dependency trees in the important stuff. Where 'important' ends up being defined as 'the packages which everyone else gravitates to, and therefore can't be avoided'. Ideally we'd see security updates for previous major versions of things, for those of us without feature addiction, but that would demand more of the devs producing this crap.
SideburnsOfDoom 7y ago
Doesn't this just re-invent the Libyears metric? https://libyear.com/ https://libyear.com/ I have skimmed through this article, it seems fairly long but light on details. How does this differ from libyears? What actually is your metric? That site (libyear.com) IMHO makes the concept refreshingly clear, this "Dependency Drift" site seems vague and markety. It's trying to make a big deal out of something, but what remains unclear after reading it. The libyear site is the opposite - clear and not a big deal.

49 points by colinbartlett 7y ago

I agree this is an important metric and a kind of technical debt that's hard to capture. How do you discriminate this from RenovateBot or Dependa-Bot? Both of those will not only track the drift but they'll also generate the PR to fix it.

colinbartlett 7y ago
Co-author here, thanks for the input. We use Dependabot on all our projects and love it. I've not seen any kind of numeric metric in Dependabot or any of their competitors. What I really want is a graph that shows, numerically, the drift over time. You can show this to your stakeholders: "See, look! We haven't dedicated time to stack upgrades and now look how far out of date we are creeping." And then, consequently, I want to see that chart burn down to zero as you do stay on top of things with the help of Dependabot or similar.
- k__ 7y ago
  As a freelancer, this sounds really interesting to me. It's always hard to sell people maintenance.
- SideburnsOfDoom 7y ago
  This is the libyears metric ( https://libyear.com/ https://libyear.com/ ) tracked over time. Or are you using a different metric that you haven't defined for us?

yourpalkeith 7y ago

A variant of this idea that would be more useful to me personally would be some measure of how actively maintained my dependencies are. It would be nice to know if my long-lived project depends on something that has been abandoned. Details might be tricky, given that some very useful packages simply don't need frequent updates.

k__ 7y ago
I like the drift metric more, because some libraries are simply stable and don't need much updates anymore, this doesn't mean they are bad for your system.

guitarbill 7y ago

The whole process of upgrading dependencies is annoying, mainly because stuff keeps breaking. I currently try to keep everything up-to-date, simply because then the number of breakages are spread out and easier to deal with, and when critical fixes come through, they're easier to apply. Automation does help with this. But ideally, I'd like the "it it ain't broke don't fix it" approach. It feels like such a waste of time. I would love to see Dependabot go further and track upgrade breakage. Being notified about vulnerabilities is a great service and takes the load off. But too many times, it still breaks. Recent example: Checkstyle security issue, Dependabot cuts a PR. Great. Except almost every minor release [0] has a breaking change. Some libraries are much better than this, and in fact I ended up simply ditching Checkstyle. (It's open source, they don't owe me anything. I get it.) Until we quantify the breakage issue, you don't have enough data to make a decision if upgrading dependencies is worth it outside of a security fix. And in those cases, it's hard to make an argument keeping up to date is the best choice (other than personal philosophy). [0] https://checkstyle.org/releasenotes.html https://checkstyle.org/releasenotes.html

colinbartlett 7y ago
I believe Dependabot quantifies this to some extent with a "Compatibility score": https://dependabot.com/#how-it-works https://dependabot.com/#how-it-works But it can only measure what people have test coverage for, presumably.
- guitarbill 7y ago
  I did see that, but it's pretty opaque (some percentage). And when I click through the badge, it takes me to GitHub's help page [0]. What would be helpful is to link to other public PRs so you might be able to find out how other people solved the issues. Is there any way to see historic compatibility scores to evaluate if it's worth to drop a certain dependency? Edit: For normal dependency updates, compatibility probably more useful. For security fixes, what choice to you have other than to apply it, or spend more time investigating and trying to come up with your own mitigation? [0] https://help.github.com/articles/configuring-automated-security-fixes https://help.github.com/articles/configuring-automated-secur...

alangibson 7y ago

I think you're on to something here. This is hot topic amongst our teams right now. Dependency maintenance looks to me to be one of those things like code duplication that correlates strongly with a multitude of sins. It's also interesting to see someone else building a tool around a new metric. I'm doing something similar (shameless plug -> https://without.fail/estimate-accuracy https://without.fail/estimate-accuracy) for determining how well estimates correlate with actual development times. I could definitely take some lessons from you because Dependency Drift is way more brandable than Estimate Accuracy.

jacques_chester 7y ago

At Pivotal (now part of VMware, but it's easier to explain this way) we have an increasing number of teams that track "vulnerability budgets" and "legacy budgets", by way of analogy to "error budgets" from SRE approaches.[0] Essentially, a vulnerability budget tracks days since last release for a dependency or system; a legacy budget tracks time left until a dependency or system falls out of support. Our CloudOps team and telemetry teams now plot these for their own systems and customer systems, producing a characteristic sawtooth pattern. I think these are good steps forward, but I see some possible evolutions. One evolution is to combine these into a more general risk framework like FAIR, so that we're able to use a single budget for all these categories of risk. Another is to consider risk exposure as a volume. At any given instant I have a wide range of risks in play, distributed over a wide range of systems. As time evolves these risks come and go. A single score (like the one sketched on the far side of the link) only gives me one axis of risk. I also need to know what systems are currently expressing that risk by the presence or absence of software, giving me a second axis. Finally, these evolve over time, giving a third axis. The sum of volume for any time slice is what ought to be budgeted. Another thing is to make these problems behave less like "big-bang control", where we switch modes drastically when crossing a threshold value. What would be preferred is a greater degree of smoothness. And finally, to make all of this tractable: automation. All the metrics and scorecards and reports and graphs in the world won't make this problem easier. They can at best make it visible and demoralising. [0] This presentation by David Laing and Jim Thomson is a good introduction: https://www.youtube.com/watch?v=lXPmh9Ap114 https://www.youtube.com/watch?v=lXPmh9Ap114 David & Tim are responsible for an ongoing multi-year systematic investigation of upgrades as a technical and organisational behaviour.

desc 7y ago

The problem is that this is a bunch of bollocks. But at the same time, that might actually resolve my problem with it as the pain gets fed back. If one is very very lucky, a library will have accurate changenotes explaining what the version increment actually means, distinguishing between security updates and wanking-over-new-subdependency-for-shiny-irrelevant-features updates. However, if people are penalised for not wasting their time chasing the latest version of leftpad-with-custom-characters-except-in-locations-which-are-prime-or-square.0.4.3-beta.butnotreally, maybe we'll see shallower dependency trees in the important stuff. Where 'important' ends up being defined as 'the packages which everyone else gravitates to, and therefore can't be avoided'. Ideally we'd see security updates for previous major versions of things, for those of us without feature addiction, but that would demand more of the devs producing this crap.

SideburnsOfDoom 7y ago

Doesn't this just re-invent the Libyears metric? https://libyear.com/ https://libyear.com/ I have skimmed through this article, it seems fairly long but light on details. How does this differ from libyears? What actually is your metric? That site (libyear.com) IMHO makes the concept refreshingly clear, this "Dependency Drift" site seems vague and markety. It's trying to make a big deal out of something, but what remains unclear after reading it. The libyear site is the opposite - clear and not a big deal.