14 ms·

The pain that minimal version selection solves

59 points by ngaut 8y ago

AndrewDucker 8y ago
It does solve that pain. But it doesn't solve the "A dependency of a dependency has a massive security hole in it, and there's a patched version which has just been released" pain. Which I'd argue is more important.
- heavenlyhash 8y ago
  This can be solved with other tools and community choices. For example, introducing automated systems which update dependencies to more recent versions, run automated tests (you already have CI set up, right?), and notify authors (including a PR; why not?) when versions can be bumped automatically. This sort of system A) doesn't involve a SAT solver and B) gets things up to date all the time. It's also roughly analogous to the principles that `go get` has already forced into the norm in the community: things should build and work against master in the dependencies; if they don't, you've got problems. At the same time, since precise dependency versions are tracked, changes can be absorbed at a controlled rate. What a trade!
  - greysteil 8y ago
    This is exactly what we do with Dependabot. We have support for dep right now (along with other languages) and will be adding modules support in the next few days.
    - jacques_chester 8y ago
      I am required by law to beg you to write a Concourse resource that spits out streams of candidate upgrades. I've wanted someone to do this for years (and been too lazy to try it myself).
  - jacques_chester 8y ago
    > This sort of system A) doesn't involve a SAT solver and B) gets things up to date all the time. I've argued for this in various forms for years. Regardless of how the dependency version is calculated, having that version chosen outside the context of version control is asking for trouble. The reason that maximal version selection existed in the first place was that manually updating dependencies has been seen as too hard, so it gets punted to the tool. The net result is that the code in production can't necessarily be associated with a particular dependency. Of course there's a cottage industry for doing this automatically, as their should be. In the case where proper CI and automation exists, I feel like minimal selection has a slight edge in that it is less likely to click unexpectedly upwards. But in practice, the practice of automation makes the overall policy of version selection close to irrelevant. Automate automate automate. Never leave this stuff to chance.
  - felixfbecker 8y ago
    This would still mean that (the deeper the dependency tree) updates take longer to trickle down to your project, and to spread across the whole ecosystem. Not everyone will have these tools set up, and they will still cause a delay since it's a waterfall - one maintainer merges the PR days after it gets raised, then they release a new version maybe a week or month later, then PRs for that package get raised, and so on. As the writer of an application I like the power of short-cutting that path if needed for an important update of a transitive dependency by simply telling my package manager to update the lockfile, vs having to wait for the update to trickle through the chain of dependencies. This may not be a problem in Go, where projects don't have that many dependencies and dependency trees are usually very flat, i.e. dependencies don't often have subdependencies¹. But in the JS ecosystem an average application has thousands of dependencies, many levels deep (as it's super common for libraries to have many subdependencies), so the effect on the ecosystem would be huge. And if there are more old versions around, that creates more maintenance effort for maintainers (you cannot avoid this with policies no matter how hard you try, proved empirically by having been an open source maintainer of many projects). ¹As to why this is, I think it is partly because Go dependency management has been so bad - you can't really depend on other packages in your library if there is no standard way to declare them. Better and more standardized dependency management will likely also make people rely on dependencies more (especially in libraries, so the tree will become deeper). But it's also some culture in Go I've observed that I've heard being described as "copy+paste oriented programming". Go is more verbose than other languages and people don't have a problem with duplicating code that much. And the Go standard library is enough for most use cases - on the other side, native JS APIs are notoriously bad (especially historically with browser differences etc), so relying on libraries for everything is completely natural (you're weird if you don't).
    - heavenlyhash 8y ago
      > updates take longer to trickle down to your project Not necessarily. Who said that we're honoring constraints every single library declared for its transitive dependencies? We could. But perhaps we should be careful not to assume all systems have to be built this way. > in Go ... dependency trees are usually very flat ; in the JS ecosystem ... many levels deep Right. So if we made the choice by fiat to simply disregard version constraints proposed by libraries for their transitive dependencies, and determine our own version selections purely by testing satisfaction of interfaces and tests, this would work out very differently in these two ecosystems due to those coefficients. In one of them, it's quite tenable in practice.
- kjksf 8y ago
  Here's a problem MSV guards against: a new checkin in a dependency introduces a massive security hole and it silently gets pulled into your project. The "security issue" argument is bogus because while new code can fix bugs it can also introduce new bugs. Every competent dev shop has code reviews. A code review exists because you don't trust your coworkers to not introduce bugs. But somehow you're ok with people you don't know to effectively check in code into your code base without as much as running a smoke test.
  - Jyaif 8y ago
    There's a difference between an old version where everybody know about the vulnerabilities because they were patched, and a new version where pretty much nobody knows any vulnerability (otherwise they would have been patched).
    - jessaustin 8y ago
      Everyone who knows about the old vulns can avoid them easily with minimum version, just by setting their dependency versions properly. The new unknown vulns about which no one knows are avoided automatically. Re-reading the thread, maybe this is what you meant? I'm not sure, possibly because this is also what 'kjksf meant.
      - jacques_chester 8y ago
        The arguments are directed past each other. Minimal selection breaks down when you ignore it and thereby allow known bugs to pile up. Maximal selection breaks down when you ignore it and thereby allow unknown, untracked dependencies to pile up. The answer for both is that version updating should trigger CI activity and then be automatically applied when tested. Dependency tools should be as strictly dumb as possible. This gap exists because dependency management tools blossomed and matured in a manual developer-at-a-keyboard context before the kind of automation that makes their clever bits less relevant or necessary.
        weberc2 8y ago
- coldtea 8y ago
  >But it doesn't solve the "A dependency of a dependency has a massive security hole in it, and there's a patched version which has just been released" pain. There's no way to solve this problem mechanically, other than to blindly trust the upstream that it only fixes security issues. They could just as well introduce ones (even in a patch that fixes an older one). Instead, we could have tools that check our "freeze" files, and get advice (from a community resource that combines upstream notices "e.g. my package xxx < v4.5 is unsafe" and community voting/tagging) to give advisory ("you better update this package").
  - majewsky 8y ago
    These tools exist: https://github.com/coreos/clair https://github.com/coreos/clair https://help.github.com/articles/about-security-alerts-for-vulnerable-dependencies/ https://help.github.com/articles/about-security-alerts-for-v...
    - heavenlyhash 8y ago
      I'm not sure this is the same as what GP is saying; these are more tools which look for known security vulnerabilities, which is fine for what it is, but... GP is pointing out the asymmetrical presumptiveness that updating at all is going to only reduce and never increase security vulnerabilities present.
      - weberc2 8y ago
        Also these tools are more useful with MVS since lower versions are better known (vulns more likely to be in these databases). If you’re getting the latest version the odds of an unknown vulnerability are much higher.
  - kornish 8y ago
    I believe there are companies tackling that pain point, such as FOSSA (http://fossa.io/ http://fossa.io/).
- karmakaze 8y ago
  Arbitrarily upgrading dependencies is how insecure versions get spread in the first place so saying rapid updating helps against it is null at best.
- nicksnyder 8y ago
  You can argue about what is more important for your project, but the worst part is that npm doesn't give project owners to decide otherwise. As kjksf already stated, patch versions can introduce security vulnerabilities too.
ainar-g 8y ago
>I was upset because the design of NPM assumes things that just aren't true in general: >People perfectly understand and apply semantic versioning (see also: compatible versioning). Doesn't any dependency management tool assume that some form of versioning is followed strictly? Go modules have another assumption besides that one, the Semantic Import Versioning principle. I understand OP's pain, but it seems to stem from bad design decisions made by NPM authors, and doesn't have a lot to do with Minimal Version Selection.
- nicksnyder 8y ago
  NPM specially assumes that "patch versions are safer to update than minor or major version". MVS doesn't make that assumption. > it seems to stem from bad design decisions made by NPM authors Yep. > doesn't have a lot to do with Minimal Version Selection The pain I experienced has nothing to do with MVS (because npm doesn't use MVS). The observation of the post is that my pain would not have happened with MVS. No more, no less.
  - felixfbecker 8y ago
    > NPM specially assumes that "patch versions are safer to update than minor or major version". I don't see where that assumption is baked into npm - the creator of a package declares the dependency constraints, so they are the ones making the choice which versions can be trusted well to work. You could pin all dependencies but then users would need to download tons of duplicate dependencies because of slight version differences everywhere. So it's a tradeoff, but the user makes it, not npm.
    - nicksnyder 8y ago
      Rephrased: npm’s design incentivizes projects to assume “patch versions are safer to update than minor versions are safer to update than major versions” to avoid dependency duplication. Such incentives do not exist with MVS because all that is required is a total ordering and there is no duplication within a major version of a module.
TheCoelacanth 8y ago
> avoids solving Version-SAT, which is NP-Complete This is just a terrible argument. Sure the problem is NP-Complete to find an optimal solution for the general case, but you don't need to do that. You need to find a good-enough solution for the cases that people actually use. Heuristics for that problem are plenty fast and even finding the optimal solution is fast enough for most of the problems people are actually trying to use.
- littlestymaar 8y ago
  Yeah, this argument is ridiculous. It's like arguing that people shouldn't have individual homes with garden because lawn mowing is NP-complete… (Not that I'm saying individual homes are a good idea in general, but the NP-complete argument is pretty dumb).
- nicksnyder 8y ago
  It isn't an argument, it is just a fact. All else equal, it is a nice fact.
- no_identd 8y ago
  Question #1: Is Version-SAT FPT? (Fixed Parameter Tractable)
jblow 8y ago
This isn't really about minimal version selection. It's about how it is bad for your code to change behind your back ... which is a much more general and widely-applicable point.
- dajohnson89 8y ago
  right. >Renovate Bot noticed that there was a new version of codeintellify so it helpfully created a pull request to update our main repository to codeintellify 3.6.0. while this sounds really cool and I'll check it out later, automatically updating your code just because a dependency has a new version seems like a REALLY bad idea.
  - nicksnyder 8y ago
    It doesn't happen automatically if you don't want it to (and it didn't happen automatically in this case). The value here is that it notifies you if an update is available (by opening a pull request), automatically tests the new dependency (when CI runs on your pull request), and makes it easy for you to apply the update if you want (by merging the pull request).
- cuspycode 8y ago
  And that is why I never do automatic OS/platform upgrades on a development machine. When something stops working that was known to work before, I want to know exactly what things have changed, and not have to troubleshoot multiple concurrent changes.
bouke 8y ago
It looks like the major problem was that NPM updated an inrelated package, despite a lock file. I would classify this as rather unexpected behaviour. I don’t see why that’s relevant to dependency resolution in situations where there’s no lock file.
- hinkley 8y ago
  Unexpected in the sense of unwanted? Yes. Unexpected in the sense of “oh my puppy chewed up the table leg”? No, this is pretty much expected behavior from npm. Unfortunately Yarn has a bug in it that I don’t know how to fix. There’s a lot of stuff going on in both code bases and I can’t help but feel like they’ve exceeded their intrinsic complexity via coupling, and that’s why people are struggling to get them to behave correctly. I wonder if splitting this job up into separate tasks and working on them separately would net some sanity.
- twic 8y ago
  The fix for this problem is not a whole new philosophy of version selection, it's being more sensible in implementing the current one. Do any of Yarn, Bundler, or Cargo have this behaviour?
  - felixfbecker 8y ago
    Yarn has the same issue (it used to do shallow updates, but this was changed due to community demand, and now theres the demand to change it back again)
jules 8y ago
This is the opposite of the monorepo philosophy: everyone always uses the most recent version of the code to minimise hassle. I wonder if minimal version selection works nicely initially, but becomes more of a hassle over time because you're basically procrastinating on moving to newer versions of your dependencies. On the other hand, package managers should not silently upgrade packages behind your back.
falcor84 8y ago
How about if we had a feedback loop between the the continuous integration/delivery system and the dependency version lock file? It would probably be very difficult to set up, but I imagine it could start by testing the minimal version, and if that works well, then do a binary search to find the latest versions of all dependencies that still work well. In the case of an exception stack trace, it could perhaps find the offensive dependency quickly, but even if not, a brute force binary search should still work.
- jacques_chester 8y ago
  I think that once such systems are set up and have reached parity with the most recent releases of dependencies, the business of keeping up is much easier, which makes it more likely to happen. We've all been in the situation where we fell behind some dependency and, upon trying to upgrade, found that we were so far behind that everything was broken in this-will-take-weeks-to-fix ways. That kind of work tends to get shunted into the too hard pile, where it generally only gets harder. Eventually the codebase ossifies around a very particular dependency set that can't be upgraded without stopping feature development. The fact that it may be carrying a stew of published vulnerabilities tends not to be enough to trigger a change: work to upgrade is concrete, immediate and costly. Exploited vulnerabilities are probabilistic, sometime in the future and not taken from the current budget. In practice stuff is kept up to date by rigid process discipline or by automation. Of the two, automation is more likely to be successful and sustained.

59 points by ngaut 8y ago

It does solve that pain. But it doesn't solve the "A dependency of a dependency has a massive security hole in it, and there's a patched version which has just been released" pain. Which I'd argue is more important.

heavenlyhash 8y ago
This can be solved with other tools and community choices. For example, introducing automated systems which update dependencies to more recent versions, run automated tests (you already have CI set up, right?), and notify authors (including a PR; why not?) when versions can be bumped automatically. This sort of system A) doesn't involve a SAT solver and B) gets things up to date all the time. It's also roughly analogous to the principles that `go get` has already forced into the norm in the community: things should build and work against master in the dependencies; if they don't, you've got problems. At the same time, since precise dependency versions are tracked, changes can be absorbed at a controlled rate. What a trade!
- greysteil 8y ago
  This is exactly what we do with Dependabot. We have support for dep right now (along with other languages) and will be adding modules support in the next few days.
  - jacques_chester 8y ago
    I am required by law to beg you to write a Concourse resource that spits out streams of candidate upgrades. I've wanted someone to do this for years (and been too lazy to try it myself).
- jacques_chester 8y ago
  > This sort of system A) doesn't involve a SAT solver and B) gets things up to date all the time. I've argued for this in various forms for years. Regardless of how the dependency version is calculated, having that version chosen outside the context of version control is asking for trouble. The reason that maximal version selection existed in the first place was that manually updating dependencies has been seen as too hard, so it gets punted to the tool. The net result is that the code in production can't necessarily be associated with a particular dependency. Of course there's a cottage industry for doing this automatically, as their should be. In the case where proper CI and automation exists, I feel like minimal selection has a slight edge in that it is less likely to click unexpectedly upwards. But in practice, the practice of automation makes the overall policy of version selection close to irrelevant. Automate automate automate. Never leave this stuff to chance.
- felixfbecker 8y ago
  This would still mean that (the deeper the dependency tree) updates take longer to trickle down to your project, and to spread across the whole ecosystem. Not everyone will have these tools set up, and they will still cause a delay since it's a waterfall - one maintainer merges the PR days after it gets raised, then they release a new version maybe a week or month later, then PRs for that package get raised, and so on. As the writer of an application I like the power of short-cutting that path if needed for an important update of a transitive dependency by simply telling my package manager to update the lockfile, vs having to wait for the update to trickle through the chain of dependencies. This may not be a problem in Go, where projects don't have that many dependencies and dependency trees are usually very flat, i.e. dependencies don't often have subdependencies¹. But in the JS ecosystem an average application has thousands of dependencies, many levels deep (as it's super common for libraries to have many subdependencies), so the effect on the ecosystem would be huge. And if there are more old versions around, that creates more maintenance effort for maintainers (you cannot avoid this with policies no matter how hard you try, proved empirically by having been an open source maintainer of many projects). ¹As to why this is, I think it is partly because Go dependency management has been so bad - you can't really depend on other packages in your library if there is no standard way to declare them. Better and more standardized dependency management will likely also make people rely on dependencies more (especially in libraries, so the tree will become deeper). But it's also some culture in Go I've observed that I've heard being described as "copy+paste oriented programming". Go is more verbose than other languages and people don't have a problem with duplicating code that much. And the Go standard library is enough for most use cases - on the other side, native JS APIs are notoriously bad (especially historically with browser differences etc), so relying on libraries for everything is completely natural (you're weird if you don't).
  - heavenlyhash 8y ago
    > updates take longer to trickle down to your project Not necessarily. Who said that we're honoring constraints every single library declared for its transitive dependencies? We could. But perhaps we should be careful not to assume all systems have to be built this way. > in Go ... dependency trees are usually very flat ; in the JS ecosystem ... many levels deep Right. So if we made the choice by fiat to simply disregard version constraints proposed by libraries for their transitive dependencies, and determine our own version selections purely by testing satisfaction of interfaces and tests, this would work out very differently in these two ecosystems due to those coefficients. In one of them, it's quite tenable in practice.
kjksf 8y ago
Here's a problem MSV guards against: a new checkin in a dependency introduces a massive security hole and it silently gets pulled into your project. The "security issue" argument is bogus because while new code can fix bugs it can also introduce new bugs. Every competent dev shop has code reviews. A code review exists because you don't trust your coworkers to not introduce bugs. But somehow you're ok with people you don't know to effectively check in code into your code base without as much as running a smoke test.
- Jyaif 8y ago
  There's a difference between an old version where everybody know about the vulnerabilities because they were patched, and a new version where pretty much nobody knows any vulnerability (otherwise they would have been patched).
  - jessaustin 8y ago
    Everyone who knows about the old vulns can avoid them easily with minimum version, just by setting their dependency versions properly. The new unknown vulns about which no one knows are avoided automatically. Re-reading the thread, maybe this is what you meant? I'm not sure, possibly because this is also what 'kjksf meant.
    - jacques_chester 8y ago
      The arguments are directed past each other. Minimal selection breaks down when you ignore it and thereby allow known bugs to pile up. Maximal selection breaks down when you ignore it and thereby allow unknown, untracked dependencies to pile up. The answer for both is that version updating should trigger CI activity and then be automatically applied when tested. Dependency tools should be as strictly dumb as possible. This gap exists because dependency management tools blossomed and matured in a manual developer-at-a-keyboard context before the kind of automation that makes their clever bits less relevant or necessary.
      - weberc2 8y ago
coldtea 8y ago
>But it doesn't solve the "A dependency of a dependency has a massive security hole in it, and there's a patched version which has just been released" pain. There's no way to solve this problem mechanically, other than to blindly trust the upstream that it only fixes security issues. They could just as well introduce ones (even in a patch that fixes an older one). Instead, we could have tools that check our "freeze" files, and get advice (from a community resource that combines upstream notices "e.g. my package xxx < v4.5 is unsafe" and community voting/tagging) to give advisory ("you better update this package").
- majewsky 8y ago
  These tools exist: https://github.com/coreos/clair https://github.com/coreos/clair https://help.github.com/articles/about-security-alerts-for-vulnerable-dependencies/ https://help.github.com/articles/about-security-alerts-for-v...
  - heavenlyhash 8y ago
    I'm not sure this is the same as what GP is saying; these are more tools which look for known security vulnerabilities, which is fine for what it is, but... GP is pointing out the asymmetrical presumptiveness that updating at all is going to only reduce and never increase security vulnerabilities present.
    - weberc2 8y ago
      Also these tools are more useful with MVS since lower versions are better known (vulns more likely to be in these databases). If you’re getting the latest version the odds of an unknown vulnerability are much higher.
- kornish 8y ago
  I believe there are companies tackling that pain point, such as FOSSA (http://fossa.io/ http://fossa.io/).
karmakaze 8y ago
Arbitrarily upgrading dependencies is how insecure versions get spread in the first place so saying rapid updating helps against it is null at best.
nicksnyder 8y ago
You can argue about what is more important for your project, but the worst part is that npm doesn't give project owners to decide otherwise. As kjksf already stated, patch versions can introduce security vulnerabilities too.

ainar-g 8y ago

>I was upset because the design of NPM assumes things that just aren't true in general: >People perfectly understand and apply semantic versioning (see also: compatible versioning). Doesn't any dependency management tool assume that some form of versioning is followed strictly? Go modules have another assumption besides that one, the Semantic Import Versioning principle. I understand OP's pain, but it seems to stem from bad design decisions made by NPM authors, and doesn't have a lot to do with Minimal Version Selection.

nicksnyder 8y ago
NPM specially assumes that "patch versions are safer to update than minor or major version". MVS doesn't make that assumption. > it seems to stem from bad design decisions made by NPM authors Yep. > doesn't have a lot to do with Minimal Version Selection The pain I experienced has nothing to do with MVS (because npm doesn't use MVS). The observation of the post is that my pain would not have happened with MVS. No more, no less.
- felixfbecker 8y ago
  > NPM specially assumes that "patch versions are safer to update than minor or major version". I don't see where that assumption is baked into npm - the creator of a package declares the dependency constraints, so they are the ones making the choice which versions can be trusted well to work. You could pin all dependencies but then users would need to download tons of duplicate dependencies because of slight version differences everywhere. So it's a tradeoff, but the user makes it, not npm.
  - nicksnyder 8y ago
    Rephrased: npm’s design incentivizes projects to assume “patch versions are safer to update than minor versions are safer to update than major versions” to avoid dependency duplication. Such incentives do not exist with MVS because all that is required is a total ordering and there is no duplication within a major version of a module.

TheCoelacanth 8y ago

> avoids solving Version-SAT, which is NP-Complete This is just a terrible argument. Sure the problem is NP-Complete to find an optimal solution for the general case, but you don't need to do that. You need to find a good-enough solution for the cases that people actually use. Heuristics for that problem are plenty fast and even finding the optimal solution is fast enough for most of the problems people are actually trying to use.

littlestymaar 8y ago
Yeah, this argument is ridiculous. It's like arguing that people shouldn't have individual homes with garden because lawn mowing is NP-complete… (Not that I'm saying individual homes are a good idea in general, but the NP-complete argument is pretty dumb).
nicksnyder 8y ago
It isn't an argument, it is just a fact. All else equal, it is a nice fact.
no_identd 8y ago
Question #1: Is Version-SAT FPT? (Fixed Parameter Tractable)

jblow 8y ago

This isn't really about minimal version selection. It's about how it is bad for your code to change behind your back ... which is a much more general and widely-applicable point.

dajohnson89 8y ago
right. >Renovate Bot noticed that there was a new version of codeintellify so it helpfully created a pull request to update our main repository to codeintellify 3.6.0. while this sounds really cool and I'll check it out later, automatically updating your code just because a dependency has a new version seems like a REALLY bad idea.
- nicksnyder 8y ago
  It doesn't happen automatically if you don't want it to (and it didn't happen automatically in this case). The value here is that it notifies you if an update is available (by opening a pull request), automatically tests the new dependency (when CI runs on your pull request), and makes it easy for you to apply the update if you want (by merging the pull request).
cuspycode 8y ago
And that is why I never do automatic OS/platform upgrades on a development machine. When something stops working that was known to work before, I want to know exactly what things have changed, and not have to troubleshoot multiple concurrent changes.

bouke 8y ago

It looks like the major problem was that NPM updated an inrelated package, despite a lock file. I would classify this as rather unexpected behaviour. I don’t see why that’s relevant to dependency resolution in situations where there’s no lock file.

hinkley 8y ago
Unexpected in the sense of unwanted? Yes. Unexpected in the sense of “oh my puppy chewed up the table leg”? No, this is pretty much expected behavior from npm. Unfortunately Yarn has a bug in it that I don’t know how to fix. There’s a lot of stuff going on in both code bases and I can’t help but feel like they’ve exceeded their intrinsic complexity via coupling, and that’s why people are struggling to get them to behave correctly. I wonder if splitting this job up into separate tasks and working on them separately would net some sanity.
twic 8y ago
The fix for this problem is not a whole new philosophy of version selection, it's being more sensible in implementing the current one. Do any of Yarn, Bundler, or Cargo have this behaviour?
- felixfbecker 8y ago
  Yarn has the same issue (it used to do shallow updates, but this was changed due to community demand, and now theres the demand to change it back again)

jules 8y ago

This is the opposite of the monorepo philosophy: everyone always uses the most recent version of the code to minimise hassle. I wonder if minimal version selection works nicely initially, but becomes more of a hassle over time because you're basically procrastinating on moving to newer versions of your dependencies. On the other hand, package managers should not silently upgrade packages behind your back.

falcor84 8y ago

How about if we had a feedback loop between the the continuous integration/delivery system and the dependency version lock file? It would probably be very difficult to set up, but I imagine it could start by testing the minimal version, and if that works well, then do a binary search to find the latest versions of all dependencies that still work well. In the case of an exception stack trace, it could perhaps find the offensive dependency quickly, but even if not, a brute force binary search should still work.

jacques_chester 8y ago
I think that once such systems are set up and have reached parity with the most recent releases of dependencies, the business of keeping up is much easier, which makes it more likely to happen. We've all been in the situation where we fell behind some dependency and, upon trying to upgrade, found that we were so far behind that everything was broken in this-will-take-weeks-to-fix ways. That kind of work tends to get shunted into the too hard pile, where it generally only gets harder. Eventually the codebase ossifies around a very particular dependency set that can't be upgraded without stopping feature development. The fact that it may be carrying a stew of published vulnerabilities tends not to be enough to trigger a change: work to upgrade is concrete, immediate and costly. Exploited vulnerabilities are probabilistic, sometime in the future and not taken from the current budget. In practice stuff is kept up to date by rigid process discipline or by automation. Of the two, automation is more likely to be successful and sustained.