5 ms·

On breaking changes in transitive dependencies

24 points by luu 3y ago

dboreham 3y ago
This: "Imagine a world where software could be tested against a version of the dependency, and then allow all version updates as long as they are minor (a new functionality added in a backward-compatible manner)" is a fantasy in my experience. The Semver contract exists as a concept sure, but there's no guarantee that the publisher of a dependency you consume actually maintains compatibility, in the context of your applications usage, between minor versions. It's more of an aspiration.
- jaapz 3y ago
  As someone who has been bitten time and time again by transitive dependencies not adhering to semver while still claiming to be using semver, i totally agree. Dependency pinning is a must when you want to have full control, and even then some package managers allow pulling or updating specific versions of dependencies (looking at you, pip).
  - User23 3y ago
    One can conceive of a solution that bisects dependencies independently to find the latest version that satisfies a comprehensive test suite. But that build and test is going to have to be reasonably quick. Even moreso if you take the exhaustive approach and transitively apply the algorithm to dependency dependencies which would also require backtracking. A more durable approach might be to require every API to formally and verifiably provide precondition and postcondition that can be checked with an SMT solver for every public interface. With current tech that is pushing a lot onto module authors, but I believe there is some research being done on inferring both preconditions and postconditions from the semantics of the component statements so perhaps we can avoid forcing programmers to carry the intolerable burden of learning the first order predicate calculus.
- hinkley 3y ago
  I haven’t figured out the math yet, but on my shelf if future projects, I have a part of a plan to tie semver to unit tests. The theory so far is: there’s a subset of unit tests that specify the API. If you create new tests, that’s mostly fine. If have to alter a test, that’s probably a breaking change. The main problem I can already see with this, which is why it’s on the shelf, is that some test frameworks and many tester writers cannot guarantee the invariants as stated above. You end up changing tests for minor behavioral changes. You mix tests of private fields in with tests of public ones. And even if you split them, then functional tests end up defining parts of the API. But somewhere out there in the future is a language where types are defined by tests, and certain testing patterns indicate minor version increments and others major.
  - jrochkind1 3y ago
    I've thought about this too. The other issue is that tests can have bugs too, sometimes you are just fixing a test bug.
    - hinkley 3y ago
      Definitely. So in a robust system, you might also have to look at code coverage to see if the test interacts with any code changes since the last minor version increment. If I change the test but not the code, that may be a clarification. If I change both, it’s ambiguous.
- davidmurdoch 3y ago
  This has bitten me twice before: a transitive dep update broke my library. I now publish my npm shrinkwrap file and only officially support npm (or anything that respects the published lock file, which I think is only npm).
  - no_wizard 3y ago
    I think yarn and pnpm can read these
    - davidmurdoch 3y ago
      They don't use published lock files.
  - 3np 3y ago
    Why do you prefer shrinkwrap instead of npm's package-lock.json?
    - davidmurdoch 3y ago
      Package locks are ignored by npm when published.
      - 3np 3y ago
        Oh, so do I understand it right if you want your users to use specific pinned version for all of your dependencies? In that case why not pin to specific versions (as opposed to semver ranges) in package.json, which would be respected identically by npm/yarn/pnpm and again remove the need for shrinkwrap?
        davidmurdoch 3y ago
        Yes. Because that doesn't pin dependencies of your dependencies (this is what "transitive dependencies" means).
Joker_vD 3y ago
> That change is absolutely trivial to handle, and it will be a while before the behavior change. It doesn’t require a major version bump. That change, however trivial to handle, is still a bloody breaking change and it still has to be handled, by the library users, at one point of time or another. And how do you notify your users that they need to handle something that's changed? That's right, you mention it in the RELEASE file, a major version bump is entirely superfluous. /s This is why we can't have nice things: apparently there are library maintainers out there who just don't get it what semver is actually about.
globalreset 3y ago
Integers used for errors, lack of static typing, no semver, no tests to verify that behavior. I guess Ruby also can't use multiple versions of the same gem(?).
stilist 3y ago
This seems like a case where Go’s ‘minimum version selection’ approach would have helped——Sidekiq would have continued using an older version of the dependency that didn’t have the breaking change.
waffletower 3y ago
It is good to hear that the Ruby ecosystem is improving advocacy and utilization of Semantic Versioning given, in my experience, its unstable past.
waffletower 3y ago
No, Semantic Versioning (SemVer) is not a silver bullet. But the concept can be an effective software stabilizer. However, utilization of unbounded versions in transitive dependencies (for example, `pandas>=1.5` in a python requirements.txt) can silently break this contract. Even if a transitive dependency utilizes SemVer, a consumer of such a library has to also conform. The pandas example was found in a transitive dependency in one of our production processes, and one day this April it broke our production service when pandas released its 2.0 version. We had pinned all our dependencies, but the man in the middle had not. The python ecosystem is notorious for this time consuming naiveté.
- jonhohle 3y ago
  One possible issue is that compile or test time dependency resolution, especially for uncompiled languages, doesn’t distinguish betweeen explicit and transitive dependencies in most package managers/build systems. If you have a direct source dependency, imho, it should be explicit (not relied on transitively), and if not (unless a specific runtime version is necessary), transitive dependencies should not be locked or specifically annotated in a lock file. The former is overly flexible, the latter overly rigid. I also prefer an external manifest that can lock specific versions of all dependencies, but that usually requires a separate package manager from the language provided one, which introduces a whole new set of issues.
taeric 3y ago
My hot take is that SemVer can only really work if the team also restricts how much of the existing code they are willing to change. In particular, the point is stability of existing code, which means that you basically have to somewhat restrict yourself to use the existing ways as much as you can. Any codebase where folks are building new ways of doing things will build up odd mismatches in what is actually supported versus what exists, and then gives incentives for folks to remove old ways. Worse, SemVer is useless from small teams where they are not budgeting time and effort to maintain all of the old version bumps. Pointedly, just updating the feature version can help folks know it is dangerous to upgrade. But, if you aren't going to maintain the old versions forever, it is equally dangerous to not upgrade.

24 points by luu 3y ago

This: "Imagine a world where software could be tested against a version of the dependency, and then allow all version updates as long as they are minor (a new functionality added in a backward-compatible manner)" is a fantasy in my experience. The Semver contract exists as a concept sure, but there's no guarantee that the publisher of a dependency you consume actually maintains compatibility, in the context of your applications usage, between minor versions. It's more of an aspiration.

jaapz 3y ago
As someone who has been bitten time and time again by transitive dependencies not adhering to semver while still claiming to be using semver, i totally agree. Dependency pinning is a must when you want to have full control, and even then some package managers allow pulling or updating specific versions of dependencies (looking at you, pip).
- User23 3y ago
  One can conceive of a solution that bisects dependencies independently to find the latest version that satisfies a comprehensive test suite. But that build and test is going to have to be reasonably quick. Even moreso if you take the exhaustive approach and transitively apply the algorithm to dependency dependencies which would also require backtracking. A more durable approach might be to require every API to formally and verifiably provide precondition and postcondition that can be checked with an SMT solver for every public interface. With current tech that is pushing a lot onto module authors, but I believe there is some research being done on inferring both preconditions and postconditions from the semantics of the component statements so perhaps we can avoid forcing programmers to carry the intolerable burden of learning the first order predicate calculus.
hinkley 3y ago
I haven’t figured out the math yet, but on my shelf if future projects, I have a part of a plan to tie semver to unit tests. The theory so far is: there’s a subset of unit tests that specify the API. If you create new tests, that’s mostly fine. If have to alter a test, that’s probably a breaking change. The main problem I can already see with this, which is why it’s on the shelf, is that some test frameworks and many tester writers cannot guarantee the invariants as stated above. You end up changing tests for minor behavioral changes. You mix tests of private fields in with tests of public ones. And even if you split them, then functional tests end up defining parts of the API. But somewhere out there in the future is a language where types are defined by tests, and certain testing patterns indicate minor version increments and others major.
- jrochkind1 3y ago
  I've thought about this too. The other issue is that tests can have bugs too, sometimes you are just fixing a test bug.
  - hinkley 3y ago
    Definitely. So in a robust system, you might also have to look at code coverage to see if the test interacts with any code changes since the last minor version increment. If I change the test but not the code, that may be a clarification. If I change both, it’s ambiguous.
davidmurdoch 3y ago
This has bitten me twice before: a transitive dep update broke my library. I now publish my npm shrinkwrap file and only officially support npm (or anything that respects the published lock file, which I think is only npm).
- no_wizard 3y ago
  I think yarn and pnpm can read these
  - davidmurdoch 3y ago
    They don't use published lock files.
- 3np 3y ago
  Why do you prefer shrinkwrap instead of npm's package-lock.json?
  - davidmurdoch 3y ago
    Package locks are ignored by npm when published.
    - 3np 3y ago
      Oh, so do I understand it right if you want your users to use specific pinned version for all of your dependencies? In that case why not pin to specific versions (as opposed to semver ranges) in package.json, which would be respected identically by npm/yarn/pnpm and again remove the need for shrinkwrap?
      - davidmurdoch 3y ago
        Yes. Because that doesn't pin dependencies of your dependencies (this is what "transitive dependencies" means).

Joker_vD 3y ago

> That change is absolutely trivial to handle, and it will be a while before the behavior change. It doesn’t require a major version bump. That change, however trivial to handle, is still a bloody breaking change and it still has to be handled, by the library users, at one point of time or another. And how do you notify your users that they need to handle something that's changed? That's right, you mention it in the RELEASE file, a major version bump is entirely superfluous. /s This is why we can't have nice things: apparently there are library maintainers out there who just don't get it what semver is actually about.

globalreset 3y ago

Integers used for errors, lack of static typing, no semver, no tests to verify that behavior. I guess Ruby also can't use multiple versions of the same gem(?).

stilist 3y ago

This seems like a case where Go’s ‘minimum version selection’ approach would have helped——Sidekiq would have continued using an older version of the dependency that didn’t have the breaking change.

waffletower 3y ago

It is good to hear that the Ruby ecosystem is improving advocacy and utilization of Semantic Versioning given, in my experience, its unstable past.

waffletower 3y ago

No, Semantic Versioning (SemVer) is not a silver bullet. But the concept can be an effective software stabilizer. However, utilization of unbounded versions in transitive dependencies (for example, `pandas>=1.5` in a python requirements.txt) can silently break this contract. Even if a transitive dependency utilizes SemVer, a consumer of such a library has to also conform. The pandas example was found in a transitive dependency in one of our production processes, and one day this April it broke our production service when pandas released its 2.0 version. We had pinned all our dependencies, but the man in the middle had not. The python ecosystem is notorious for this time consuming naiveté.

jonhohle 3y ago
One possible issue is that compile or test time dependency resolution, especially for uncompiled languages, doesn’t distinguish betweeen explicit and transitive dependencies in most package managers/build systems. If you have a direct source dependency, imho, it should be explicit (not relied on transitively), and if not (unless a specific runtime version is necessary), transitive dependencies should not be locked or specifically annotated in a lock file. The former is overly flexible, the latter overly rigid. I also prefer an external manifest that can lock specific versions of all dependencies, but that usually requires a separate package manager from the language provided one, which introduces a whole new set of issues.

taeric 3y ago

My hot take is that SemVer can only really work if the team also restricts how much of the existing code they are willing to change. In particular, the point is stability of existing code, which means that you basically have to somewhat restrict yourself to use the existing ways as much as you can. Any codebase where folks are building new ways of doing things will build up odd mismatches in what is actually supported versus what exists, and then gives incentives for folks to remove old ways. Worse, SemVer is useless from small teams where they are not budgeting time and effort to maintain all of the old version bumps. Pointedly, just updating the feature version can help folks know it is dangerous to upgrade. But, if you aren't going to maintain the old versions forever, it is equally dangerous to not upgrade.