8 ms·

Did I break you? Reverse dependency verification

73 points by r4um 5y ago

ltbarcly3 5y ago
Ok, if I understand this properly: They have a core library that other projects use. They want to release a new version of this core library, and upgrade all the projects to use the newer version. If the CI breaks for the projects using the library, they want to catch it. This still depends on the projects using the library having enough and good enough testing for something to break in the CI when this core library does something bad. That is a huge investment (having those high quality tests). The rest of this is just a script that runs the tests from those projects when people merge code into the core library, which seems like very much not a big deal.
- treffer 5y ago
  The former approach was release + upgrade all projects. Now if one of those fails post-update you have to build a new release and upgrade all projects _again_. Multiply that by hundreds of services and it can become pretty annoying. By pulling all the reverse dependencies and testing those _before_ releasing & upgrading you reduce the chance that you need another bugfix release. And with hundreds of projects the average quality of tests does not matter that much. Some projects will happily pass, while some may fail. Those that fail will provide valuable insights.
  - __alexs 5y ago
    > The former approach was release + upgrade all projects. Now if one of those fails post-update you have to build a new release and upgrade all projects _again_. Multiply that by hundreds of services and it can become pretty annoying. I think there are actually at least 3 possibilities when you find a new version of a component breaks something that depends on it? 1) Continue using the old version for however long it still meets it's requirements and then do (2). 2) Make the local changes required to be compatible with the new version. 3) There is actually a bug in the new version and you need to fix the dependency. If (3) is happening a lot it suggests that your test suite isn't good enough or you have some design problems that are making it impossible to provide a stable API.
    - Mathnerd314 5y ago
      What it sounds like is that they don't have many unit tests for jvmkit. So rather than write extensive unit tests, they're using integration testing with all the dependencies. I guess it makes sense, network stacks are hard to unit test.
    - treffer 5y ago
      Yes, and the whole solution allows you to plan / decide on (2) and (3) - both need that awareness. And (1) is common in the open source and cross organization world. But it is less desirable if both the consumers and producers of the library work in the same place. Version spread causes maintenance cost. And at one point it was a common theme that "an upgrade of JVMKit would have avoided this incident".
stuaxo 5y ago
Nice - I've been thinking stuff should work like this for a decade+
pavel_lishin 5y ago
Skimming this, because I'm in a meeting, but it reminds me of contract testing, which I'm working on implementing at work today with Pact.
masklinn 5y ago
For other projects doing something similar, the Rust project has Crater (https://crater.rust-lang.org https://crater.rust-lang.org), which I believe is automatically run on betas, and can be run explicitly on PRs. But is not by default. Because it builds tens of thousands of crates using the specified compiler, and takes 3 days to complete a run.
- seoaeu 5y ago
  Crater doesn't just run builds! It also compiles and runs unit tests to look for regressions between compiler versions. When I last looked the tool grabbed every library on crates.io and every repository on GitHub with Rust code, but they may have cut down the scope slightly as the ecosystem has grown
dec0dedab0de 5y ago
This might be unpopular, but I don't think libraries should ever break backwards compatibility, at least not with enough regularity to require all of this. Sure if there is a major security thing, or something outside of your control then you might have to. But if you just want to change a function's signature, or return value, because you think it will help "clean things up" could you please just make a new function, or make it work both ways if your language allows it. I understand the desire to tear out a messy bit of code and make it beautiful, but it always leads to a bunch of headaches for little gain.
- Mathnerd314 5y ago
  There's no way to entirely avoid breaking compatibility. If you allow small API breakages then at least all of the team is aware of the possibility and designs their processes to handle backwards incompatible changes. If you design your processes assuming backwards compatibility then you'll run into the 1-in-1000 case where it's not possible and have to stop everything to handle the major update.
  - jnxx 5y ago
    > There's no way to entirely avoid breaking compatibility. Why?
    - masklinn 5y ago
      Because you've got no idea how your code's being used in the wild an one person's bug is an other's feature. So any observable change in behaviour is a possibly breaking change. And then people go and muck with your internals even if you protected them, because that's the only way they found to do what they needed to do, and they were much more interested in doing the thing than following your guidelines. And that's before even mentioning broken features which requires changes which will invariably break other stuff. The only way to know for sure you're never breaking compatibility is to never release any update. Otherwise you're playing the odds.
      - 5y ago
catern 5y ago
This is a good technique, and it's preferable to investing a bunch of effort into building a comprehensive suite of unit tests. But this technique is applicable to more than just core libraries; you can apply this to your entire stack, and to your services: Test that you don't break your reverse dependencies, by running those reverse dependencies, like in http://catern.com/usertests.html http://catern.com/usertests.html
BugsJustFindMe 5y ago
Following reverse dependencies on github (trivial with the search api for enough cases to make a difference) is something that I very very strongly wish that Pypa had done for the pip 20 resolver change. It was even suggested to them by several people, but the suggestions were dismissed because breaking countless CI toolchains across the globe was considered fine because the new resolution behavior was "correct". Instead we got https://github.com/pypa/pip/issues/8713 https://github.com/pypa/pip/issues/8713 back in August and then https://github.com/pypa/pip/issues/9187 https://github.com/pypa/pip/issues/9187 in November which is pinned and still open to this day.

73 points by r4um 5y ago

Ok, if I understand this properly: They have a core library that other projects use. They want to release a new version of this core library, and upgrade all the projects to use the newer version. If the CI breaks for the projects using the library, they want to catch it. This still depends on the projects using the library having enough and good enough testing for something to break in the CI when this core library does something bad. That is a huge investment (having those high quality tests). The rest of this is just a script that runs the tests from those projects when people merge code into the core library, which seems like very much not a big deal.

treffer 5y ago
The former approach was release + upgrade all projects. Now if one of those fails post-update you have to build a new release and upgrade all projects _again_. Multiply that by hundreds of services and it can become pretty annoying. By pulling all the reverse dependencies and testing those _before_ releasing & upgrading you reduce the chance that you need another bugfix release. And with hundreds of projects the average quality of tests does not matter that much. Some projects will happily pass, while some may fail. Those that fail will provide valuable insights.
- __alexs 5y ago
  > The former approach was release + upgrade all projects. Now if one of those fails post-update you have to build a new release and upgrade all projects _again_. Multiply that by hundreds of services and it can become pretty annoying. I think there are actually at least 3 possibilities when you find a new version of a component breaks something that depends on it? 1) Continue using the old version for however long it still meets it's requirements and then do (2). 2) Make the local changes required to be compatible with the new version. 3) There is actually a bug in the new version and you need to fix the dependency. If (3) is happening a lot it suggests that your test suite isn't good enough or you have some design problems that are making it impossible to provide a stable API.
  - Mathnerd314 5y ago
    What it sounds like is that they don't have many unit tests for jvmkit. So rather than write extensive unit tests, they're using integration testing with all the dependencies. I guess it makes sense, network stacks are hard to unit test.
  - treffer 5y ago
    Yes, and the whole solution allows you to plan / decide on (2) and (3) - both need that awareness. And (1) is common in the open source and cross organization world. But it is less desirable if both the consumers and producers of the library work in the same place. Version spread causes maintenance cost. And at one point it was a common theme that "an upgrade of JVMKit would have avoided this incident".

stuaxo 5y ago

Nice - I've been thinking stuff should work like this for a decade+

pavel_lishin 5y ago

Skimming this, because I'm in a meeting, but it reminds me of contract testing, which I'm working on implementing at work today with Pact.

masklinn 5y ago

For other projects doing something similar, the Rust project has Crater (https://crater.rust-lang.org https://crater.rust-lang.org), which I believe is automatically run on betas, and can be run explicitly on PRs. But is not by default. Because it builds tens of thousands of crates using the specified compiler, and takes 3 days to complete a run.

seoaeu 5y ago
Crater doesn't just run builds! It also compiles and runs unit tests to look for regressions between compiler versions. When I last looked the tool grabbed every library on crates.io and every repository on GitHub with Rust code, but they may have cut down the scope slightly as the ecosystem has grown

dec0dedab0de 5y ago

This might be unpopular, but I don't think libraries should ever break backwards compatibility, at least not with enough regularity to require all of this. Sure if there is a major security thing, or something outside of your control then you might have to. But if you just want to change a function's signature, or return value, because you think it will help "clean things up" could you please just make a new function, or make it work both ways if your language allows it. I understand the desire to tear out a messy bit of code and make it beautiful, but it always leads to a bunch of headaches for little gain.

Mathnerd314 5y ago
There's no way to entirely avoid breaking compatibility. If you allow small API breakages then at least all of the team is aware of the possibility and designs their processes to handle backwards incompatible changes. If you design your processes assuming backwards compatibility then you'll run into the 1-in-1000 case where it's not possible and have to stop everything to handle the major update.
- jnxx 5y ago
  > There's no way to entirely avoid breaking compatibility. Why?
  - masklinn 5y ago
    Because you've got no idea how your code's being used in the wild an one person's bug is an other's feature. So any observable change in behaviour is a possibly breaking change. And then people go and muck with your internals even if you protected them, because that's the only way they found to do what they needed to do, and they were much more interested in doing the thing than following your guidelines. And that's before even mentioning broken features which requires changes which will invariably break other stuff. The only way to know for sure you're never breaking compatibility is to never release any update. Otherwise you're playing the odds.
    - 5y ago

catern 5y ago

This is a good technique, and it's preferable to investing a bunch of effort into building a comprehensive suite of unit tests. But this technique is applicable to more than just core libraries; you can apply this to your entire stack, and to your services: Test that you don't break your reverse dependencies, by running those reverse dependencies, like in http://catern.com/usertests.html http://catern.com/usertests.html

BugsJustFindMe 5y ago

Following reverse dependencies on github (trivial with the search api for enough cases to make a difference) is something that I very very strongly wish that Pypa had done for the pip 20 resolver change. It was even suggested to them by several people, but the suggestions were dismissed because breaking countless CI toolchains across the globe was considered fine because the new resolution behavior was "correct". Instead we got https://github.com/pypa/pip/issues/8713 https://github.com/pypa/pip/issues/8713 back in August and then https://github.com/pypa/pip/issues/9187 https://github.com/pypa/pip/issues/9187 in November which is pinned and still open to this day.