4 ms·

This is all great, but I feel like all these problems could be caught just with properly written tests. If your tests correctly cover the API usage of your code

by jarfil 8y ago

This is all great, but I feel like all these problems could be caught just with properly written tests. If your tests correctly cover the API usage of your code, and I mean both your code complying with the intended API and the API complying with the intended usage, then the implementation behind that API should be totally transparent. No need to check versions, release notes, or any of that, just run the API compliance tests on the new version, and if it works then your code should work too.

lotyrin 8y ago
Treating the test suite as your contract and having a well structured + documented test suite also makes SemVer considerations a matter of what tests changed and how did they change (new cases, new features, changes to existing features, removed features, etc.)?
jakobegger 8y ago
Relying on tests is naive. Your tests can't cover every case. The article even mentions this -- their tests passed, but it failed in production.
- crdoconnor 8y ago
  They built tests that explicitly assumed that the library's interface wasn't going to change: >When we upgraded, all our tests passed (because our test fixtures emulated the old behavior) Doing that, upgrading your dependencies and expecting everything to work just because those tests passed? That's naivete. If they'd built decent integration tests that used the actual library (instead of "assume nothing changes" fixtures) and made more of an effort to simulate realistic scenarios then their tests probably would have flagged up most of the issues they had. Alas, this seems to be one of the side effects of following the "test pyramid" "best practice".
  - jakobegger 8y ago
    I wasn't talking about that paragraph, but the following paragraph where they had tests, but they didn't test with large enough packets. Tests can never cover every scenario. They are very useful, and they catch a lot of unexpected regressions. But they're just a part of the puzzle, not a replacement for good development practices. Updating a dependency without bothering to read the release notes because you have tests -- maybe naive is the wrong word, maybe hubris fits better.
    - crdoconnor 8y ago
      Tests can't cover every scenario, no, but had they made a bit more of an effort to test realistically then it's absolutely possible that they could have covered every scenario that mattered here. Over-reliance on unrealistic unit tests (which is likely what led to them not testing large packets) is a pattern I've seen cause issues like this many, many times before. I upgrade pretty regularly without reading release notes - relying on realistic tests to catch everything. What they do catch is usually not in the slightest bit obvious from release notes (often a regression in the dependency). Call it hubris if you like, but it works for me.

by jarfil 8y ago

lotyrin 8y ago

Treating the test suite as your contract and having a well structured + documented test suite also makes SemVer considerations a matter of what tests changed and how did they change (new cases, new features, changes to existing features, removed features, etc.)?

jakobegger 8y ago

Relying on tests is naive. Your tests can't cover every case. The article even mentions this -- their tests passed, but it failed in production.

crdoconnor 8y ago
They built tests that explicitly assumed that the library's interface wasn't going to change: >When we upgraded, all our tests passed (because our test fixtures emulated the old behavior) Doing that, upgrading your dependencies and expecting everything to work just because those tests passed? That's naivete. If they'd built decent integration tests that used the actual library (instead of "assume nothing changes" fixtures) and made more of an effort to simulate realistic scenarios then their tests probably would have flagged up most of the issues they had. Alas, this seems to be one of the side effects of following the "test pyramid" "best practice".
- jakobegger 8y ago
  I wasn't talking about that paragraph, but the following paragraph where they had tests, but they didn't test with large enough packets. Tests can never cover every scenario. They are very useful, and they catch a lot of unexpected regressions. But they're just a part of the puzzle, not a replacement for good development practices. Updating a dependency without bothering to read the release notes because you have tests -- maybe naive is the wrong word, maybe hubris fits better.
  - crdoconnor 8y ago
    Tests can't cover every scenario, no, but had they made a bit more of an effort to test realistically then it's absolutely possible that they could have covered every scenario that mattered here. Over-reliance on unrealistic unit tests (which is likely what led to them not testing large packets) is a pattern I've seen cause issues like this many, many times before. I upgrade pretty regularly without reading release notes - relying on realistic tests to catch everything. What they do catch is usually not in the slightest bit obvious from release notes (often a regression in the dependency). Call it hubris if you like, but it works for me.