4 ms·

While it is a good example, you could also use the same example and conclude that the problem was inadequate tests. SemVer is great, but you can't count on depe

While it is a good example, you could also use the same example and conclude that the problem was inadequate tests. SemVer is great, but you can't count on dependencies that you do not control actually adhering to it, either intentionally or unintentionally. The only thing that could have prevented something like this for sure was mentioned: > And while nothing in our early testing sent messages larger than 256k, there were plenty of production instances that did. To me, this was the clear failure; not the fact that some dependency broke semver. Their production system relied on being able to send messages larger than 256k, and their tests did not.

pvorb 8y ago
But isn't it impractical to test every feature of every library you are using? In an ideal world you would have everything tested in isolation as well as integration. But in practice there will always be a corner case that remains untested because you don't know all internals of the libraries you use.
- CoryG89 8y ago
  If you want to be able to randomly upgrade those dependencies and not have to worry about a breaking change, then yes. Semver is not going to help you there. Server is only going to help you when someone knows they are releasing breaking change. And even then, only if they are nice enough to actually follow the spec. You don't have to test every bit of every dependency you use, but upgrading them without either carefully reviewing the changes or having tests in place for at least critical functionality is asking for something like this to happen eventually.
gregmac 8y ago
While it's easy to say, how far do you go? Do you test every bit of every upstream library you use? The ideal is probably yes, but the reality is this rarely happens. Even with a test, you may not find this. In the IOException example, the author calls out why: > When we upgraded, all our tests passed (because our test fixtures emulated the old behavior and our network was not unstable enough to trigger bad conditions) The only way to catch this type of thing is to emulate the entire network side of things, and that's still only as good as your simulation of the real world. Again, reality is even if you test your upstream to this extent, you're probably mocking a bunch of things, and that may mask something in a way you won't see until possibly production use.
- Swizec 8y ago
  > While it's easy to say, how far do you go? Do you test every bit of every upstream library you use? The ideal is probably yes, but the reality is this rarely happens. It depends. I have a friend who used to work on banking systems. They had full test coverage of every dependency. Even standard lib functions and language features. One time they found a bug in the md5 implementation in a minor version of a popular database.
  - pavel_lishin 8y ago
    Tests Georg is an outlier, and should not have been counted.
  - nostalgeek 8y ago
    > It depends. I have a friend who used to work on banking systems. They had full test coverage of every dependency. Even standard lib functions and language features. these are not dependencies anymore then, they are part of your code source and should be vendored with it. I don't know what language your friend is using but I'm pretty sure most std libs and languages already have tests with very good coverage. > One time they found a bug in the md5 implementation in a minor version of a popular database. Every piece of code can have bugs. 100% code coverage doesn't eliminate bugs, it just says all code path are tested, an algorithm can still be wrong for some values even if 100% code path are tested.
    - Swizec 8y ago
      The lesson learned isn’t about code coverage or oaths tested. It’s to not blindly trust 3rd part anything, even “languages already have test with very good coverage” when the stakes are high. If billions of dollars are riding on your code, you better be damn sure you trust everything it relies on. Fun side note: every piece of internal code was always developed in parallel to the same spec by 3+ teams so they could cross validate. If all 3 functions don’t return the same value for the same input, every team gets to build it again until all implementations behave the exact same. High reliability engineering sounds “fun”

pvorb 8y ago

But isn't it impractical to test every feature of every library you are using? In an ideal world you would have everything tested in isolation as well as integration. But in practice there will always be a corner case that remains untested because you don't know all internals of the libraries you use.

CoryG89 8y ago
If you want to be able to randomly upgrade those dependencies and not have to worry about a breaking change, then yes. Semver is not going to help you there. Server is only going to help you when someone knows they are releasing breaking change. And even then, only if they are nice enough to actually follow the spec. You don't have to test every bit of every dependency you use, but upgrading them without either carefully reviewing the changes or having tests in place for at least critical functionality is asking for something like this to happen eventually.

gregmac 8y ago

While it's easy to say, how far do you go? Do you test every bit of every upstream library you use? The ideal is probably yes, but the reality is this rarely happens. Even with a test, you may not find this. In the IOException example, the author calls out why: > When we upgraded, all our tests passed (because our test fixtures emulated the old behavior and our network was not unstable enough to trigger bad conditions) The only way to catch this type of thing is to emulate the entire network side of things, and that's still only as good as your simulation of the real world. Again, reality is even if you test your upstream to this extent, you're probably mocking a bunch of things, and that may mask something in a way you won't see until possibly production use.

Swizec 8y ago
> While it's easy to say, how far do you go? Do you test every bit of every upstream library you use? The ideal is probably yes, but the reality is this rarely happens. It depends. I have a friend who used to work on banking systems. They had full test coverage of every dependency. Even standard lib functions and language features. One time they found a bug in the md5 implementation in a minor version of a popular database.
- pavel_lishin 8y ago
  Tests Georg is an outlier, and should not have been counted.
- nostalgeek 8y ago
  > It depends. I have a friend who used to work on banking systems. They had full test coverage of every dependency. Even standard lib functions and language features. these are not dependencies anymore then, they are part of your code source and should be vendored with it. I don't know what language your friend is using but I'm pretty sure most std libs and languages already have tests with very good coverage. > One time they found a bug in the md5 implementation in a minor version of a popular database. Every piece of code can have bugs. 100% code coverage doesn't eliminate bugs, it just says all code path are tested, an algorithm can still be wrong for some values even if 100% code path are tested.
  - Swizec 8y ago
    The lesson learned isn’t about code coverage or oaths tested. It’s to not blindly trust 3rd part anything, even “languages already have test with very good coverage” when the stakes are high. If billions of dollars are riding on your code, you better be damn sure you trust everything it relies on. Fun side note: every piece of internal code was always developed in parallel to the same spec by 3+ teams so they could cross validate. If all 3 functions don’t return the same value for the same input, every team gets to build it again until all implementations behave the exact same. High reliability engineering sounds “fun”