6 ms·

> Testing third-party code is often well within the purview of first-party integration testing. Not really. While you may end up testing third-party code incid

> Testing third-party code is often well within the purview of first-party integration testing. Not really. While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented.

dragonwriter 3y ago
> The value proposition of testing is that it documents [...] No, the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Documentation of anything is not the central value proposition of testing.
- randomdata 3y ago
  > the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Incorrect with respect to the documentation, yes. That's what I just said. Tests are not intended to find undocumented incorrect behaviour. This is provable by taking it to the logical extreme of a codebase that is completely undocumented (no tests). When no tests run, no incorrect behaviour will be uncovered by it. Only documented cases can reveal instances where the code does not conform to what is documented.
  - tourist2d 3y ago
    [dead]
- danielovichdk 3y ago
  I would urge anyone to read this book on unit testing, which is reviewed in this Gist. https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb5952c https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb...
- wharvle 3y ago
  My experience has been that the documentation value of a good test suite is its primary value, by a mile. It’s documentation a machine can validate, which is the best kind of documentation. (Static types are documentation a machine can validate, that also happen to be very good communication tools)
throwbadubadu 3y ago
Why cant you test third-party code the same as you ideally test first party, by the interface and spec, as black box? Did that, and most often then just happens implicitly anyway in integration testing? Because your code uses third-party as specced? Confused.
- randomdata 3y ago
  Third-party code hidden in a black box cannot be within the purview of your first-party tests. There is no way for those tests to know that the third-party code exists. It's hidden away in a black box. If a third-party library within the black box is executed during the execution of a test, you may incidentally discovery cases where the library produces something not conformant with your documentation, but that is in no way explicit.
  - Sakos 3y ago
    I'm confused by this line of reasoning. The point of the black box is that you have clearly defined the expected input and output and don't care about the implementation details of the thing sitting inside the box. You can test both what you pass in to the black box and what you get out of it. And it's incredibly useful and important to do so. If you aren't testing third party libraries (or other dependencies) in this way, you're opening yourself up to a lot of unnecessary risk. Those assumptions about your expected output are always there, whether you define and test them or not. So if for some reason the library changes its output unexpectedly and you aren't explicitly testing your assumptions, it can easily happen that you won't find out until it manifests in a production bug.
    - randomdata 3y ago
      > I'm confused by this line of reasoning. That's because you haven't read the discussion. Why would you expect anything else? Your comment straight up reiterates what was already said. For example, "The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented." Explicitly testing third-party code introduces testing of implementation details. It is impossible to explicitly test another codebase without knowing of its existence. But it is not an area of concern for a first-party application to worry about. The third-party should already have those tests in the third-party codebase.
MaulingMonkey 3y ago
> Not really. Yes really. > While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. Depends. I'm certainly not advocating testing implementation details - 3rd party or 1st party - that your codebase doesn't rely upon. That is brittle as you say. But I do advocate for being willing to test anything you rely on - be that documented, hinted at, undocumented, or even explicitly warned against assuming if for some horrible reason you have a terrible need to make such assumptions anyways. 1st party or 3rd party. When so written, if the tests are brittle, the codebase is brittle, and the tests correctly identify that needs to be fixed. Deleting or not writing the tests won't fix the problem - it'll merely shove the problem into production. Amelioration might include carefully controlling and vetting updates, switching libraries, writing your own, rewriting your code to make fewer assumptions... or perhaps just yeeting the entire feature relying upon it out of your codebase, if you're desperate enough. > The value proposition of testing is that it documents the user interface in a way that is independent of implementation That is a value proposition, but not the only one. Others include simplifying debugging when you break things, when other people break things, and catching bugs before they go live instead of after (even if it'd be trivial to fix when someone belatedly notices.) Fuzzing-generated regression tests are often unreadable garbage when it comes to the purpouses of "documentation", a cargo cult of redundancies and red herrings and voodoo that, once, caused a crash. Another value proposition - and I do find value in this in bugs caught in my code, their code, and their documentation - is "documenting" my understanding of upstream documentation, for which upstream tests alone will be useless. After all, that verified someone else's understanding, not mine. And it turns out this is important, because the documentation is outdated, the documentation lies, the documentation is insufficient, the documentation didn't consider that edge case, and the documentation foolishly presupposes common sense. Anyone telling you otherwise has a bridge to sell. Even worse: the documentation may be technically correct... but misleading. Can't even blame the author - it made sense to them! > allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented. Such iteration might include updating third party dependencies. I do this frequently. Poor test coverage means heisenbugs and fear around updates. Good test coverage might explicitly compare two different backends, or different versions of the same backend - 100% implementation details - and ensure the publicly visible behavior of my own APIs remains identical when switching between them. This means knowing which versions of which third party libraries to forbid or write workarounds for. Such tests should make no stupid assumptions, but should absolutely test for sane assumptions. Such iteration might include upgrading compilers. I do this frequently. We've had unit tests catch codegen bugs. This is good.

dragonwriter 3y ago

> The value proposition of testing is that it documents [...] No, the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Documentation of anything is not the central value proposition of testing.

randomdata 3y ago
> the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Incorrect with respect to the documentation, yes. That's what I just said. Tests are not intended to find undocumented incorrect behaviour. This is provable by taking it to the logical extreme of a codebase that is completely undocumented (no tests). When no tests run, no incorrect behaviour will be uncovered by it. Only documented cases can reveal instances where the code does not conform to what is documented.
- tourist2d 3y ago
  [dead]
danielovichdk 3y ago
I would urge anyone to read this book on unit testing, which is reviewed in this Gist. https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb5952c https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb...
wharvle 3y ago
My experience has been that the documentation value of a good test suite is its primary value, by a mile. It’s documentation a machine can validate, which is the best kind of documentation. (Static types are documentation a machine can validate, that also happen to be very good communication tools)

throwbadubadu 3y ago

Why cant you test third-party code the same as you ideally test first party, by the interface and spec, as black box? Did that, and most often then just happens implicitly anyway in integration testing? Because your code uses third-party as specced? Confused.

randomdata 3y ago
Third-party code hidden in a black box cannot be within the purview of your first-party tests. There is no way for those tests to know that the third-party code exists. It's hidden away in a black box. If a third-party library within the black box is executed during the execution of a test, you may incidentally discovery cases where the library produces something not conformant with your documentation, but that is in no way explicit.
- Sakos 3y ago
  I'm confused by this line of reasoning. The point of the black box is that you have clearly defined the expected input and output and don't care about the implementation details of the thing sitting inside the box. You can test both what you pass in to the black box and what you get out of it. And it's incredibly useful and important to do so. If you aren't testing third party libraries (or other dependencies) in this way, you're opening yourself up to a lot of unnecessary risk. Those assumptions about your expected output are always there, whether you define and test them or not. So if for some reason the library changes its output unexpectedly and you aren't explicitly testing your assumptions, it can easily happen that you won't find out until it manifests in a production bug.
  - randomdata 3y ago
    > I'm confused by this line of reasoning. That's because you haven't read the discussion. Why would you expect anything else? Your comment straight up reiterates what was already said. For example, "The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented." Explicitly testing third-party code introduces testing of implementation details. It is impossible to explicitly test another codebase without knowing of its existence. But it is not an area of concern for a first-party application to worry about. The third-party should already have those tests in the third-party codebase.

MaulingMonkey 3y ago

> Not really. Yes really. > While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. Depends. I'm certainly not advocating testing implementation details - 3rd party or 1st party - that your codebase doesn't rely upon. That is brittle as you say. But I do advocate for being willing to test anything you rely on - be that documented, hinted at, undocumented, or even explicitly warned against assuming if for some horrible reason you have a terrible need to make such assumptions anyways. 1st party or 3rd party. When so written, if the tests are brittle, the codebase is brittle, and the tests correctly identify that needs to be fixed. Deleting or not writing the tests won't fix the problem - it'll merely shove the problem into production. Amelioration might include carefully controlling and vetting updates, switching libraries, writing your own, rewriting your code to make fewer assumptions... or perhaps just yeeting the entire feature relying upon it out of your codebase, if you're desperate enough. > The value proposition of testing is that it documents the user interface in a way that is independent of implementation That is a value proposition, but not the only one. Others include simplifying debugging when you break things, when other people break things, and catching bugs before they go live instead of after (even if it'd be trivial to fix when someone belatedly notices.) Fuzzing-generated regression tests are often unreadable garbage when it comes to the purpouses of "documentation", a cargo cult of redundancies and red herrings and voodoo that, once, caused a crash. Another value proposition - and I do find value in this in bugs caught in my code, their code, and their documentation - is "documenting" my understanding of upstream documentation, for which upstream tests alone will be useless. After all, that verified someone else's understanding, not mine. And it turns out this is important, because the documentation is outdated, the documentation lies, the documentation is insufficient, the documentation didn't consider that edge case, and the documentation foolishly presupposes common sense. Anyone telling you otherwise has a bridge to sell. Even worse: the documentation may be technically correct... but misleading. Can't even blame the author - it made sense to them! > allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented. Such iteration might include updating third party dependencies. I do this frequently. Poor test coverage means heisenbugs and fear around updates. Good test coverage might explicitly compare two different backends, or different versions of the same backend - 100% implementation details - and ensure the publicly visible behavior of my own APIs remains identical when switching between them. This means knowing which versions of which third party libraries to forbid or write workarounds for. Such tests should make no stupid assumptions, but should absolutely test for sane assumptions. Such iteration might include upgrading compilers. I do this frequently. We've had unit tests catch codegen bugs. This is good.