4 ms·

Why cant you test third-party code the same as you ideally test first party, by the interface and spec, as black box? Did that, and most often then just happens

Why cant you test third-party code the same as you ideally test first party, by the interface and spec, as black box? Did that, and most often then just happens implicitly anyway in integration testing? Because your code uses third-party as specced? Confused.

randomdata 3y ago
Third-party code hidden in a black box cannot be within the purview of your first-party tests. There is no way for those tests to know that the third-party code exists. It's hidden away in a black box. If a third-party library within the black box is executed during the execution of a test, you may incidentally discovery cases where the library produces something not conformant with your documentation, but that is in no way explicit.
- Sakos 3y ago
  I'm confused by this line of reasoning. The point of the black box is that you have clearly defined the expected input and output and don't care about the implementation details of the thing sitting inside the box. You can test both what you pass in to the black box and what you get out of it. And it's incredibly useful and important to do so. If you aren't testing third party libraries (or other dependencies) in this way, you're opening yourself up to a lot of unnecessary risk. Those assumptions about your expected output are always there, whether you define and test them or not. So if for some reason the library changes its output unexpectedly and you aren't explicitly testing your assumptions, it can easily happen that you won't find out until it manifests in a production bug.
  - randomdata 3y ago
    > I'm confused by this line of reasoning. That's because you haven't read the discussion. Why would you expect anything else? Your comment straight up reiterates what was already said. For example, "The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented." Explicitly testing third-party code introduces testing of implementation details. It is impossible to explicitly test another codebase without knowing of its existence. But it is not an area of concern for a first-party application to worry about. The third-party should already have those tests in the third-party codebase.
  - scott_w 3y ago
    If a library changes its output unexpectedly, then either: - It’s a local library so the tests covering your code should fail - It’s changed in a way you didn’t anticipate so why would testing it directly fail when your prior test didn’t? - Your end user monitoring should light up like a Christmas tree
- wruza 3y ago
  What if we switch to the purview of first-party tests of third-party code? Feels like you’re pushing on definitions here instead of matters. Obviously, you can black box test third party code. It’s absurd to say otherwise.
  - danielovichdk 3y ago
    Sure you can. But why would you test software you didn't write or own?
  - randomdata 3y ago
    > Obviously, you can black box test third party code. It’s absurd to say otherwise. No. You cannot explicitly test code within a black box. How could you? You wouldn't even know it is there. Is it that you don't understand what 'black box' means? If you remove the explicit qualifier, then okay, but if you do that you would stupidly be having a completely different conversation to the one that is taking place here. The explicit qualifier was explicitly qualified.
- throwbadubadu 3y ago
  I have the feeling that this is a very high theoretical ebony tower view. E.g., I have a SWC that stores and reads something, and that similarly uses another SWC to do that. If I do an integration test to write and then read something, I test very explicitly at the same time that my SWC does what it specs and is expected, as well as the lower layer SWC. (And at the same time this is also then a good regression finder for when you replace the black box you maybe shouldn't know about). However, similarly I can also write more explicit tests that test the lower SWC in isolation and cover even more cases I cannot cover by just integration testing. Arguing with purview in testing, and what should be done or not, never helped me so far in my career to end up with good problem and regression finding tests, and provide well working and robust systems. The contrary, discussions and nit-picking at that level are usually a huge red flag. Nice in theory, failed in practice.
  - danielovichdk 3y ago
    You are then testing other peoples software, which is probably not what you want. In integration testing I see no issue in using a real instance of whatever dependency but I would never assert on the inner workings of that dependency - that is within the testing scope for the people that did that dependency, not you.
    - randomdata 3y ago
      > You are then testing other peoples software, which is probably not what you want. And if it is what you want, why would you clutter an unrelated application's codebase with those tests? They are tests that apply to many projects. Ideally they would go alongside the work of the third-party, but failing that, surely the are better found in a new project focused on those tests?
  - scott_w 3y ago
    I don’t see this as theoretical at all, it’s a pragmatic requirement to not explicitly test the third party tools you depend on. I’d never personally write a test which, in its entirety, calls a function owned by a third party and checks its return value and include that in my CI process. What would I do? - Write a unit test for the code that depends on this function, including the function call. I don’t need to test the third party behaviour specifically because I’m really interested in the behaviour of the entire code under test. If the code is broken, my own code shouldn’t work either. - Wrap and inject fakes for the contracted behaviour. This is usually for things where the behaviour is non-deterministic, say because of a network call. I don’t want CI failing because a third party temporarily fails. I’ve worked under these conditions before and it's really unproductive. Yes, I write tests for failure cases (network interruptions, for example). - I may encapsulate the database in my “unit,” depending on the application. The frameworks I like to use tend to integrate error handling for the DB, so I’m relaxed about network errors here.

randomdata 3y ago

Third-party code hidden in a black box cannot be within the purview of your first-party tests. There is no way for those tests to know that the third-party code exists. It's hidden away in a black box. If a third-party library within the black box is executed during the execution of a test, you may incidentally discovery cases where the library produces something not conformant with your documentation, but that is in no way explicit.

Sakos 3y ago
I'm confused by this line of reasoning. The point of the black box is that you have clearly defined the expected input and output and don't care about the implementation details of the thing sitting inside the box. You can test both what you pass in to the black box and what you get out of it. And it's incredibly useful and important to do so. If you aren't testing third party libraries (or other dependencies) in this way, you're opening yourself up to a lot of unnecessary risk. Those assumptions about your expected output are always there, whether you define and test them or not. So if for some reason the library changes its output unexpectedly and you aren't explicitly testing your assumptions, it can easily happen that you won't find out until it manifests in a production bug.
- randomdata 3y ago
  > I'm confused by this line of reasoning. That's because you haven't read the discussion. Why would you expect anything else? Your comment straight up reiterates what was already said. For example, "The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented." Explicitly testing third-party code introduces testing of implementation details. It is impossible to explicitly test another codebase without knowing of its existence. But it is not an area of concern for a first-party application to worry about. The third-party should already have those tests in the third-party codebase.
- scott_w 3y ago
  If a library changes its output unexpectedly, then either: - It’s a local library so the tests covering your code should fail - It’s changed in a way you didn’t anticipate so why would testing it directly fail when your prior test didn’t? - Your end user monitoring should light up like a Christmas tree
wruza 3y ago
What if we switch to the purview of first-party tests of third-party code? Feels like you’re pushing on definitions here instead of matters. Obviously, you can black box test third party code. It’s absurd to say otherwise.
- danielovichdk 3y ago
  Sure you can. But why would you test software you didn't write or own?
- randomdata 3y ago
  > Obviously, you can black box test third party code. It’s absurd to say otherwise. No. You cannot explicitly test code within a black box. How could you? You wouldn't even know it is there. Is it that you don't understand what 'black box' means? If you remove the explicit qualifier, then okay, but if you do that you would stupidly be having a completely different conversation to the one that is taking place here. The explicit qualifier was explicitly qualified.
throwbadubadu 3y ago
I have the feeling that this is a very high theoretical ebony tower view. E.g., I have a SWC that stores and reads something, and that similarly uses another SWC to do that. If I do an integration test to write and then read something, I test very explicitly at the same time that my SWC does what it specs and is expected, as well as the lower layer SWC. (And at the same time this is also then a good regression finder for when you replace the black box you maybe shouldn't know about). However, similarly I can also write more explicit tests that test the lower SWC in isolation and cover even more cases I cannot cover by just integration testing. Arguing with purview in testing, and what should be done or not, never helped me so far in my career to end up with good problem and regression finding tests, and provide well working and robust systems. The contrary, discussions and nit-picking at that level are usually a huge red flag. Nice in theory, failed in practice.
- danielovichdk 3y ago
  You are then testing other peoples software, which is probably not what you want. In integration testing I see no issue in using a real instance of whatever dependency but I would never assert on the inner workings of that dependency - that is within the testing scope for the people that did that dependency, not you.
  - randomdata 3y ago
    > You are then testing other peoples software, which is probably not what you want. And if it is what you want, why would you clutter an unrelated application's codebase with those tests? They are tests that apply to many projects. Ideally they would go alongside the work of the third-party, but failing that, surely the are better found in a new project focused on those tests?
- scott_w 3y ago
  I don’t see this as theoretical at all, it’s a pragmatic requirement to not explicitly test the third party tools you depend on. I’d never personally write a test which, in its entirety, calls a function owned by a third party and checks its return value and include that in my CI process. What would I do? - Write a unit test for the code that depends on this function, including the function call. I don’t need to test the third party behaviour specifically because I’m really interested in the behaviour of the entire code under test. If the code is broken, my own code shouldn’t work either. - Wrap and inject fakes for the contracted behaviour. This is usually for things where the behaviour is non-deterministic, say because of a network call. I don’t want CI failing because a third party temporarily fails. I’ve worked under these conditions before and it's really unproductive. Yes, I write tests for failure cases (network interruptions, for example). - I may encapsulate the database in my “unit,” depending on the application. The frameworks I like to use tend to integrate error handling for the DB, so I’m relaxed about network errors here.