4 ms·

You don't. Testing third-party code is beyond the purview of first-party tests. If you have first-party code that is adjacent to a third-party library in a way

You don't. Testing third-party code is beyond the purview of first-party tests. If you have first-party code that is adjacent to a third-party library in a way that the library hinders testing of that first-party code, you would substitute a (minimal) implementation that you control in for the third-party library while under test. If the library is open source or similarly code accessible, you would fork it or coordinate control with its stakeholders to see the testing of it in that codebase (and refactor it as necessary to support that). This is where the library needs to ensure it does what the documentation claims. It is not the concern of your application. Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using.

MaulingMonkey 3y ago
> You don't. Testing third-party code is beyond the purview of first-party tests. Testing third-party code is often well within the purview of first-party integration testing. Aside from validating your code, and your understanding of the documentation, these sometimes incidentally catch third-party bugs. Working around bugs until upstream fixes their code may also be within the purview of first-party code... and perhaps catching regressions even if they've supposedly been fixed. > Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using. We unit test our code because we can't even trust ourselves - nevermind our coworkers, nevermind third party vendors. Granted, unit tests for third party libraries are often worth upstreaming, and you may trust upstream enough to delete your downstream copies of those unit tests once you do...
- randomdata 3y ago
  > Testing third-party code is often well within the purview of first-party integration testing. Not really. While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented.
  - dragonwriter 3y ago
    > The value proposition of testing is that it documents [...] No, the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Documentation of anything is not the central value proposition of testing.
    - randomdata 3y ago
      > the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Incorrect with respect to the documentation, yes. That's what I just said. Tests are not intended to find undocumented incorrect behaviour. This is provable by taking it to the logical extreme of a codebase that is completely undocumented (no tests). When no tests run, no incorrect behaviour will be uncovered by it. Only documented cases can reveal instances where the code does not conform to what is documented.
      - tourist2d 3y ago
        [dead]
    - danielovichdk 3y ago
      I would urge anyone to read this book on unit testing, which is reviewed in this Gist. https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb5952c https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb...
    - wharvle 3y ago
      My experience has been that the documentation value of a good test suite is its primary value, by a mile. It’s documentation a machine can validate, which is the best kind of documentation. (Static types are documentation a machine can validate, that also happen to be very good communication tools)
  - throwbadubadu 3y ago
    Why cant you test third-party code the same as you ideally test first party, by the interface and spec, as black box? Did that, and most often then just happens implicitly anyway in integration testing? Because your code uses third-party as specced? Confused.
- unscaled 3y ago
  > Testing third-party code is often well within the purview of first-party integration testing. Yes, integration tests do tend to cover integration with third party libraries and even entire products (such as databases). But even in integration tests, the third party code is incidental. For instance, when created integration tests for databases, it's very common to use an embedded or in-memory DB such as SQLite or H2. You want to test integration with third-party modules when it is possible and "cheap", but the highest priorities are testing first-party module integration and having tests than can run fast without requiring a full-fledged replica of our production environment. If I come back to the GP statement "this only works for code you control", they either meant that you can't use this this techniques unit tests of third-party code or that you can't use this technique in integration tests of third-party code. Either way, it doesn't make sense. You cannot and should not unit test code you don't own, and you're not supposed use mocks (like a mocked clock) in integration tests.
  - MoreQARespect 3y ago
    >Yes, integration tests do tend to cover integration with third party libraries and even entire products (such as databases). But even in integration tests, the third party code is incidental. It definitely shouldnt be. At least, not if you want your tests to tell you when upgrading a 3rd party dependency will break something. >For instance, when created integration tests for databases, it's very common to use an embedded or in-memory DB I worked on a project that did this once and they very quickly got blocked writing a test by the in memory database not supporting a feature postgres had. Result? "Oh well, I guess we dont write an automated test for that." Manual QA's problem now. That's slow. Realism matters. Sacrificing realism for speed often means you will get neither. >you're not supposed use mocks (like a mocked clock) I do do this and it usually works well. Why am I wrong?
    - ivan_gammel 3y ago
      >>I worked on a project that did this once and they very quickly got blocked writing a test by the in memory database not supporting a feature postgres had. >Realism matters It is often just a small subset of tests which have to be very expensive to build, maintain and run because of an external dependency that cannot be mocked. Mocks work for 95% of use cases and should be used even if it is not 100%.
- OJFord 3y ago
  > Testing third-party code is often well within the purview of first-party integration testing. Emphasis mine. Exactly? Not unit testing as OP is about and GP claimed (rightly imo) doesn't cover it. Unless we mean offline integration testing as a mid-tier before system testing; in which case I'd probably mock third party services (and others of my own not under test) personally.
  - randomdata 3y ago
    Assuming 'integration test' is being said under a common definition, and not one being made up on the spot, the integration points don't expose third-party code in way that you could even begin to explicitly test them. it is fundamentally impossible. If it were possible, you would have no reason to be writing software – the third-party already did the work for you.
    - OJFord 3y ago
      I'm not making up anything on the spot, it's massively overloaded/differently used by different people. At it's 'lowest' level it's the integration of multiple units: basically anything that covers multiple functions. The line's also drawn (with the above still being a 'unit test') at the API: so if you exercise your handler code like `views.Widget.post(...)` then it's unit; if you do `requests.post("/api/widget")` it's integration. And at it's 'highest' level it's used to mean system or end to end testing (depending which of those you you use to mean automated testing of a deployed system).
      - deleted 3y ago
        [deleted]
      - randomdata 3y ago
        > At it's 'lowest' level it's the integration of multiple units: basically anything that covers multiple functions. So, testing...? test_unit_1 { assert(foo()) } test_unit_2 { assert(bar()) } That's not a use of 'integration' that anyone would ever find useful. If they do, they must not be software developers. 'Integration' adds nothing. But, if this is the definition you have made up on the spot, then, yes, you could use this to explicitly test a third-party library. It is no surprise that you can explicitly test a third-party library using testing. test_unit_1 { assert(third_party_library.foo()) } But even then, it is not clear why you would meld that into your application's project and not its own project? It has absolutely nothing to do with your first-party application. Logically, that kind of test is best kept in the third-party library's own project. -- Perhaps you misspoke and meant a single unit that covers multiple functions? test_unit_1 { write() assert(read()) } But that is also just testing... As soon as you have internal mutable state (every program that does something; not even the FP diehards are able to completely avoid internal mutable state), you are always going to have to call at least two functions – one to mutate the state, another to observe the state afterwards. Here too, 'integration' adds nothing. There is no practical situation where you would ever communicate this as being distinct from the above. If this is it, it is also clearly made up on the spot. -- The remaining two I guess say something, albeit flimsily, but obviously do not allow explicit testing of third-party code. Take the second case, since it provides some concrete code. Show us how you would update that code to explicitly test some third-party library. Remember that you said it calls "your handler code". I will wait.
crabbone 3y ago
Unless you are working directly with something like reading the signals off the clock on the motherboard you are bound to deal with third-party code when dealing with clocks. What's the point of your home-made timer if it doesn't faithfully emulate the actual clock that's going to be used in your actual program? How will you have confidence that your tests are even doing anything useful? There are plenty of things that might be out of your control. If you try to emulate time. For example, the system, when using a realistic timer may decide to do some context switches, seeing how your code is blocking waiting on some wakeup event, while if you test against a timer that tries to speed things up those context switches won't happen. And there could be plenty examples of similar things. If not, then how are you going to implement your home-made timer? Is your home-made timer going to be an actual timer (i.e. it will have to read time, thus relying on third-party code to do it) or will it pretend that time doesn't exist (and thus test something different from what your code is actually supposed to do)? Your ideas about "substituting" system clock with hand-written code are out of reach for most developers out there. Even those who theoretically could embark on such a task probably won't because that's just too much effort. What I'm getting at: the whole idea of mocking time for testing is a fool's errand. You'll be testing things that you don't actually need and won't be testing that actually need testing.
- deleted 3y ago
  [deleted]
- randomdata 3y ago
  Huh? I take it that you pressed the wrong reply button. Perhaps you meant to reply to this [https://news.ycombinator.com/item?id=38704899 https://news.ycombinator.com/item?id=38704899] comment?

MaulingMonkey 3y ago

> You don't. Testing third-party code is beyond the purview of first-party tests. Testing third-party code is often well within the purview of first-party integration testing. Aside from validating your code, and your understanding of the documentation, these sometimes incidentally catch third-party bugs. Working around bugs until upstream fixes their code may also be within the purview of first-party code... and perhaps catching regressions even if they've supposedly been fixed. > Otherwise, if the code is truly beyond your reach, you just have to trust that the vendor's claims are true. If you cannot establish that trust, use a different library that you can trust. Testing concerns are the least of your problems when you have no trust for the library you are using. We unit test our code because we can't even trust ourselves - nevermind our coworkers, nevermind third party vendors. Granted, unit tests for third party libraries are often worth upstreaming, and you may trust upstream enough to delete your downstream copies of those unit tests once you do...

randomdata 3y ago
> Testing third-party code is often well within the purview of first-party integration testing. Not really. While you may end up testing third-party code incidentally in the process of you testing your first-party code, explicit tests directed at third-party code necessarily requires testing implementation details, and testing implementation details is how you get constantly breaking tests every time you try to change something and developers giving up on testing in frustration. The value proposition of testing is that it documents the user interface in a way that is independent of implementation, allowing you to continue to iterate on the implementation while having assurances that, no matter what you do under the hood, the user experience does not deviate from what is documented.
- dragonwriter 3y ago
  > The value proposition of testing is that it documents [...] No, the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Documentation of anything is not the central value proposition of testing.
  - randomdata 3y ago
    > the value proposition of testing is that it identifies incorrect system behavior before it manifests in live use. Incorrect with respect to the documentation, yes. That's what I just said. Tests are not intended to find undocumented incorrect behaviour. This is provable by taking it to the logical extreme of a codebase that is completely undocumented (no tests). When no tests run, no incorrect behaviour will be uncovered by it. Only documented cases can reveal instances where the code does not conform to what is documented.
    - tourist2d 3y ago
      [dead]
  - danielovichdk 3y ago
    I would urge anyone to read this book on unit testing, which is reviewed in this Gist. https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb5952c https://gist.github.com/gniemann/adaf12895c22eb5c11c0591f8cb...
  - wharvle 3y ago
    My experience has been that the documentation value of a good test suite is its primary value, by a mile. It’s documentation a machine can validate, which is the best kind of documentation. (Static types are documentation a machine can validate, that also happen to be very good communication tools)
- throwbadubadu 3y ago
  Why cant you test third-party code the same as you ideally test first party, by the interface and spec, as black box? Did that, and most often then just happens implicitly anyway in integration testing? Because your code uses third-party as specced? Confused.
unscaled 3y ago
> Testing third-party code is often well within the purview of first-party integration testing. Yes, integration tests do tend to cover integration with third party libraries and even entire products (such as databases). But even in integration tests, the third party code is incidental. For instance, when created integration tests for databases, it's very common to use an embedded or in-memory DB such as SQLite or H2. You want to test integration with third-party modules when it is possible and "cheap", but the highest priorities are testing first-party module integration and having tests than can run fast without requiring a full-fledged replica of our production environment. If I come back to the GP statement "this only works for code you control", they either meant that you can't use this this techniques unit tests of third-party code or that you can't use this technique in integration tests of third-party code. Either way, it doesn't make sense. You cannot and should not unit test code you don't own, and you're not supposed use mocks (like a mocked clock) in integration tests.
- MoreQARespect 3y ago
  >Yes, integration tests do tend to cover integration with third party libraries and even entire products (such as databases). But even in integration tests, the third party code is incidental. It definitely shouldnt be. At least, not if you want your tests to tell you when upgrading a 3rd party dependency will break something. >For instance, when created integration tests for databases, it's very common to use an embedded or in-memory DB I worked on a project that did this once and they very quickly got blocked writing a test by the in memory database not supporting a feature postgres had. Result? "Oh well, I guess we dont write an automated test for that." Manual QA's problem now. That's slow. Realism matters. Sacrificing realism for speed often means you will get neither. >you're not supposed use mocks (like a mocked clock) I do do this and it usually works well. Why am I wrong?
  - ivan_gammel 3y ago
    >>I worked on a project that did this once and they very quickly got blocked writing a test by the in memory database not supporting a feature postgres had. >Realism matters It is often just a small subset of tests which have to be very expensive to build, maintain and run because of an external dependency that cannot be mocked. Mocks work for 95% of use cases and should be used even if it is not 100%.
OJFord 3y ago
> Testing third-party code is often well within the purview of first-party integration testing. Emphasis mine. Exactly? Not unit testing as OP is about and GP claimed (rightly imo) doesn't cover it. Unless we mean offline integration testing as a mid-tier before system testing; in which case I'd probably mock third party services (and others of my own not under test) personally.
- randomdata 3y ago
  Assuming 'integration test' is being said under a common definition, and not one being made up on the spot, the integration points don't expose third-party code in way that you could even begin to explicitly test them. it is fundamentally impossible. If it were possible, you would have no reason to be writing software – the third-party already did the work for you.
  - OJFord 3y ago
    I'm not making up anything on the spot, it's massively overloaded/differently used by different people. At it's 'lowest' level it's the integration of multiple units: basically anything that covers multiple functions. The line's also drawn (with the above still being a 'unit test') at the API: so if you exercise your handler code like `views.Widget.post(...)` then it's unit; if you do `requests.post("/api/widget")` it's integration. And at it's 'highest' level it's used to mean system or end to end testing (depending which of those you you use to mean automated testing of a deployed system).
    - deleted 3y ago
      [deleted]
    - randomdata 3y ago
      > At it's 'lowest' level it's the integration of multiple units: basically anything that covers multiple functions. So, testing...? test_unit_1 { assert(foo()) } test_unit_2 { assert(bar()) } That's not a use of 'integration' that anyone would ever find useful. If they do, they must not be software developers. 'Integration' adds nothing. But, if this is the definition you have made up on the spot, then, yes, you could use this to explicitly test a third-party library. It is no surprise that you can explicitly test a third-party library using testing. test_unit_1 { assert(third_party_library.foo()) } But even then, it is not clear why you would meld that into your application's project and not its own project? It has absolutely nothing to do with your first-party application. Logically, that kind of test is best kept in the third-party library's own project. -- Perhaps you misspoke and meant a single unit that covers multiple functions? test_unit_1 { write() assert(read()) } But that is also just testing... As soon as you have internal mutable state (every program that does something; not even the FP diehards are able to completely avoid internal mutable state), you are always going to have to call at least two functions – one to mutate the state, another to observe the state afterwards. Here too, 'integration' adds nothing. There is no practical situation where you would ever communicate this as being distinct from the above. If this is it, it is also clearly made up on the spot. -- The remaining two I guess say something, albeit flimsily, but obviously do not allow explicit testing of third-party code. Take the second case, since it provides some concrete code. Show us how you would update that code to explicitly test some third-party library. Remember that you said it calls "your handler code". I will wait.

crabbone 3y ago

Unless you are working directly with something like reading the signals off the clock on the motherboard you are bound to deal with third-party code when dealing with clocks. What's the point of your home-made timer if it doesn't faithfully emulate the actual clock that's going to be used in your actual program? How will you have confidence that your tests are even doing anything useful? There are plenty of things that might be out of your control. If you try to emulate time. For example, the system, when using a realistic timer may decide to do some context switches, seeing how your code is blocking waiting on some wakeup event, while if you test against a timer that tries to speed things up those context switches won't happen. And there could be plenty examples of similar things. If not, then how are you going to implement your home-made timer? Is your home-made timer going to be an actual timer (i.e. it will have to read time, thus relying on third-party code to do it) or will it pretend that time doesn't exist (and thus test something different from what your code is actually supposed to do)? Your ideas about "substituting" system clock with hand-written code are out of reach for most developers out there. Even those who theoretically could embark on such a task probably won't because that's just too much effort. What I'm getting at: the whole idea of mocking time for testing is a fool's errand. You'll be testing things that you don't actually need and won't be testing that actually need testing.

deleted 3y ago
[deleted]
randomdata 3y ago
Huh? I take it that you pressed the wrong reply button. Perhaps you meant to reply to this [https://news.ycombinator.com/item?id=38704899 https://news.ycombinator.com/item?id=38704899] comment?