14 ms·

This is good advice not just for compilers, but for any kind of software. Test the behavior of the public interface, not of the internals. Doing the latter too

by matrss 2y ago

This is good advice not just for compilers, but for any kind of software. Test the behavior of the public interface, not of the internals. Doing the latter too much will just unnecessarily lock you into one implementation of your software, without much gain. As examples: If you are creating a web API, only test against the exposed public routes of that API, instead of writing tests for internal helpers. If you are creating a GUI application, programmatically exercise the GUI instead of starting your tests halfway into the innards that respond to the GUI buttons. Tests lock behavior into place, so you should only write tests for behavior that needs to be locked.

pfdietz 2y ago
For the most reliable software, don't choose between different testing approaches, use them all. Testing is not like choosing an architecture or programming language for your system; the more approaches the merrier. As each testing approach experiences diminishing returns, it encourages using a different approach to start near the top of its curve. Unit testing enables hidden functionality to be tested. This can prevent future bugs when changes in the system suddenly uncloaks those functions, exposing them to higher level tests.
- matrss 2y ago
  I don't fully agree. Yes, employ as many testing approaches as possible (e2e tests, property-based testing, golden tests, whatever), but only test for behavior that you expose and want to guarantee will stay as-is. Otherwise you will get into the situation where every refactor will require a test change. Unit testing is fine, if you do it for your public interface. If you are writing a math library then sure, unit test that `add(1, 2) == 3`. But if you just have an internal helper function for that, then think about if you really want to lock its existence and behavior into place, or if that would just hinder future architectural changes. You can always test the exposed functionality that uses the helper and achieve full coverage of it that way. If you can't, then you have dead code. Of course this is all a bit more nuanced. Past a certain size it might make sense to e.g. consider one modules interface to be public for the rest of your application and test it. But you can definitely overdo it and testing every single function you write (as I've seen people unironically suggest) is very likely detrimental.
  - randomdata 2y ago
    While that is a good rule for the general case, there are exceptional circumstances where testing an internal helper function can help improve development velocity. One should not shy away from using what is useful. What is important to remember is that "public" tests are your documentation that remain for the lifetime of your application. "private" tests are throwaway. A good language will provide clear boundaries such that it is obvious which is which.
    - senbrow 2y ago
      Alternatively, if an internal function is important enough to need good coverage, it should be pulled out into an internal "library" that exposes the interface explicitly (even if this is just a separate file or folder with limited visibility to the rest of the codebase). Testing internals is almost always a code organization smell IMO.
      - randomdata 2y ago
        If you seek good coverage, you undoubtedly would be better off moving that against the public interface. "private" tests are more for like when you're having trouble figuring out an edge case failure and want to narrow it down to a specific helper function to aid debugging or if you need help coming up with the right design for an internal function. As before, we're talking exceptional circumstances. Rarely would you need such a thing. But if it helps, no need to fear it. Either way, I'm not sure you would be looking for good coverage, only the bare necessities to reach the goal. Once settled, the tests are disposable. Organizing your project into internal libraries in case you encounter a debugging problem in need of assistance, for example, is extreme overkill.
        senbrow 2y ago
        Hmm, I guess I don't really consider throwaway assertions in pursuit of debugging "tests" in the typical sense. I certainly wouldn't advocate for code reorganization for that case, but if there is a property that is important to maintain over time that isn't easily expressed by exercising the public API, it does suggest that reorganization is probably in order.

by matrss 2y ago

pfdietz 2y ago

For the most reliable software, don't choose between different testing approaches, use them all. Testing is not like choosing an architecture or programming language for your system; the more approaches the merrier. As each testing approach experiences diminishing returns, it encourages using a different approach to start near the top of its curve. Unit testing enables hidden functionality to be tested. This can prevent future bugs when changes in the system suddenly uncloaks those functions, exposing them to higher level tests.

matrss 2y ago
I don't fully agree. Yes, employ as many testing approaches as possible (e2e tests, property-based testing, golden tests, whatever), but only test for behavior that you expose and want to guarantee will stay as-is. Otherwise you will get into the situation where every refactor will require a test change. Unit testing is fine, if you do it for your public interface. If you are writing a math library then sure, unit test that `add(1, 2) == 3`. But if you just have an internal helper function for that, then think about if you really want to lock its existence and behavior into place, or if that would just hinder future architectural changes. You can always test the exposed functionality that uses the helper and achieve full coverage of it that way. If you can't, then you have dead code. Of course this is all a bit more nuanced. Past a certain size it might make sense to e.g. consider one modules interface to be public for the rest of your application and test it. But you can definitely overdo it and testing every single function you write (as I've seen people unironically suggest) is very likely detrimental.
- randomdata 2y ago
  While that is a good rule for the general case, there are exceptional circumstances where testing an internal helper function can help improve development velocity. One should not shy away from using what is useful. What is important to remember is that "public" tests are your documentation that remain for the lifetime of your application. "private" tests are throwaway. A good language will provide clear boundaries such that it is obvious which is which.
  - senbrow 2y ago
    Alternatively, if an internal function is important enough to need good coverage, it should be pulled out into an internal "library" that exposes the interface explicitly (even if this is just a separate file or folder with limited visibility to the rest of the codebase). Testing internals is almost always a code organization smell IMO.
    - randomdata 2y ago
      If you seek good coverage, you undoubtedly would be better off moving that against the public interface. "private" tests are more for like when you're having trouble figuring out an edge case failure and want to narrow it down to a specific helper function to aid debugging or if you need help coming up with the right design for an internal function. As before, we're talking exceptional circumstances. Rarely would you need such a thing. But if it helps, no need to fear it. Either way, I'm not sure you would be looking for good coverage, only the bare necessities to reach the goal. Once settled, the tests are disposable. Organizing your project into internal libraries in case you encounter a debugging problem in need of assistance, for example, is extreme overkill.
      - senbrow 2y ago
        Hmm, I guess I don't really consider throwaway assertions in pursuit of debugging "tests" in the typical sense. I certainly wouldn't advocate for code reorganization for that case, but if there is a property that is important to maintain over time that isn't easily expressed by exercising the public API, it does suggest that reorganization is probably in order.