5 ms·

Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage

• lmilcin 5y ago
  It is very unpopular view. I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented. This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client. Since the outside interface of the application is less likely to change compared to the implementation, the tests tend to be more stable and require less maintenance over life of the application. Unit tests were supposed to help refactoring (by making it easier to ensure the code still works after change). The sad reality is that I can never trust unit tests. I still need to research the code around the function I am modifying to be sure I am not breaking anything. Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like. This absolutely precludes writing unit tests up front. But also, after I have spend so much time polishing the code, writing unit tests for it is absolutely the last thing I want to do. There is absolutely no possibility that people will put effort into writing tests well if they did good job writing the actual implementation. Yet another reason why unit tests are broken is that there is no mechanism, no feedback loop to ensure quality of tests. When you write application code, if it is broken you will have feedback in the form of defects and outages or various other problems. One reason that documentation is usually broken is because there typically does not exist mechanism that would ensure that documents stay in sync with design and implementation. The same problem with unit tests. The only strong signal you get is when the test does not work. But there is no good signal to fix real important problems like missing tests for important part of the contract.
  • hsn915 5y ago
    > Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like. Same here. I do think there's value in some unit testing: mostly for small units whose behavior is a bit tricky to verify by just looking at the code or running the application. These are usually functions with tricky mathematical expressions, or something like that.
  • ffhhj 5y ago
    So true. Testing and documentation "freeze" a solution. Going too deep with them from the beginning has a similar effect to early optimization. There is an economic factor in which code is mature enough to invest in testing/optimization/documentation, before that point effort is wasted.
  • ummonk 5y ago
    Agreed. In general, I prefer integration tests, especially end to end tests. The one exception is for utility functions. I advocate TDD for utility functions, with heavy testing of standard and edge cases, to ensure that the functions conform to spec and any future tweaks / refactors of the functions continue to conform to spec.
• bronson 5y ago
  3. Your code base may start to become contorted. I've seen good programmers create bogus classes to allow test-time mocking, or add oddball env vars and configurations to let the test harness manually reach every last line. Even if that line is not worth testing: if(!(x=malloc(BUF_SIZ)) || ENV[TEST_MEM_FAIL_12]) { exit(1); } Tying code and tests this tightly discourages refactoring. Another example: a different code base that I helped on had some insanely long one-line conditionals that anyone would break into multiple lines ... but doing that would have blown the "100%" coverage.
  • hsn915 5y ago
    "Mocking" is such a weird thing to me and I don't think it serves a good purpose. It's the kind of thing that would only arise if you assume a-priori that 100% test coverage is a non-negotiable must. If you have a function A that calls B to get some data (by doing I/O) then process it using C, then the 100% cov rule would force you to mock B when you test A. But then what is the value of this test? What guarantees is it giving you? You are basically testing the internals of A, you are testing the implementation details. You are testing that calling A is the same as calling B and passing the result to C.
    • berkes 5y ago
      Mocking helps with architecture, though. When A calls C through B, you have a tight coupling between A and C, but mightn't be aware of that, because you only see A calling B. Lets say C reads a CSV file. By testing implementation only, you might 'create a csv file' then call A.import() and assert some records in, say D are created. By testing A in isolation, you make apparent that it couples to C and D because you have to mock them out. B is a direct dependency of A, so you don't mock that. At the least, you have now documented the accidental coupling. But at the best, your tests showed you a design issue.
    • chriswarbo 5y ago
      Mocking and coverage aren't that closely related, but I agree that mocking makes tests less realistic, and therefore less useful. Mocking can be helpful when that downside is still better than the alternative, e.g. if 'real' calls have significant latency, or a significant monetary cost, etc. In those cases mocking lets us do a whole lot more testing for the same amount of time/money/etc., which can make up for the loss of realism. Mocking can be especially useful when adding tests to a legacy system. Greenfield work should aim to minimise the amount of mocking required, ideally to zero.