3 ms·
The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically
by mannykannot 9y ago
The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. In reality, this approach would be useless even for functionally-invisible refactoring or rewriting, precisely because it would be at the unit level. For that, let alone any substantive changes, Coplien's claim that higher-level tests are better stands, so this article cannot be dismissed as "beating a dead horse."
- alkonaut 9y ago> The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. I don't think that conclusion is any more valid than the proposition that if we just skipped the developer alltogether and generated both the business code AND the test, we'd have something of value. > Coplien's claim that higher-level tests are better stands, so this article cannot be dismissed as "beating a dead horse." i think higher level tests are good, but I don't think that makes lower level tests bad. I think there should usually be a health mix, with systems tested on several levels.
- mannykannot 9y ago> I don't think that conclusion is any more valid than the proposition that if we just skipped the developer alltogether and generated both the business code AND the test, we'd have something of value. That would have some value, as evidenced by all the useful spreadsheets produced by end users, but the the value of the conjunction ("the business code AND the test") lies entirely in the 'business code' part, which is why we do not automatically generate unit tests for spreadsheet macros, or any other code for that matter. >I think higher level tests are good, but I don't think that makes lower level tests bad. That's not my point; my point is that unit tests are not automatically useful. Once we accept that, Coplien's argument that there are better ways to spend our limited resources cannot be dismissed (and resources are always limited, because of the combinatorial explosion, which is most evident at the unit level).
- ekidd 9y ago> The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. Actually, this is a surprisingly useful strategy! There are two common versions of this idea: 1. Invariant testing with random data. This usually goes by the name of "QuickCheck", and it allows writing tests that say things like, "If we reverse a random list twice, then it should equal the original list." In practice, this catches tons of corner case bugs. 2. Fuzz testing. In this case, the invariant is, "No matter how broken the input, my program should never crash (or corrupt the heap, or access uninitialized memory, or whatever)." Then you generate a billion random test cases and see what happens. This usually finds tons of bugs, and it's a staple of modern security testing. So yeah, even random test cases are very valuable. :-)
- mannykannot 9y agoThese are indeed useful strategies, but they do not invalidate the point I am making here. Fuzz testing is system-level testing, not unit tests, and so is along the lines Coplien is proposing (especially when combined with assertions in the code.) Invariant testing implies an invariant, i.e. an interface-level design contract, while the post I was replying to was predicated on systems where "the code is the spec", and my reply considers only automated unit-test generation that is based only on what the code does, not what it is supposed, in more abstract terms, to do. The "logical conclusion" is based on the possibility of automatically generating unit test cases that function as such, up to any given level of coverage, without adding value.
- ekidd 9y agoOK, just to back up a step, here's my best argument in favor of automated testing: Virtually nobody just makes a change and ships it users without some kind of testing. Maybe you run your program by hand and mess around with the UI. Maybe you call the modified function from a listener with a few arguments. Maybe you have sadistic paid testers who spend 8 weeks hammering on each version before you print DVDs. (And those testers will inevitably write a "test plan" containing things to try with each new release.) The goal of automated testing is to take all those things you were going to do anyway (calling functions, messing with the UI, following a test plan), and automate them. Different kinds of testing result in different kinds of tests: 1. Manually calling a function can be replaced by unit tests, or even better, doc tests. Everybody loves accurate documentation showing how to use an API! 2. Manually messing around with the UI can be replaced by integration tests. 3. Certain things are much faster to test thoroughly in isolation, so you may mock out parts of your system while testing other parts in great detail, then tie everything together with a few system-wide tests. People love to invent elaborate taxonomies of testing (unit, request, system, integration, property, etc.). But really, they're all just variations on the same idea: You'd never ship code without testing it somehow, and it's boring and error-prone to do all that testing by hand every time. So automate it!