4 ms·

I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a clas

by sulam 7y ago

I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a class may take on (true!) but then followed up with espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely”. He also implied at the very beginning that somehow this was driven by the switch from FORTRAN to OOP, as if FORTRAN avoided this combinatorial explosion (magic!) and it’s only because of polymorphism that we live in the “unit testing is good” world. The logical contradictions eventually overcame me.

danmaz74 7y ago
I saw cases where there really was a waste of stupid unit tests, and no significant tests for the interaction of those units, so I can see where the author is coming from. But my conclusion, after many years of developing with different levels of testing, is that it's a good idea to: * Have at least one unit test for each nontrivial unit, but no unit tests for the trivial ones. But, in these unit tests, use mocking and stubbing only when using real objects isn't feasible, or very inconvenient - isolation is nice to have, but generally overrated in my experience * Create as many unit tests as necessary to feel confident about the edge cases for the most critical units * Implement higher level tests covering the main use cases for the interaction of all the units
hdfbdtbcdg 7y ago
Ironically modern fortran is object orientated...
cryptica 7y ago
For the same number of lines of test code, an integration test case traverses through much more source code than a unit test case. Not only that, but it also checks that the units are interacting properly and passing each other the right data and returning the right data. Good integration tests can also fee us from static typing because the integration tests implicitly check that the interface contracts between all the different units are met internally. It also verifies many other factors which affect the robustness of the software. For example, they can identify race conditions and prevent them in the future.
- ptx 7y ago
  True, but they're also slower, so you'll have to either test fewer cases or run the tests less often.
  - cryptica 7y ago
    They are usually slower but they can still be pretty fast. Also, with integration tests, you can choose the granularity of your test. You don't need to test end-to-end (of course this is the slowest), you can just test a single module with all its dependencies. If you don't mock out dependencies, then it's still technically called an integration test.
arendtio 7y ago
Same here. That articles just mixes true facts and weird ideas without coming to a real punchline at the end. At least, I didn't take away more than 'throw away tests that haven't failed in a year' and I don't agree.
deathanatos 7y ago
I'm going to pile onto your comment, since I heartily agree. > In a given computing context, the exact function to be called is determined at run-time and cannot be deduced from the source code [in OOP languages] as it could in FORTRAN. This is not necessarily true, and I don't think this was true when this was written, either. (The Wayback Machine first saw this in 2014, and the paper doesn't date itself.) Most member function calls in C++ (non-virtual ones) can be resolved at compile-time. Where idiomatic C++ could would use a virtual function would require some equally uninspectable construction in any language, because one would use it when one needed run-time switching. Rust lacks OOP in the usual sense, but if I needed a virtual function of sorts, I might reach for a trait type, which also wouldn't be inspectable (beyond whatever semantics the trait establishes). There's Java, JS, and Python, I do admit. And I find my unit tests often fill in the static analysis I wish I had. But the truth is more nuanced than I think the author conveys. > Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it. > (Trillion is not used rhetorically here, but is based on the different possible states given that the average object size is four words, and the conservative estimate that you are using 16-bit words). This is as vapid a definition of test coverage as the line coverage the author derides in the prior paragraphs. A trivial function, fn add_these(a: u64, b: 64) -> u64 { a + b } could not be adequately unit-tested, because I could never cover the many trillions of input states it has. I think there's always an implicit line to "well, if the code under test is absolutely nuts", it's not the test's fault for not catching it, and that if the function under test (with the same contract as the function above) is errantly coded something like, fn add_these(a: u64, b: 64) -> u64 { if a == 11771008849893880921 && b == 5668622331333919113 { 0 } else { a + b } } then what amount of testing is going to catch that? > I define 100% coverage as having examined all possible combinations of all possible paths through all methods of a class, having reproduced every possible configuration of data bits accessible to those methods, at every machine language instruction along the paths of execution. While this somewhat contradicts the earlier example of states. I don't think we should aspire to this, since I think many "combinations" are not interesting to test; they're just pieces of the code that really don't interact; testing them is redundant. Yes, this hurts our ability to formally define a protocol for testing or not testing, but it prevents the inevitable combinatorical explosion. > Business people, rather than programmers, should design most functional tests. Absolutely not. A large part of my job is coming up with the actual requirements of the system from the vague machinations of business people who barely know how a computer works, let alone can specify salient requirements. How are they to write the tests, when they can't clearly articulate the requirements?¹ > Turn unit tests into assertions. But I want to ensure these don't fire under normal circumstances. We do this, in my current code base, and we get notified of them, and of any crashes in general. But it's a failure that was noticed, and might have downstream consequences, as opposed to one caught by a unit-test. Then the parts about Eastern Europeans with bad Internet make better programmers because they had to think instead of using the Internet, and how he grew up under similar circumstances. Real Programmers! ¹I'm not saying business types "should" (or should not) be able to articulate requirements/specifications.
pknopf 7y ago
> espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely” The point is that when you do integration tests, you will test the underlying classes the way in which it actually matters. You can't test every possible condition a unit can have, but you can test for the most likely.
- crdoconnor 7y ago
  This is my experience too. Realism in testing is criminally underrated while code coverage is criminally overrated. IME unit tests can only effectively substitute for integration tests where you're testing logical/algorithmic code with simple function inputs/outputs.
  - 0xB31B1B 7y ago
    I somewhat agree, somewhat disagree. IMO the largest value I get from unit tests is the confidence that I can make changes and understand what breaks. Refactoring w/o unit tests makes me feel like I am flying blind
    - closeparen 7y ago
      The units we test are almost never big enough for internal refactoring. It’s the decomposition into units that wants refactoring, and there the test suite actively fights back (mock expectations in particular). If there were enough code involved in a test that we could meaningfully refactor it while keeping the test green, we would call it an integration test.
- sulam 7y ago
  This is no more or less likely in an integration test vs a unit test. One of the first things you learn in writing unit tests is that you should be selective in picking your inputs. This is even taught in those “trade schools” he subtly derides. You regularly see both public test cases and hidden test cases run against problem sets, at least that’s what my interns tell me.

by sulam 7y ago

danmaz74 7y ago

I saw cases where there really was a waste of stupid unit tests, and no significant tests for the interaction of those units, so I can see where the author is coming from. But my conclusion, after many years of developing with different levels of testing, is that it's a good idea to: * Have at least one unit test for each nontrivial unit, but no unit tests for the trivial ones. But, in these unit tests, use mocking and stubbing only when using real objects isn't feasible, or very inconvenient - isolation is nice to have, but generally overrated in my experience * Create as many unit tests as necessary to feel confident about the edge cases for the most critical units * Implement higher level tests covering the main use cases for the interaction of all the units

hdfbdtbcdg 7y ago

Ironically modern fortran is object orientated...

cryptica 7y ago

For the same number of lines of test code, an integration test case traverses through much more source code than a unit test case. Not only that, but it also checks that the units are interacting properly and passing each other the right data and returning the right data. Good integration tests can also fee us from static typing because the integration tests implicitly check that the interface contracts between all the different units are met internally. It also verifies many other factors which affect the robustness of the software. For example, they can identify race conditions and prevent them in the future.

ptx 7y ago
True, but they're also slower, so you'll have to either test fewer cases or run the tests less often.
- cryptica 7y ago
  They are usually slower but they can still be pretty fast. Also, with integration tests, you can choose the granularity of your test. You don't need to test end-to-end (of course this is the slowest), you can just test a single module with all its dependencies. If you don't mock out dependencies, then it's still technically called an integration test.

arendtio 7y ago

Same here. That articles just mixes true facts and weird ideas without coming to a real punchline at the end. At least, I didn't take away more than 'throw away tests that haven't failed in a year' and I don't agree.

deathanatos 7y ago

I'm going to pile onto your comment, since I heartily agree. > In a given computing context, the exact function to be called is determined at run-time and cannot be deduced from the source code [in OOP languages] as it could in FORTRAN. This is not necessarily true, and I don't think this was true when this was written, either. (The Wayback Machine first saw this in 2014, and the paper doesn't date itself.) Most member function calls in C++ (non-virtual ones) can be resolved at compile-time. Where idiomatic C++ could would use a virtual function would require some equally uninspectable construction in any language, because one would use it when one needed run-time switching. Rust lacks OOP in the usual sense, but if I needed a virtual function of sorts, I might reach for a trait type, which also wouldn't be inspectable (beyond whatever semantics the trait establishes). There's Java, JS, and Python, I do admit. And I find my unit tests often fill in the static analysis I wish I had. But the truth is more nuanced than I think the author conveys. > Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it. > (Trillion is not used rhetorically here, but is based on the different possible states given that the average object size is four words, and the conservative estimate that you are using 16-bit words). This is as vapid a definition of test coverage as the line coverage the author derides in the prior paragraphs. A trivial function, fn add_these(a: u64, b: 64) -> u64 { a + b } could not be adequately unit-tested, because I could never cover the many trillions of input states it has. I think there's always an implicit line to "well, if the code under test is absolutely nuts", it's not the test's fault for not catching it, and that if the function under test (with the same contract as the function above) is errantly coded something like, fn add_these(a: u64, b: 64) -> u64 { if a == 11771008849893880921 && b == 5668622331333919113 { 0 } else { a + b } } then what amount of testing is going to catch that? > I define 100% coverage as having examined all possible combinations of all possible paths through all methods of a class, having reproduced every possible configuration of data bits accessible to those methods, at every machine language instruction along the paths of execution. While this somewhat contradicts the earlier example of states. I don't think we should aspire to this, since I think many "combinations" are not interesting to test; they're just pieces of the code that really don't interact; testing them is redundant. Yes, this hurts our ability to formally define a protocol for testing or not testing, but it prevents the inevitable combinatorical explosion. > Business people, rather than programmers, should design most functional tests. Absolutely not. A large part of my job is coming up with the actual requirements of the system from the vague machinations of business people who barely know how a computer works, let alone can specify salient requirements. How are they to write the tests, when they can't clearly articulate the requirements?¹ > Turn unit tests into assertions. But I want to ensure these don't fire under normal circumstances. We do this, in my current code base, and we get notified of them, and of any crashes in general. But it's a failure that was noticed, and might have downstream consequences, as opposed to one caught by a unit-test. Then the parts about Eastern Europeans with bad Internet make better programmers because they had to think instead of using the Internet, and how he grew up under similar circumstances. Real Programmers! ¹I'm not saying business types "should" (or should not) be able to articulate requirements/specifications.

pknopf 7y ago

> espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely” The point is that when you do integration tests, you will test the underlying classes the way in which it actually matters. You can't test every possible condition a unit can have, but you can test for the most likely.

crdoconnor 7y ago
This is my experience too. Realism in testing is criminally underrated while code coverage is criminally overrated. IME unit tests can only effectively substitute for integration tests where you're testing logical/algorithmic code with simple function inputs/outputs.
- 0xB31B1B 7y ago
  I somewhat agree, somewhat disagree. IMO the largest value I get from unit tests is the confidence that I can make changes and understand what breaks. Refactoring w/o unit tests makes me feel like I am flying blind
  - closeparen 7y ago
    The units we test are almost never big enough for internal refactoring. It’s the decomposition into units that wants refactoring, and there the test suite actively fights back (mock expectations in particular). If there were enough code involved in a test that we could meaningfully refactor it while keeping the test green, we would call it an integration test.
sulam 7y ago
This is no more or less likely in an integration test vs a unit test. One of the first things you learn in writing unit tests is that you should be selective in picking your inputs. This is even taught in those “trade schools” he subtly derides. You regularly see both public test cases and hidden test cases run against problem sets, at least that’s what my interns tell me.