3 ms·
In summary with my comments in brackets: 1. Keep regression tests around for up to a year - but most of those will be system-level tests rather than unit tests
by nickpsecurity 10y ago
In summary with my comments in brackets:
1. Keep regression tests around for up to a year - but most of those will be system-level tests rather than unit tests. [System-level since context of how software was being used is important.]
2. Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.
3. Except for No 2, if X has business value and you can use a system or unit test use a system test for X. [Context again.]
4. Design a test with more care than you design the code. [They need to do something meaningful rather than just be there as a metric.]
5. Turn most unit tests into assertions. [Assertions describe properties you code should always have before, during, or after executions. Just use assertions for such checks that instead of hiding them in tests.]
6.Throw away tests that haven’t failed in a year. [Controversial claim. Says they tell you nothing. I believe the author thinks the design and assertions should make the software right from the beginning with the tests telling you where you're screwing up.]
7. Testing can't replace good development: a high test failure rare suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth.
8. If you find that individual functions being tested are trivial, double-check the way you incentivize developers performance. Rewarding coverage or some other meaningless metrics can lead to rapid architectural decay.
9. Be humble about what tests can achieve. Tests don't improve quality: developers do.
There's a lot of good points in there, though. Talks about combinatorial explosion where tests often don't even measure correctness. Talks about how maintenance burden goes up over just good system testing and assertions with minimal unit testing. Meantions Toyota Production System showed you keep human in the loop and let them do any analysis that requires brains while automating just the mundane stuff. Mentions how hardware engineers duing Design-for-Testing philosophy embed little probes in most of their blocks to catch violation of correctness conditions during testing. Kind of black box plus deep, white box. Says that systems could similarly be wired so the system tests would set off alarms when they should. Just a few gems I saw.
I agree a shorter version of this essay would be beneficial.
- specialist 10y agoPondering #5, assertions. When doing "dynamic" programming (Java + Spring + Maven, Python, anything JavaScript), I feel compelled to write more unit tests. Because the sand shifts under my feet and my crap breaks in surprising ways. I despise most modern development, where entropy continues to creep in and nothing stays done. Maybe more assertions would help mitigate my feelings of hopelessness and despair.
- nickpsecurity 10y agoI tried to find you a decent intro to the best approach at this, Design-by-Contract. The Eiffel site has a nice, short intro with examples and benefits: https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.com/values/design-by-contract/introductio... It has been implemented in Java with the stuff inside JavaDocs. I know JML/Krakatoa integrated it with provers. Simplest is using it for interface checks, though, of expectations of what component is doing vs what it's actually doing. The DbC style lets you document your assumptions, have formal properties to derive tests from, work right there with the code itself, and even automatically generate tests in some tools. That last one is huge benefit as you know the tests will always be relevant to the module and don't have to maintain them. You can do your own tests at the system-level on stuff the module-level contracts might not capture. I'm not sure I can help you for most of the others. Remember, though, you can implement the DbC assertions in code that's in the modules themselves called before and after they run. These can be set to turn off with a flag during production compiles. You can also write generators that pull them out of comments in your source files then produce a version of your program with the checks in them as tests. I can't find my original articles post-PC crash but the one below builds in Ruby a generator for C language DbC: http://www.onlamp.com/pub/a/onlamp/2004/10/28/design_by_contract_in_c.html http://www.onlamp.com/pub/a/onlamp/2004/10/28/design_by_cont... EDIT: Two for Python on StackOverflow that are kludgey but seem to help. https://stackoverflow.com/questions/8563464/using-design-by-contract-in-python https://stackoverflow.com/questions/8563464/using-design-by-...