6 ms·

Most Unit Testing Is Waste (2014) [pdf]

66 points by ptr 10y ago

whatnotests 10y ago
Totally agree 100%. Everyone please stop testing your code. ... ... ... ... Ok folks - everyone who's still testing can keep their jobs (or take the jobs of those who stopped).
- daveguy 10y ago
  To be fair, the paper is about optimizing your test cases and being realistic about what you can accomplish with testing. It is a push toward more efficient testing rather than the lack of testing.
- lj3 10y ago
  There are more types of tests than manually written unit tests. There's a whole other world out there: http://danluu.com/testing/ http://danluu.com/testing/
choward 10y ago
Can we get a tl;dr? 21 pages? Ain't nobody got time for that.
- currriuosly 10y ago
  Agree!
- deleted 10y ago
  [deleted]
- dang 10y ago
  It's important for HN to keep its distance from tldr culture. Longer articles are ok, even if many of us won't read them. It's true of any post that most people won't read it. Snap summaries make for reflexive reactions. That's the shallow kind of discussion we're hoping to avoid here. What we want is reflection, which is slower, takes more energy, and leads to more considered exchanges. More on the reflexive/reflective distinction: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20reflexive%20reflect&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - arjie 10y ago
    Scientific papers aim for a similar objective. They also recognize the importance of being able to summarize. Abstracts are a good idea, and don't necessarily mean people will make snap judgments. There's a lot of long rubbish on the Internet. Being able to quickly distinguish it from good stuff is essential to using your time sensibly.
    - noir_lord 10y ago
      The abstracts are often written by the people who wrote the paper, a tldr often isn't, this allows at best inaccuracies in the summation and at worst complete bias which then results in a big argument completely unrelated to the source material. Frankly if you aren't willing to read the content you simply shouldn't comment on it until you have.
      - corecoder 10y ago
        HN is not just for commenting, it is also for reading other people's submissions and comments. So, granted, it's better not to write comments without reading the article first, but why should that mean that people who have read the article should not write a summary?
      - arjie 10y ago
        None of this is relevant. Summaries are not substitutes for the content. They're to indicate if the content is relevant or not. Frankly, if you aren't capable of summarizing the content, any other comment you'd make is probably rubbish.
    - tptacek 10y ago
      The problem isn't summarizing or recommending or condensing submissions. Most comments do some or all of those things. The problem is tl;dr culture, which actively encourages people not to read the story, and, worse, to discuss the story without having read it. This is so easy: you can almost entirely avoid the issue simply by avoiding the term "tldr".
- nickpsecurity 10y ago
  I just posted a summary for you in this thread.
fermigier 10y ago
This is from 2014 and has already been discussed (~300 comments on HN): https://news.ycombinator.com/item?id=7353767 https://news.ycombinator.com/item?id=7353767 See also: https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-is-waste/ https://henrikwarne.com/2014/09/04/a-response-to-why-most-un...
thatswrong0 10y ago
> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. The value of the information they produce may not be worth the expense of maintaining and running the tests. This seems.. entirely unreasonable to me. Just because some area of code is not touched frequently doesn't mean that we should throw out tests pertaining to that section of code. Can someone justify this to me? This seems like a _terrible_ mistake.
- inimino 10y ago
  The context is trivial tests which only failed before the developer wrote the code and have never failed since. An example given is a method setting x to 5 and then testing that method. Because maintaining code has a cost, tests such as these should be removed. Regression tests are mentioned later and are not what the author is addressing here.
  - cle 10y ago
    Depending on what language you're using, x=5 may not be as trivial as you think.
    - stcredzero 10y ago
      The most dangerous tools in the shop are often the deceptively simple and familiar ones.
- cle 10y ago
  I would argue that "producing information" is only one benefit of tests. Having worked with decent-sized legacy code bases, another big benefit of tests is preventing future loss of information. Deleting tests removes those guardrails. If what you write is successful, then it will become a legacy system, and your predecessors who inherit it (and ultimately your customers) will be glad you didn't delete tests.
jondubois 10y ago
I agree that unit testing is a waste if the project requirements change quickly. Unit tests can take ages to write and if the project requirements are changing constantly, then you keep having to update them and it slows down development and kills productivity - I've seen this happen many times in previous companies. Unit tests only make sense for components of a system which are BOTH critical and stable. I think that integration tests are often way more useful for most projects.
- stcredzero 10y ago
  Unit tests can take ages to write and if the project requirements are changing constantly, then you keep having to update them and it slows down development and kills productivity - I've seen this happen many times in previous companies. In some environments, refactorings to the code will also refactor the tests. Doing test first is only a win in the situation where you don't have to duplicate work, and only if management wouldn't otherwise give you the time to write tests after the fact. Unfortunately, the former is less common, and the latter is more common.
nickpsecurity 10y ago
In summary with my comments in brackets: 1. Keep regression tests around for up to a year - but most of those will be system-level tests rather than unit tests. [System-level since context of how software was being used is important.] 2. Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value. 3. Except for No 2, if X has business value and you can use a system or unit test use a system test for X. [Context again.] 4. Design a test with more care than you design the code. [They need to do something meaningful rather than just be there as a metric.] 5. Turn most unit tests into assertions. [Assertions describe properties you code should always have before, during, or after executions. Just use assertions for such checks that instead of hiding them in tests.] 6.Throw away tests that haven’t failed in a year. [Controversial claim. Says they tell you nothing. I believe the author thinks the design and assertions should make the software right from the beginning with the tests telling you where you're screwing up.] 7. Testing can't replace good development: a high test failure rare suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth. 8. If you find that individual functions being tested are trivial, double-check the way you incentivize developers performance. Rewarding coverage or some other meaningless metrics can lead to rapid architectural decay. 9. Be humble about what tests can achieve. Tests don't improve quality: developers do. There's a lot of good points in there, though. Talks about combinatorial explosion where tests often don't even measure correctness. Talks about how maintenance burden goes up over just good system testing and assertions with minimal unit testing. Meantions Toyota Production System showed you keep human in the loop and let them do any analysis that requires brains while automating just the mundane stuff. Mentions how hardware engineers duing Design-for-Testing philosophy embed little probes in most of their blocks to catch violation of correctness conditions during testing. Kind of black box plus deep, white box. Says that systems could similarly be wired so the system tests would set off alarms when they should. Just a few gems I saw. I agree a shorter version of this essay would be beneficial.
- specialist 10y ago
  Pondering #5, assertions. When doing "dynamic" programming (Java + Spring + Maven, Python, anything JavaScript), I feel compelled to write more unit tests. Because the sand shifts under my feet and my crap breaks in surprising ways. I despise most modern development, where entropy continues to creep in and nothing stays done. Maybe more assertions would help mitigate my feelings of hopelessness and despair.
programmarchy 10y ago
For the most part I find myself writing unit tests to save time debugging. With a unit test I can create a limited context for my module to execute within instead of having to manually run through through several steps in the larger application to test out some piece of functionality.
EugeneOZ 10y ago
Tests are for refactoring and other changes in code. To make them safe.
guessmyname 10y ago
In my previous job I was maintaining a big project which originally had zero unit-tests, it was apparently working well but once I started adding TDD I found several security issues, edge cases, and memory overflows; by the end of my contract I had written more than +2,800 unit and integration tests and the code coverage was still low at around 60% but I am very confident that new developers are going to take ownership of the project from the beginning without much training just because I wrote those tests. I consider testing an useful way to allow new contributors to join the development process because they will not be afraid to touch the code because the tests will throw a warning when something breaks after any modification, without tests they would surely have to spend more time checking the details of their modifications more than integrating into the team.
- _RPM 10y ago
  What was the definition of code coverage at your shop? Does it mean testing every line of execution?
  - exception_e 10y ago
    He/she probably used a tool to capture the coverage %.
    - throwaway745234 10y ago
      That only rephrases the question to what definition the tool used.
    - _RPM 10y ago
      This comment reminds me of a typical response from a politician to a pressing question.
      - exception_e 10y ago
        Sorry for the very literal answer. I honestly thought you were interested in how the coverage was calculated. As an aside, I've worked on projects where coverage wasn't used and the rule was to "test the important bits" and I've also been on projects where coverage had to be >= ~98%. I wonder if a middle-ground approach would be effective.
bit_logic 10y ago
The biggest problem with unit tests is they are abused as a metric for code quality. Pretty code coverage graphs and percent numbers are easy to present to managers. And outsource firms love these because it adds pointless work they can justify as "code quality". But what happens is the unit tests become filled completely useless logic. Like a test with only mock objects that just tests if a method can be called. Useless but good for padding those code coverage numbers. And ironically it greatly decreases code quality. Because when there's these useless unit tests covering everything, it makes refactoring very difficult. And so no one refactors to improve the code because its too much work.
- FLUX-YOU 10y ago
  "Quick, we need higher cyclomatic complexity scores for our unit tests!"
- stcredzero 10y ago
  Because when there's these useless unit tests covering everything, it makes refactoring very difficult. We had this solved in Smalltalk in the early 2000's. Why has the rest of the programming industry screwed the pooch on this, since?
  - initram 10y ago
    Care to provide any details on how this was already solved? I'd love to know and I'm sure the many people here who write tools for developers would also be interested.
    - stcredzero 10y ago
      Care to provide any details on how this was already solved? Our standard refactoring tool, the original Refactoring Browser (RB), covered Unit Tests just the same as any other code. If you fired off a "canned" refactoring in it, it was guaranteed to be correct, you could multiply undo/redo it, and it covered everything currently loaded in the image, which typically included Unit Tests. I notice in Visual Studio and also with CLion and some other "refactoring" tools for C++ I've played around with, that some "refactorings" amount to: "We'll generate/move code for you, but you're on your own with regards to correctness." This reduces the usability and productivity of the tool considerably. In addition, the Refactoring Browser was considerably more responsive. Most "modern" refactoring tools feel heavy and lugubrious by comparison. The Refactoring Browser was very snappy. Brant and Roberts, the authors of the RB, did this cute demo where they want and replaced every reference to Object in Smalltalk, and transformed Smalltalk into a Thingy-Oriented Programming language. It took under a minute. (This was in VisualWorks 3.1.) On top of that, you could also write SQL-like queries of the code base in a parser tool, that had full syntactic equivalence to the entire language, with wildcards, and even fully scriptable conditional parse-node matching. You could pop the results of this up in a browser, then rinse, repeat. On top of that, you could use the same parser engine to do wild-card replace! In doing this, you would have gone beyond the "canned" refactorings, so the undo/redo stacks in the Refactoring Browser would no longer be valid, but in Smalltalk, you had the Change Log, which is basically a checkpointed transactional log for all the changes to the image/loaded-codebase. You could even search and filter this log in a GUI tool. As this was entirely local to your install, this was very handy and fast as well. (Until it came time to re-run your change log, and you hadn't saved your image (checkpointed) in like a week. But if you were such a lazy dev, you were getting what you deserved!) Most of the above has to do with the easy access to the meta-level of the Smalltalk language, combined with the simplicity of the language itself. This makes it 100x easier to write tools, so more tools get written. (1) In particular, very high quality parsing/rewriting tools can get written. Such tools are much lighter weight, and so are more responsive. On the other hand, the high cost of parsing and accessing the meta level in languages like C++, Ruby (parsing only), and Java, just to name a few, has a huge impact on the "landscape" of those tool's programming environments. It's akin to the affect that different kinds of landscape and climate can have on communities of people living there. So the lessons here aren't just for the tool makers. Really, it starts with the language designers, who set what becomes the landscape for the tool makers. (1) - You can literally start writing a full Smalltalk parser by hand, using top-down LL(1) parsing, have most of the language parsed the same afternoon, and reasonably expect to finish in 1 or 2 days. You can also literally start writing a Smalltalk debugger and have something that lets you browse exception stack frames in 10 minutes. EDIT: And to head off the predictable "obobjection" -- in VisualWorks 7+ the resulting parser would actually run faster than the equivalent YACC/Lex parser in naive C.
k__ 10y ago
Did my first big project with >200 Unit-Test, like we learned at university, haha. Well, then requirements changed, and most of them started failing. In the end I spent most of my time fixing the now wrong tests. On the other hand, stability goes down if I don't do any testing. At the moment I do automated UI testing. I'm a front-end dev so this seems to catch many things, especially thanks to test-videos and screen-shoots. I'll try TypeScript in my next production project, simply because I don't know how to write the right amount of good unit-tests. But I know that TypeScript doesn't prevent all bugs, so I guess I'll have a blind spot between the type-checker and the UI tests, but hopefully it won't be too big :\
- texuf 10y ago
  In my ideal world, when requirements change, you first change the tests to reflect the new requirements. Then you change code till all the tests pass. Then you're done. Things don't always work in this manner, but it feels great when they do. UI tests are probably the hardest to write this way.
- ajmurmann 10y ago
  I think the cost of unit tests when changing code also varies with the quality of the design. If your code manages to some degree to follow the open/closed principle you should be able to just throw out some of your classes. On the other hand that also implies that maybe in those cases the unit tests could have been tossed earlier. However, there always will be things that won't be extended but actually need to be changed and you won't know in advance which is which. I do however like the idea of unit tests as a scaffold that guide you to build the right thing and then you toss them. Usually I have only very few construction sides in my code base that I am confident enough to burn the scaffold down. It would be an interesting idea to TDD a project and always delete all unit tests after you are done with them and only keep cheaper acceptance tests around. Maybe you would actually ho faster
z3t4 10y ago
im a fan of built in tests/traps, like assertions that makes bugs "explode": file x, line 22 did not expect state x to be y. Call Allen. When discovered, make a test that repeats the steps that lead up to the error.
SomeCallMeTim 10y ago
I think most unit testing is a waste because it duplicates what a good type checker would do for you. I'm using TypeScript now, and after a break from using typed languages, it's a huge breath of fresh air. Much of the code I write doesn't get unit tests at all, because it's simple enough that it won't ever fail. Refactoring major blocks of code is safe, even without unit tests, because the type checker ensures that everything is wired up in a sane manner when you're done. Good design can obviate the need for many unit tests. When people talk about test code coverage in JavaScript/Ruby/Python, I think the main reason they want close to 100% coverage is that many runtime failures in those languages occur because some line of code somewhere is accessing a type incorrectly. That doesn't happen if you're using static typing. If you've got some complex logic, making sure it works using unit tests is fine. I still do that with anything I consider non-trivial. But if you've got a really simple function that obviously works, and TypeScript ensures the function will always get the types it's expecting, writing tests to ensure it will keep working forever is just a waste of time, unless it's to verify for your own sake that the "obvious" function does what you think it should. But in that "TDD" case, keeping the test around just makes the code base more brittle, since if you decide you need to change the way the function works you now have two functions to maintain instead of just one.
- aprdm 10y ago
  TDD "got mainstream" with Java which is a compiled language. Therefore I don't think you're correct. It's usually the most simple functions that hide the most subtle bugs.
  - thatswrong0 10y ago
    And subtle bugs often arise out of edge cases, and edge case issues aren't caught by most type systems.
  - SomeCallMeTim 10y ago
    Really? I saw it mostly in the Ruby community first, followed by Python and JavaScript. Looking at the WikiPedia page, it looks like it came from Extreme Programming and the C3 project [1], which was SmallTalk. And in case you aren't sure, SmallTalk is dynamically typed. [2] Java has a huge enterprise presence as well, so I'm not too surprised that it's popular in Java circles. Enterprise developers aren't famously top-notch developers in general. >It's usually the most simple functions that hide the most subtle bugs. I can't remember the last time a type-checked simple function I wrote hid a subtle bug. But I'm admittedly a few sigmas above average, so my experience is probably not typical. In studies, TDD is a wash: It neither improves overall programmer throughput nor makes it worse. So use it if it's what makes you happy. I've used it for more complex goals myself, so I understand the draw. I just usually don't need it. [1] https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens... [2] https://en.wikipedia.org/wiki/Smalltalk-80 https://en.wikipedia.org/wiki/Smalltalk-80
vpeters25 10y ago
I think it would be worth considering that we are long ways from the punch card days. There are programming languages, frameworks and tools which would be unlikely to exist without "advances" such as TDD.
- joesmo 10y ago
  Please name one technology that wouldn't exist without TDD.
  - alblue 10y ago
    JUnit was created by Kent Beck et al to support automated testing with what came to be known as XP and subsequently TDD. This inspired subsequent testing frameworks such as TestNG and implementations for other languages like nUnit.
    - joesmo 10y ago
      No, JUnit would not exist without unit tests. Unit tests are not TDD. Try again?
aab0 10y ago
The obsession with information theory here seems like a classic nail-hammer thing. The number of bits my tests convey is totally useless to think about and certainly not worth spending pages on. All I want from my tests from a code base I maintain for thousands of patches is a tiny fraction of a bit: did my latest change break an important behavior or invariant encoded in a unit test? If I only screw up once in every 100 patches, then formally, my unit tests are doing all that work to emit 0.01 bits of information (-log(99/100)), which is formally a totally irrelevant thing to know about my unit testing framework. ('Hey Joe, what have you been up to?' 'Fixing my unit testing framework - I'm up to 0.03 bits per patch!' 'I see.')
swift 10y ago
Reading over the comments, a lot of people seem to be concerned that unit tests compromise their ability to change code quickly as requirements change - they find themselves spending too much time updating tests instead of doing real implementation work. In the experimental stages of a project, I'd buy that. But once a project has matured to the point where it's working and the architecture is broadly in place, requirements changes are usually not so fundamental that there is no resemblance between the old and new requirements. If you're finding yourself having to rewrite large swathes of unit tests when the requirements change, you need to ask yourself if the real problem is that your code is poorly factored. If you're breaking down your code into simple, independent, cleanly composable pieces, you'll find that changing requirements poses much less of a testing burden.
sakopov 10y ago
Unit tests enforce good design and naturally provide decent documentation on how components fit together. Unit tests also allow you to spend a bit more time on the details of each component and find issues you'd otherwise find in production. If nothing else at least have some integration tests so that you can confidently refactor knowing that you're not breaking existing functionality somewhere.
Roboprog 10y ago
One key thing the author started to allude to: the value of functions - pure functions, vs a cyclic graph of mutable objects with temporal coupling of state changes and spaghetti inheritance out the wazoo. OK, he didn't say that exactly, but he really did start heading that way. The "static types will save us crowd" is drowning out a lot of the rest of the discussions that need to happen. * Somebody mentioned Eiffel in other comments. Eiffel's design-by-contract assertions about invariants are vastly preferable to Java's magic beans that eventually (maybe) reach a valid useful state. I guess unit tests sort of compensate for this, but not really. * Mutable state needs to be pushed to the peripheries of our apps, not plastered everywhere. * Inheritance might not be such a good idea, due to reasoning about which code runs when you call a method / send a message. Do like Go-lang, and have interfaces for polymorphism, but skip inheritance - the use of an interface is a flag for later binding. Likewise, higher order functions and closures help you reuse code without resorting to spaghetti inheritance. * We need languages that make programming with types easy, using type inference where it is clear to do so (e.g. - local identifier initialization, but perhaps not for multi-line function headers?). BUT, we still need to allow for dynamic runtime types, perhaps with a few minimal flags on identifiers/modules, rather than forcing people into monstrous reflection frameworks and XML situps. FWIW, I agree (with the paper) that feature/integration tests are very good, but unit tests are often a waste.

66 points by ptr 10y ago

Totally agree 100%. Everyone please stop testing your code. ... ... ... ... Ok folks - everyone who's still testing can keep their jobs (or take the jobs of those who stopped).

daveguy 10y ago
To be fair, the paper is about optimizing your test cases and being realistic about what you can accomplish with testing. It is a push toward more efficient testing rather than the lack of testing.
lj3 10y ago
There are more types of tests than manually written unit tests. There's a whole other world out there: http://danluu.com/testing/ http://danluu.com/testing/

choward 10y ago

Can we get a tl;dr? 21 pages? Ain't nobody got time for that.

currriuosly 10y ago
Agree!
deleted 10y ago
[deleted]
dang 10y ago
It's important for HN to keep its distance from tldr culture. Longer articles are ok, even if many of us won't read them. It's true of any post that most people won't read it. Snap summaries make for reflexive reactions. That's the shallow kind of discussion we're hoping to avoid here. What we want is reflection, which is slower, takes more energy, and leads to more considered exchanges. More on the reflexive/reflective distinction: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20reflexive%20reflect&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- arjie 10y ago
  Scientific papers aim for a similar objective. They also recognize the importance of being able to summarize. Abstracts are a good idea, and don't necessarily mean people will make snap judgments. There's a lot of long rubbish on the Internet. Being able to quickly distinguish it from good stuff is essential to using your time sensibly.
  - noir_lord 10y ago
    The abstracts are often written by the people who wrote the paper, a tldr often isn't, this allows at best inaccuracies in the summation and at worst complete bias which then results in a big argument completely unrelated to the source material. Frankly if you aren't willing to read the content you simply shouldn't comment on it until you have.
    - corecoder 10y ago
      HN is not just for commenting, it is also for reading other people's submissions and comments. So, granted, it's better not to write comments without reading the article first, but why should that mean that people who have read the article should not write a summary?
    - arjie 10y ago
      None of this is relevant. Summaries are not substitutes for the content. They're to indicate if the content is relevant or not. Frankly, if you aren't capable of summarizing the content, any other comment you'd make is probably rubbish.
  - tptacek 10y ago
    The problem isn't summarizing or recommending or condensing submissions. Most comments do some or all of those things. The problem is tl;dr culture, which actively encourages people not to read the story, and, worse, to discuss the story without having read it. This is so easy: you can almost entirely avoid the issue simply by avoiding the term "tldr".
nickpsecurity 10y ago
I just posted a summary for you in this thread.

fermigier 10y ago

This is from 2014 and has already been discussed (~300 comments on HN): https://news.ycombinator.com/item?id=7353767 https://news.ycombinator.com/item?id=7353767 See also: https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-is-waste/ https://henrikwarne.com/2014/09/04/a-response-to-why-most-un...

thatswrong0 10y ago

> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. The value of the information they produce may not be worth the expense of maintaining and running the tests. This seems.. entirely unreasonable to me. Just because some area of code is not touched frequently doesn't mean that we should throw out tests pertaining to that section of code. Can someone justify this to me? This seems like a _terrible_ mistake.

inimino 10y ago
The context is trivial tests which only failed before the developer wrote the code and have never failed since. An example given is a method setting x to 5 and then testing that method. Because maintaining code has a cost, tests such as these should be removed. Regression tests are mentioned later and are not what the author is addressing here.
- cle 10y ago
  Depending on what language you're using, x=5 may not be as trivial as you think.
  - stcredzero 10y ago
    The most dangerous tools in the shop are often the deceptively simple and familiar ones.
cle 10y ago
I would argue that "producing information" is only one benefit of tests. Having worked with decent-sized legacy code bases, another big benefit of tests is preventing future loss of information. Deleting tests removes those guardrails. If what you write is successful, then it will become a legacy system, and your predecessors who inherit it (and ultimately your customers) will be glad you didn't delete tests.

jondubois 10y ago

I agree that unit testing is a waste if the project requirements change quickly. Unit tests can take ages to write and if the project requirements are changing constantly, then you keep having to update them and it slows down development and kills productivity - I've seen this happen many times in previous companies. Unit tests only make sense for components of a system which are BOTH critical and stable. I think that integration tests are often way more useful for most projects.

stcredzero 10y ago
Unit tests can take ages to write and if the project requirements are changing constantly, then you keep having to update them and it slows down development and kills productivity - I've seen this happen many times in previous companies. In some environments, refactorings to the code will also refactor the tests. Doing test first is only a win in the situation where you don't have to duplicate work, and only if management wouldn't otherwise give you the time to write tests after the fact. Unfortunately, the former is less common, and the latter is more common.

nickpsecurity 10y ago

In summary with my comments in brackets: 1. Keep regression tests around for up to a year - but most of those will be system-level tests rather than unit tests. [System-level since context of how software was being used is important.] 2. Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value. 3. Except for No 2, if X has business value and you can use a system or unit test use a system test for X. [Context again.] 4. Design a test with more care than you design the code. [They need to do something meaningful rather than just be there as a metric.] 5. Turn most unit tests into assertions. [Assertions describe properties you code should always have before, during, or after executions. Just use assertions for such checks that instead of hiding them in tests.] 6.Throw away tests that haven’t failed in a year. [Controversial claim. Says they tell you nothing. I believe the author thinks the design and assertions should make the software right from the beginning with the tests telling you where you're screwing up.] 7. Testing can't replace good development: a high test failure rare suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth. 8. If you find that individual functions being tested are trivial, double-check the way you incentivize developers performance. Rewarding coverage or some other meaningless metrics can lead to rapid architectural decay. 9. Be humble about what tests can achieve. Tests don't improve quality: developers do. There's a lot of good points in there, though. Talks about combinatorial explosion where tests often don't even measure correctness. Talks about how maintenance burden goes up over just good system testing and assertions with minimal unit testing. Meantions Toyota Production System showed you keep human in the loop and let them do any analysis that requires brains while automating just the mundane stuff. Mentions how hardware engineers duing Design-for-Testing philosophy embed little probes in most of their blocks to catch violation of correctness conditions during testing. Kind of black box plus deep, white box. Says that systems could similarly be wired so the system tests would set off alarms when they should. Just a few gems I saw. I agree a shorter version of this essay would be beneficial.

specialist 10y ago
Pondering #5, assertions. When doing "dynamic" programming (Java + Spring + Maven, Python, anything JavaScript), I feel compelled to write more unit tests. Because the sand shifts under my feet and my crap breaks in surprising ways. I despise most modern development, where entropy continues to creep in and nothing stays done. Maybe more assertions would help mitigate my feelings of hopelessness and despair.

programmarchy 10y ago

For the most part I find myself writing unit tests to save time debugging. With a unit test I can create a limited context for my module to execute within instead of having to manually run through through several steps in the larger application to test out some piece of functionality.

EugeneOZ 10y ago

Tests are for refactoring and other changes in code. To make them safe.

guessmyname 10y ago

In my previous job I was maintaining a big project which originally had zero unit-tests, it was apparently working well but once I started adding TDD I found several security issues, edge cases, and memory overflows; by the end of my contract I had written more than +2,800 unit and integration tests and the code coverage was still low at around 60% but I am very confident that new developers are going to take ownership of the project from the beginning without much training just because I wrote those tests. I consider testing an useful way to allow new contributors to join the development process because they will not be afraid to touch the code because the tests will throw a warning when something breaks after any modification, without tests they would surely have to spend more time checking the details of their modifications more than integrating into the team.

_RPM 10y ago
What was the definition of code coverage at your shop? Does it mean testing every line of execution?
- exception_e 10y ago
  He/she probably used a tool to capture the coverage %.
  - throwaway745234 10y ago
    That only rephrases the question to what definition the tool used.
  - _RPM 10y ago
    This comment reminds me of a typical response from a politician to a pressing question.
    - exception_e 10y ago
      Sorry for the very literal answer. I honestly thought you were interested in how the coverage was calculated. As an aside, I've worked on projects where coverage wasn't used and the rule was to "test the important bits" and I've also been on projects where coverage had to be >= ~98%. I wonder if a middle-ground approach would be effective.

bit_logic 10y ago

The biggest problem with unit tests is they are abused as a metric for code quality. Pretty code coverage graphs and percent numbers are easy to present to managers. And outsource firms love these because it adds pointless work they can justify as "code quality". But what happens is the unit tests become filled completely useless logic. Like a test with only mock objects that just tests if a method can be called. Useless but good for padding those code coverage numbers. And ironically it greatly decreases code quality. Because when there's these useless unit tests covering everything, it makes refactoring very difficult. And so no one refactors to improve the code because its too much work.

FLUX-YOU 10y ago
"Quick, we need higher cyclomatic complexity scores for our unit tests!"
stcredzero 10y ago
Because when there's these useless unit tests covering everything, it makes refactoring very difficult. We had this solved in Smalltalk in the early 2000's. Why has the rest of the programming industry screwed the pooch on this, since?
- initram 10y ago
  Care to provide any details on how this was already solved? I'd love to know and I'm sure the many people here who write tools for developers would also be interested.
  - stcredzero 10y ago
    Care to provide any details on how this was already solved? Our standard refactoring tool, the original Refactoring Browser (RB), covered Unit Tests just the same as any other code. If you fired off a "canned" refactoring in it, it was guaranteed to be correct, you could multiply undo/redo it, and it covered everything currently loaded in the image, which typically included Unit Tests. I notice in Visual Studio and also with CLion and some other "refactoring" tools for C++ I've played around with, that some "refactorings" amount to: "We'll generate/move code for you, but you're on your own with regards to correctness." This reduces the usability and productivity of the tool considerably. In addition, the Refactoring Browser was considerably more responsive. Most "modern" refactoring tools feel heavy and lugubrious by comparison. The Refactoring Browser was very snappy. Brant and Roberts, the authors of the RB, did this cute demo where they want and replaced every reference to Object in Smalltalk, and transformed Smalltalk into a Thingy-Oriented Programming language. It took under a minute. (This was in VisualWorks 3.1.) On top of that, you could also write SQL-like queries of the code base in a parser tool, that had full syntactic equivalence to the entire language, with wildcards, and even fully scriptable conditional parse-node matching. You could pop the results of this up in a browser, then rinse, repeat. On top of that, you could use the same parser engine to do wild-card replace! In doing this, you would have gone beyond the "canned" refactorings, so the undo/redo stacks in the Refactoring Browser would no longer be valid, but in Smalltalk, you had the Change Log, which is basically a checkpointed transactional log for all the changes to the image/loaded-codebase. You could even search and filter this log in a GUI tool. As this was entirely local to your install, this was very handy and fast as well. (Until it came time to re-run your change log, and you hadn't saved your image (checkpointed) in like a week. But if you were such a lazy dev, you were getting what you deserved!) Most of the above has to do with the easy access to the meta-level of the Smalltalk language, combined with the simplicity of the language itself. This makes it 100x easier to write tools, so more tools get written. (1) In particular, very high quality parsing/rewriting tools can get written. Such tools are much lighter weight, and so are more responsive. On the other hand, the high cost of parsing and accessing the meta level in languages like C++, Ruby (parsing only), and Java, just to name a few, has a huge impact on the "landscape" of those tool's programming environments. It's akin to the affect that different kinds of landscape and climate can have on communities of people living there. So the lessons here aren't just for the tool makers. Really, it starts with the language designers, who set what becomes the landscape for the tool makers. (1) - You can literally start writing a full Smalltalk parser by hand, using top-down LL(1) parsing, have most of the language parsed the same afternoon, and reasonably expect to finish in 1 or 2 days. You can also literally start writing a Smalltalk debugger and have something that lets you browse exception stack frames in 10 minutes. EDIT: And to head off the predictable "obobjection" -- in VisualWorks 7+ the resulting parser would actually run faster than the equivalent YACC/Lex parser in naive C.

k__ 10y ago

Did my first big project with >200 Unit-Test, like we learned at university, haha. Well, then requirements changed, and most of them started failing. In the end I spent most of my time fixing the now wrong tests. On the other hand, stability goes down if I don't do any testing. At the moment I do automated UI testing. I'm a front-end dev so this seems to catch many things, especially thanks to test-videos and screen-shoots. I'll try TypeScript in my next production project, simply because I don't know how to write the right amount of good unit-tests. But I know that TypeScript doesn't prevent all bugs, so I guess I'll have a blind spot between the type-checker and the UI tests, but hopefully it won't be too big :\

texuf 10y ago
In my ideal world, when requirements change, you first change the tests to reflect the new requirements. Then you change code till all the tests pass. Then you're done. Things don't always work in this manner, but it feels great when they do. UI tests are probably the hardest to write this way.
ajmurmann 10y ago
I think the cost of unit tests when changing code also varies with the quality of the design. If your code manages to some degree to follow the open/closed principle you should be able to just throw out some of your classes. On the other hand that also implies that maybe in those cases the unit tests could have been tossed earlier. However, there always will be things that won't be extended but actually need to be changed and you won't know in advance which is which. I do however like the idea of unit tests as a scaffold that guide you to build the right thing and then you toss them. Usually I have only very few construction sides in my code base that I am confident enough to burn the scaffold down. It would be an interesting idea to TDD a project and always delete all unit tests after you are done with them and only keep cheaper acceptance tests around. Maybe you would actually ho faster

z3t4 10y ago

im a fan of built in tests/traps, like assertions that makes bugs "explode": file x, line 22 did not expect state x to be y. Call Allen. When discovered, make a test that repeats the steps that lead up to the error.

SomeCallMeTim 10y ago

I think most unit testing is a waste because it duplicates what a good type checker would do for you. I'm using TypeScript now, and after a break from using typed languages, it's a huge breath of fresh air. Much of the code I write doesn't get unit tests at all, because it's simple enough that it won't ever fail. Refactoring major blocks of code is safe, even without unit tests, because the type checker ensures that everything is wired up in a sane manner when you're done. Good design can obviate the need for many unit tests. When people talk about test code coverage in JavaScript/Ruby/Python, I think the main reason they want close to 100% coverage is that many runtime failures in those languages occur because some line of code somewhere is accessing a type incorrectly. That doesn't happen if you're using static typing. If you've got some complex logic, making sure it works using unit tests is fine. I still do that with anything I consider non-trivial. But if you've got a really simple function that obviously works, and TypeScript ensures the function will always get the types it's expecting, writing tests to ensure it will keep working forever is just a waste of time, unless it's to verify for your own sake that the "obvious" function does what you think it should. But in that "TDD" case, keeping the test around just makes the code base more brittle, since if you decide you need to change the way the function works you now have two functions to maintain instead of just one.

aprdm 10y ago
TDD "got mainstream" with Java which is a compiled language. Therefore I don't think you're correct. It's usually the most simple functions that hide the most subtle bugs.
- thatswrong0 10y ago
  And subtle bugs often arise out of edge cases, and edge case issues aren't caught by most type systems.
- SomeCallMeTim 10y ago
  Really? I saw it mostly in the Ruby community first, followed by Python and JavaScript. Looking at the WikiPedia page, it looks like it came from Extreme Programming and the C3 project [1], which was SmallTalk. And in case you aren't sure, SmallTalk is dynamically typed. [2] Java has a huge enterprise presence as well, so I'm not too surprised that it's popular in Java circles. Enterprise developers aren't famously top-notch developers in general. >It's usually the most simple functions that hide the most subtle bugs. I can't remember the last time a type-checked simple function I wrote hid a subtle bug. But I'm admittedly a few sigmas above average, so my experience is probably not typical. In studies, TDD is a wash: It neither improves overall programmer throughput nor makes it worse. So use it if it's what makes you happy. I've used it for more complex goals myself, so I understand the draw. I just usually don't need it. [1] https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens... [2] https://en.wikipedia.org/wiki/Smalltalk-80 https://en.wikipedia.org/wiki/Smalltalk-80

vpeters25 10y ago

I think it would be worth considering that we are long ways from the punch card days. There are programming languages, frameworks and tools which would be unlikely to exist without "advances" such as TDD.

joesmo 10y ago
Please name one technology that wouldn't exist without TDD.
- alblue 10y ago
  JUnit was created by Kent Beck et al to support automated testing with what came to be known as XP and subsequently TDD. This inspired subsequent testing frameworks such as TestNG and implementations for other languages like nUnit.
  - joesmo 10y ago
    No, JUnit would not exist without unit tests. Unit tests are not TDD. Try again?

aab0 10y ago

The obsession with information theory here seems like a classic nail-hammer thing. The number of bits my tests convey is totally useless to think about and certainly not worth spending pages on. All I want from my tests from a code base I maintain for thousands of patches is a tiny fraction of a bit: did my latest change break an important behavior or invariant encoded in a unit test? If I only screw up once in every 100 patches, then formally, my unit tests are doing all that work to emit 0.01 bits of information (-log(99/100)), which is formally a totally irrelevant thing to know about my unit testing framework. ('Hey Joe, what have you been up to?' 'Fixing my unit testing framework - I'm up to 0.03 bits per patch!' 'I see.')

swift 10y ago

Reading over the comments, a lot of people seem to be concerned that unit tests compromise their ability to change code quickly as requirements change - they find themselves spending too much time updating tests instead of doing real implementation work. In the experimental stages of a project, I'd buy that. But once a project has matured to the point where it's working and the architecture is broadly in place, requirements changes are usually not so fundamental that there is no resemblance between the old and new requirements. If you're finding yourself having to rewrite large swathes of unit tests when the requirements change, you need to ask yourself if the real problem is that your code is poorly factored. If you're breaking down your code into simple, independent, cleanly composable pieces, you'll find that changing requirements poses much less of a testing burden.

sakopov 10y ago

Unit tests enforce good design and naturally provide decent documentation on how components fit together. Unit tests also allow you to spend a bit more time on the details of each component and find issues you'd otherwise find in production. If nothing else at least have some integration tests so that you can confidently refactor knowing that you're not breaking existing functionality somewhere.

Roboprog 10y ago

One key thing the author started to allude to: the value of functions - pure functions, vs a cyclic graph of mutable objects with temporal coupling of state changes and spaghetti inheritance out the wazoo. OK, he didn't say that exactly, but he really did start heading that way. The "static types will save us crowd" is drowning out a lot of the rest of the discussions that need to happen. * Somebody mentioned Eiffel in other comments. Eiffel's design-by-contract assertions about invariants are vastly preferable to Java's magic beans that eventually (maybe) reach a valid useful state. I guess unit tests sort of compensate for this, but not really. * Mutable state needs to be pushed to the peripheries of our apps, not plastered everywhere. * Inheritance might not be such a good idea, due to reasoning about which code runs when you call a method / send a message. Do like Go-lang, and have interfaces for polymorphism, but skip inheritance - the use of an interface is a flag for later binding. Likewise, higher order functions and closures help you reuse code without resorting to spaghetti inheritance. * We need languages that make programming with types easy, using type inference where it is clear to do so (e.g. - local identifier initialization, but perhaps not for multi-line function headers?). BUT, we still need to allow for dynamic runtime types, perhaps with a few minimal flags on identifiers/modules, rather than forcing people into monstrous reflection frameworks and XML situps. FWIW, I agree (with the paper) that feature/integration tests are very good, but unit tests are often a waste.