25 ms·

Why Most Unit Testing is Waste [pdf]

• collyw 13y ago
  I have been trying to add unit tests to my app, but almost all the tutorials you find are useless. They show you how to test 1 + 1, or how to test the framework I am using. The parts of the code I am almost certain will work. None of them show you how to actually test anything useful.
  • Anderkent 13y ago
    Look for TDD tutorials for less trivial examples of what you should test. You want a general 'how do I know what to test' tutorial, rather than 'how do I use testing framework X' (which is what you were probably reading, if they show you how to test 1+1). Other than that, just be explicit about your api contracts and test those.
  • skrebbel 13y ago
    You might like http://blog.testdouble.com/posts/2014-01-25-the-failures-of-intro-to-tdd.html http://blog.testdouble.com/posts/2014-01-25-the-failures-of-... It's a tad long, but worth it. tl;dr: the first half agrees with you, and the second half describes a process that does help you test useful stuff (and design your software decently along with it)
    • collyw 13y ago
      I have been adding integration tests. Its an in house app, so constantly changing, and constantly in beta. I wanted to start adding unit tests a couple of years back, but looking back, I don't think it would have gained very much, and cost a lot of time (creating the data to test the parts). If there is a functional unit that does one specific thing, I could see it being useful to test. But most of my app basically queries a database and displays the results. The bugs I do get would rarely have been caught by unit tests.
  • mattgreenrocks 13y ago
    Testability falls out of good design. Testability is not worth optimizing for on it's own, though. It's just a nice side effect. Read Growing Object-Oriented Software Guided By Tests. The authors walk through the development of an auction sniper.
    • Iftheshoefits 13y ago
      The "testability" of a piece of software doesn't really have much to do with the discussion about the quantity or kinds of tests that should be written for it. Every test is another thing that has the potential to require debugging.
      • camus2 13y ago
        So true, debugging "tests" is infernal. Peaple always tell you testing should be easy and quick,it is neither of them.
  • michaelfeathers 13y ago
    I wrote a book called 'Working Effectively with Legacy Code' that's really all about testing code that is not easy to test - real apps beyond toy examples. There are techniques in the book but really you should try to write tests for code as you write it. It's the best way to make sure that your code is decoupled enough for testing. If you do that, you don't need the techniques in the book.
    • ollysb 13y ago
      I've heard many good things about this book, definitely on the reading list!
    • earlz 13y ago
      You're the author of that!? I love that book, and even though I stick to it's practices less than I probably should, it was incredibly helpful
    • n1ghtmare_ 13y ago
      This book is legendary! This and Uncle Bob's books Clean Code. A must read for every programmer !
  • ollysb 13y ago
    I read many tutorials and several books on TDD but never quite got it until I read Kent Beck's Test Driven Development By Example[1]. I also highly recommend his Screencast series on pragprog.com[2] which gives you a good flavour of the workflow he uses with TDD. Edit: For those that don't know Kent Beck is basically the godfather of TDD. [1] http://www.amazon.com/Test-Driven-Development-By-Example/dp/0321146530 http://www.amazon.com/Test-Driven-Development-By-Example/dp/... [2] http://pragprog.com/screencasts/v-kbtdd/test-driven-development http://pragprog.com/screencasts/v-kbtdd/test-driven-developm...
    • Chupachupski 13y ago
      Personally I can relate to Kent Beck's approach to testing rather than "test everything": "I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don't typically make a kind of mistake (like setting the wrong variables in a constructor), I don't test for it. I do tend to make sense of test errors, so I'm extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong." http://stackoverflow.com/questions/153234/how-deep-are-your-unit-tests/153565#153565 http://stackoverflow.com/questions/153234/how-deep-are-your-...
      • ollysb 13y ago
        Yeah, I think this comes with experience. I started off testing everything but over the years you get a feel for which tests aren't going to add value over the lifetime of the project.
    • lgunsch 13y ago
      I have also read Test Driven Development By Example. It really opened my eye's to how TDD is suppose to work. Its quite well written. Most of what I have seen on the web does not explain it to any sufficient level.
  • gaius 13y ago
    Well, you have missed the point of TDD. It exists to sell books, seminars, training and consulting. Not to help in the efficient creation of useful software.
• Aqueous 13y ago
  Yes poorly written or tautological tests are useless . But I think we already knew that. I primarily use unit tests as a way to clarify my own thinking about a function. It helps to step through the process of what the function should return given what parameters it is given. Why is that psychological crutch useless? Is your argument that we all should work the same way? Can't we do whatever works?
  • rubiquity 13y ago
    > Yes poorly written or tautological tests are useless > It helps to step through the process of what the function should return given what parameters it is given. Congratulations, you just described tautological unit testing. Unit testing should be about guiding design and describing the behavior of the subject under test. Focusing on input-output testing is exactly what leads to the "unit testing is worthless" blog posts.
    • Aqueous 13y ago
      Not sure at all what you mean. Can you describe what "guiding design and describing the behavior of the subject" looks like from the unit test's perspective? How else are we to specify the behavior of a function than to specify, for a given combination of typed parameters, what is the return value? As the article points out it is impossible to exhaust the space of all possible combinations - but nobody is trying to do that, so it is a straw man. For me, I test the branches that are likely to occur in a production setting, and any obvious edge cases that might occur, but infrequently.
      • jasonlotito 13y ago
        > Can you describe what "guiding design and describing the behavior of the subject" looks like from the unit test's perspective? That's the first part of unit testing. You write a test that fails. By doing that, you are describing what the API should look like. The unit test guides the design of the interface, and how that object works. People do this, even if they aren't unit testing. You've done this if you've ever written some code calling a method, knowing that the method isn't there yet. But you stub it in, and then go write the method.
• ollysb 13y ago
  I've been building my first full blown ember app since the start of the year. I haven't got up to speed with testing yet (as opposed to my usual cucumber/rspec combo with ruby). I can tell you it is painful! I'm used to being able to change code on a whim, safe in the knowledge that my test suite is going to tell me about any regressions. We're using ember-data and there's some changes (like changing a model's association from sync to async) that can have consequences far from the code that you're working on so we've been seeing a lot of regressions. This has meant we're doing a huge amount of manual testing (I'm the only dev but there's 2 biz guys so this balance works at the moment). But I'll definitely be spending the rest of the week getting up to speed with ember testing. Maybe you can get away without tests if you have a really strong type system (i.e. better than java's) but for dynamic languages I think you're in for a world of pain without that safety net.
• unoti 13y ago
  The idea of making the software worse to increase some arbitrary measure of "maturity" is sad and amusing. It makes me think of the saying "what gets measured gets done." It seems it would make more sense for management to be serious about things that actually matter, like customer satisfaction, user experience, productivity, and defects in production.
  • notastartup 13y ago
    Unfortunately, the word "testing" somehow equates to the quality of the software which it does not, not only to managers but those who have been taught to read about it and forced to repeat back it's list of advantages. It's a false sense of security found in a process that has minimal impact on the software performance and quality. When you write a crappy source code that performs crappy, writing all the unit tests in the world won't save you. You simply increase the technical debt to future changes in the source code or design of the software. Simply keeping things as simple as possible, focusing on quality and incrementally verifying along each step of the way saves far more time. It always puzzles me how quality can be increased by writing more code around existing code of questionable quality.
• herge 13y ago
  We are looking at updating our version of Django in the next year, and I am quite happy that we have a battery of unit tests that test how the features of the current version framework work against our code. I will be a lot more confident about upgrading versions because of what might appear at first glance as 'wasteful unittests'. Also, as a rebuttal to the author, how can you expect crappy developers who write crappy tests to write good code?
  • collyw 13y ago
    Integration tests would be more useful, no?
    • rhizome31 13y ago
      The more I become experienced with testing, the less I think the difference between functional, integration and unit tests is important. What's important is to have a test suite with good coverage that runs fast. I found that a mix of high level and lower level tests is a good way to achieve this goal and I don't mind mixing them in the same files. High level tests provide coverage more quickly, but some stuff, such as verifying that an exception is handled properly, is easier to test with unit tests.
    • dgrant 13y ago
      He probably meant integration tests. Generally the standard practice in Django is to do integration testing at various levels. 1) Test the model with an in-memory database, or real database with rollbacks at the end of every test. 2) Test the views, generally using the actual models and db. I don't ever use mocks much when testing a Django project. It wouldn't be too much trouble to mock the model when testing the views, but I don't bother and I don't imagine anyone else does either.
  • allochthon 13y ago
    I have had to do some major refactorings in projects I've worked on. They would have taken a lot longer to get right without the moderate-to-decent coverage of the test suites. Unit tests are a tool, like anything else in software development, but they're invaluable if used intelligently.
• xioxox 13y ago
  The tests in my largest code work at a very high level. They are a number of example documents, chosen to exercise particular features, which are processed by the program and compared with known output. The tests have been pretty effective at spotting changes in behaviour and crashes, so I've found them useful. I'm not convinced that any lower level object or function verification would be particularly helpful, but of course programs vary in their structure. Treating the program as a black box with a variety of functions is a helpful way to test I believe.
  • Roboprog 13y ago
    Agreed. I've found system testing, using things like HtmlUnit, of things the user sees to be a much better use of time that testing individual method / function / subroutine calls. For some types of applications, you might have to make your own testing tools, though: e.g. - compiler/tokenizer might need a tool to dump & examine the generated binary; batch job file-watcher & scheduler might need a tool to supply input files and manipulate apparent system time; ...
• al2o3cr 13y ago
  "Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle." Taken literally, sure. But are you seriously implying that it's useful, or even sensible, to write test cases for every possible input to the line "z = x + y"? (previous HN article on testing floating-point implementation details notwithstanding)
  • michaelfeathers 13y ago
    Unit tests are a tool for understanding. Their value does not come from targeting coverage and formal correctness. At the end of the day, we have to convince ourselves that our code works. We can do that by looking at it and seeing what it does, and by writing tests to demonstrate that we are not lying to ourselves about what we've reasoned. Write tests for things that you are curious about or don't understand - those are thing things that need to be tested.
  • davidw 13y ago
    Using something like QuickCheck, you can generate a lot of different input for one function, including edge cases and everything in between.
• agentultra 13y ago
  Source code is a poor way to specify the behavior of a system. It is complex, verbose, and difficult to reason about without proper language abstractions at appropriate levels. Worse still are machine optimizations obscuring intent as the software evolves, work-around "patterns" for inferior PL designs, and other savageries inflicted upon us. It is highly unlikely you could give a brilliant programmer a few hundred thousand or millions of lines of code and ask them what the program does in an afternoon let alone ask them to find the bug in it. Good unit tests provide the specification for a certain low-level of the code to a developer. You will never catch everything but their purpose is not to test for absolute correctness. Their purpose is to specify, for a given range of expected behavior, what the boundaries are. If I write the test first in earnest I am telling my future maintainers what I intended to write. We can execute that intention against what I wrote and make assertions about whether it works as expected. That's all I expect unit tests to do. Formal correctness proofs for all possible inputs is way out of my league and beyond the scope of almost every project I've encountered save for automated proof verification software. You should write unit tests and you should write them first. They are the executable specification you will run your program against to ensure your software behaves in the manner in which you intended it to. With solid, well-thought out tests guiding your design you should be able to optimize and refactor your software over time and see that it still behaves as expected in the face of change. And you should be able to hand a suite of well-written tests to a new developer and expect them to understand what a given piece of software does in a day (perhaps in even less time for smaller pieces).
  • radicalbyte 13y ago
    Spot on. Unit Tests aren't there to ensure that you get the right answer. It's there to ensure that you've asked the right question. Or at the very least to document the thought process of the person implementing code. Use static analysis tools + code reviews to improve correctness, and unit tests to guide design + document.
    • googletron 13y ago
      Unit tests aren't for asking the right question. There code is there and readable (hopefully) so that shouldn't be an issue. Unit tests are there to ensure when you need to make a change you don't include a regression (moving code backwards). Missed something that the function was previously doing. They aren't useless and have a clear place in testing. Unit tests alone can't guarantee a fully tested system, but they minimize engineer error when working on various pieces of a code base, but this is predicated on cleanly separated and testable code pieces.
• michaelfeathers 13y ago
  "When I was programming on a daily basis, I did make code for testability purposes but I hardly did write any unit tests. However I was renowned for my code quality and my nearly bug free software. I like to investigate WHY did this work for me?" Here's why: anything that focuses attention during development improves quality. Unit tests can be that thing but that means that their value is in a place that Cope is not describing. See: http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed-theo.html http://michaelfeathers.typepad.com/michael_feathers_blog/200...
• fredgrott 13y ago
  The way I found to maximize this is: 1. TDD is for when I am learning new programming as it keeps my wrong assumptions in check when learning new stuff 2. BDD is when you have decrease the learning curve to a low level and helps verify behavior of the system. I use a combination of both but generally my TDD number of tests decreases when I get at a certain advance level of programming in a subject while my BDD tests generally increase.
  • the_af 13y ago
    I find this puzzling. Now don't get me wrong: I'm no fan of TDD, but if I understand TDD proponents correctly, it's a design tool, not a testing tool. I assume your software will always need designing. So are you saying that as you get more familiar with programming you need less design? Or maybe instead of TDD you meant plain unit testing?
• jmnicolas 13y ago
  TL;DR Can anyone that read it provide a summary so that I know if it's worth spending time reading it ?
  • sehugg 13y ago
    • Keep regression tests around for up to a year — but most of those will be system-level tests rather than unit tests. • Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value. • Except for the preceding case, if X has business value and you can text X with either a system test or a unit test, use a system test — context is everything. • Design a test with more care than you design the code. • Turn most unit tests into assertions. • Throw away tests that haven’t failed in a year. • Testing can’t replace good development: a high test failure rate suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth • If you find that individual functions being tested are trivial, double-check the way you incentivize developers’ performance. Rewarding coverage or other meaningless metrics can lead to rapid architecture decay. • Be humble about what tests can achieve. Tests don’t improve quality: developers do
    • jmnicolas 13y ago
      Thanks.
    • Wohui 13y ago
      Beautiful.
    • TwoBit 13y ago
      >> Throw away tests that haven’t failed in a year. That's absolutely f*ing crazy. Throwing those tests away makes it impossible for me to know that any changes I make to the implementation are OK.
      • mempko 13y ago
        Why? Don't you use assertions?
• ilbe 13y ago
  "If you find your testers splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity." What? Splitting up large functions into smaller, well-named units destroys code comprehension?
  • route66 13y ago
    If your code turns into 30 one-liners with 30! unforeseen possible calling combinations thereby exposing an API of irrelevant auxiliary functions to the outside world? Yes.
    • ilbe 13y ago
      I think there's a bigger design problem then, e.g. those functions should really be classes. I'm just talking about doing a few 'extract method' refactorings.
      • collyw 13y ago
        I sometimes find that code which doesn't involve lot of nesting is a lot easier to read and comprehend as one long method / function / subroutine than chopping it up into multiple small methods and having to jump around trying to work out what is happening. There is often nothing to be gained by splitting it up in my opinion. Is that not why some languages are known as scripting languages, because you can read them like a script?
        • nthj 13y ago
          I see what you mean. I'd suggest that longer, more descriptive method names can help here, though. If I'm referring to a method body to understand what it does when reading another block of code, the method probably doesn't have a descriptive enough name. https://signalvnoise.com/posts/3250-clarity-over-brevity-in-variable-and-method-names https://signalvnoise.com/posts/3250-clarity-over-brevity-in-...
    • masmullin 13y ago
• revscat 13y ago
  I'm back in Java-land these days, which is culturally very pro-unit testing. After getting exposed to it again for a few months again I've come to side with the author here. I've never really been comfortable with the amount of time certain people dedicate to unit testing, especially the TDD crowd, but in my hiatus something has arisen in popularity which has made it all the worse: mockito. Prior to mockito, unit testing was (more or less) limited to testing that your methods behaved as expected, and would occasionally expose NullPointerExceptions or other exceptional conditions. Dependent objects were either simplified or simply ignored. With the rise of mock object frameworks, however, your tests specifically say "this method on this mock will be called X number of times, with this result". Mind you, this is all happening in the context of another method call. So, for example, if you were testing the method "calculateDueDate", and that method took a DateTime object, your test might look like this: @Before public void setup() { MyClass myClass = new MyClass(); DateTime mockDateTime = mock(DateTime.class); } @Test public void specificDueDateShouldBeTenDaysFromNow() { DateTime result = myClass.calculateDueDate(mockDateTime); verify(mockDateTime, times(2)).getHour(); // contrived } The problem with this is that the tests become obstacles in the way of refactoring the code. Should you decide that you don't want to use the DateTime library any longer you will have to not just change the code which is using it but the tests as well. Or what if, going back to the example above, you decided not to use the getHour() method any more? Every test referencing that will have to be changed. And changing those tests is very likely to be more involved than changing the code under test, because there frequently are more tests than code. This has a negative impact on the design of the application. Because companies rarely dedicate resources to making existing software better purely for its own sake, you tend to have to do what you can when you can. This means your time is limited to make that refactor, or upgrade that library, or do whatevever change it is that needs to be done. Unit tests, especially those that use mocks, can get in the way of this to such an extent as to make such efforts impossible. I think testing is important. I do not, however, share the belief that is the sole, or even primary, determinant of code quality. In fact, an over reliance on unit testing can easily be a net negative. Should unit tests be thrown out? No. Baby with the bathwater and all that. But they should not be viewed as a silver bullet, either. They're not. They can help, but they can hurt.
  • 13y ago
• jayvanguard 13y ago
  Pragmatic unit test coverage is the way to go. Most of the summary advice isn't too bad but this statement is beyond absurd: > • Throw away tests that haven’t failed in a year. Apparently the author has never worked on a large software project with many versions supported over 5+ years. The value of unit tests increases over time as the software and libraries it depends upon change.
  • boobsbr 13y ago
    Heck, I work with software that is 10 years old, and the only thing that keeps us sane is very meticulous testing. Seems like the author never heard of testing the tests: mutation testing.
• geebee 13y ago
  Some of my my comment here may be a difference in how we view unit tests vs functional or integration tests, and it may also have a lot to do with my use of ruby rather than a language that needs to be compiled. But I hate having less than 100% test coverage, I find it unsettling. One thing I've often said to sell people on the concept of testing: you're already testing. Guaranteed. If you're writing a complicated method, you're almost certainly calling it on the command line and checking output, right? Well, those things that you're writing to check it? save them in a folder called unit tests. You've probably written some code to ensure that different bits of code work properly together? Save those in a folder called integration tests. You may be opening a browser to make sure that the app is behaving the way you expect. Save those tests in a folder called integration. All of this would make testing natural, rather than something you impose onto your project that feels contrived and vaguely forced. I will admit that there is some overhead on all of this. Rather than just typing into a browser and checking the results, you'll need to figure out how to automate that process. Rails makes this quite easy in my opinion, and rails does blur the lines between a few of those categories I mentioned above. I wouldn't worry too much about that - if you're writing tests that cover these scenarios, I wouldn't bother getting too deep into the semantics of testing. You're good. I run a code coverage tool because I like to see what I haven't tested. Often, it turns out I left some code to rot and forgot about it. My gut feeling is that code rot is probably the most confusing thing to a new developer, and one of the hardest things to deal with (can I remove this? should I?) Leaving around obsolete code is kind of a burn on anyone who comes after you. I have mixed feelings about TDD. I will say that when things are going swimmingly for me, I am often doing TDD. But it requires a clarity that I sometimes just don't have. More often, I'm writing tests and code essentially in parallel with each other (like, write a line of code, write a test, write a line of code, write a test). I suppose I could reverse them. One last thing - I think that tests, especially the integration tests, should give you a very good sense of what an application does. In short, if you wanted to give someone a demo of your system, a walk through every case in your integration tests would probably be a very good way to illustrate what it does.
  • ulisesrmzroche 13y ago
    You're not doing Test-Driven Development if you're writing code and then writing tests, though. You must write your test first, and only after you see it failing, write the code to make it pass. You have to stick to this cadence if you want to put TDD in practice.
• moron4hire 13y ago
  I used to be a diehard test fanatic, but then I grew up and realized that, just like SQL ins't suitable for every task every double-plus-good, neither is TDD. Some things I don't unit test: That clicking a particular button fires a particular event. Syntax-wise, setting up the event handler was essentially an assignment, and testing the physical clicking of the button was the job of whomever wrote the operating system's GUI system. Similarly, that setting a field actually persists the value. Assignments are easy to get right. They get wrong when I'm naming my variables poorly. Unless that field is actually a property that is really syntactic-sugar for a method call, in which case it is necessary, so test away. That values adhere to certain type properties. If I have to write a test to check type information at runtime, then I'm not using the type system in my programming language (be it static or dynamic) correctly. I do not use mocks to be able to test stateful subsystems. The whole mock system creates a maintenance nightmare and makes changing APIs extremely brittle. It's far better to have scripting in place to build a base, test system from scratch and walk the program through a set of steps, checking state transitions after each step. In other words, it all becomes on, big, stateless test. The added benefit: such scripts tend to help figure out deployment issues before they happen. I try to use stateless designs for everything. Method calls do not mutate in place, the create a new object and return the results. This is not only safer, it's infinitely easier to test, as you automatically have both halves of your state transition, rather than having to clone the initial state before calling the mutating function.
  • gaius 13y ago
    Sometimes people say to me "unit testing!". And I say "great, of course you are already using a strongly typed language that the compiler can statically check, that'll give you loads of tests for free!" Then they stare at me blankly because they're Javascript or Ruby guys who are only playing at being serious software engineers.
    • gregors 13y ago
      probably very close to the way I stare at people who exclaim "Yay! It compiles! It must work!"
• peterclary 13y ago
  From the paper: "Throw away tests that haven’t failed in a year." Hell, no. Our unit tests provide reassurance that when somebody revisits a component N years from now, and makes a change, they are significantly less likely to break the existing behaviour, even subtly. Throwing away unit tests is like saying "This ship has been sailing for years and has never sunk - let's throw away the lifeboats!" Now, yes - somebody could erroneously change the test condition to make it pass, although hopefully that kind of change would be spotted by even a cursory code review. You can say the same thing about developers who carelessly suppress compiler warnings without understanding what they're telling them. However, the support is there in case I, or another developer, make a mistake. FWIW, catching future mistakes is definitely not the only thing for which we use Unit Tests. The problem domain is hard, and for every test fixture we've written we've found at least one bug. Catching and debugging these bugs from the unit test runner is a lot easier than spotting and debugging these kinds of issues at runtime.
  • e28eta 13y ago
    I think it'd be impossible to know which tests haven't failed. What if they failed on a developer machine because he introduced a regression, but it was fixed prior to checkin? Your CI build thinks the test hasn't failed, but instead it was performing an important job. Those tests probably are a good place to start if you think you have useless tests, but it's simply a heuristic.
• Roboprog 13y ago
  I liked the part about turning unit tests into assertions. Too bad most of us (including me) are programming in Java, rather than something like Eiffel. I really liked the "design by contract" stuff built into Eiffel when I read about it years ago (before there was such a thing as Java). http://en.wikipedia.org/wiki/Eiffel_(programming_language)#Design_by_Contract http://en.wikipedia.org/wiki/Eiffel_(programming_language)#D... Contrast the notion of a class invariant with the notion of a Java Bean, which is constructed as a useless empty piece of crap, and is eventually mutated enough to represent something useful. Of course, this rant takes us into immutability in general...
• morganherlocker 13y ago
  It varies from project to project, but my flow nowadays is something like this: 1. Describe the behavior of the function in a readme with a code example. 2. Create a test that is mostly just a copy/paste from the readme code. 3. Write the actual function until it passes the first test. 4. Write a few more tests for any edge cases I can think of. 5. Get the function to pass again. 6. Encounter another edge case, goto 4. It seems to work pretty well for me anyway. I think a lot of the debate really centers around unit tests vs integration tests (no one serious I know of is arguing for no testing, or all manual testing). In my experience, you get the most bang for your buck with mostly unit tests in library code, and mostly integration tests in applications. Despite the difference in use cases, I think both sides talk past each other most of the time. Unit tests all over the place in an application code tends to be a waste, because the tendency is to test the libraries your application is using. Integration tests in library code is often terrible, because they are usually slower and tightly coupled to external libs.
  • rhizome 13y ago
    I've been considering for some time to write the documentation to an app first in a README restricted to Cucumber grammar and language, and see how everything develops from there.
• EGreg 13y ago
  First of all, let me describe my bias. I don't write unit tests. I have written a giant, well-organized framework which is used by several of my web apps. I don't even have integration tests at this point. My apps are my integration tests. Whenever I make a change, I am very careful to reason about what it affects and then proceed to test it manually across the apps. Still, bugs crop up an alarmingly large fraction of the time - probably 0.25%. For any team, I wish we would do TDD. It's as simple as this: you want an automated system that will signal an alarm, just like a compiler. This is especially useful for weakly typed and duck typed languages. Right now, we do have a system but it's not automated. It's better than nothing - having many clients of an API who use it heavily allows us to make sure we didn't break anything significant. However, at this point, I would go for API Unit Testing first. Meaning - every function exposed by the API should have unit tests matching the documentation. You can document internal functions later, but start with the most outward facing ones. There is a second consideration: VERSIONING As for us, we have a system where code that potentially breaks existing API contracts is strictly kept in a branch. This branch is then imported by all stakeholders and tested. Once they sign off that their clients are compatible, it is merged into the mainline with a version number and a notice of breaking changes. Everyone pulling has to read the breaking changes before updating. If they aren't ready to deal with those changes, they have to clone their own repo of the framework and cherry-pick commits until they ARE ready. On the good side, our installer automates all this. The framework and plugins keep track of version numbers for the db schemas, apis, everything, and signal when something is out of date. Automated systems are better than blaming humans. They are worth up to 50% extra time of the project.
  • chris_mahan 13y ago
    It just seems your system is overly complicated and too hard to reason about logically. Simplify it.
  • EGreg 13y ago
    I should say I meant 25% in the above. I need to improve my patience before committing. And for that we need to put in place a good process, with a bigger team and checklists.
• bebop 13y ago
  In my opinion, I see unit test/integration tests as a way to refactor with confidence. It also gives the added benefit of allowing one to test your application with many different versions of dependent software. My current project example is porting my Django application to multiple version of both Django and Python. I can now start to make my application work with python3 just by running tests. So yes, some of the tests may be testing things that should always be correct, but it helps in other ways.
• opinali 13y ago
  Didn't read; disagreed.
• spotman 13y ago
  I think its important to note that the title says most, and doesn't say 100% of the time unit tests are bad. It specifically mentions using them for critical functions/algorithms, or 'units of computation'. But, he outlines some of the less useful parts about it, common mistakes, and my takeaway from the article really hits spot on with some poor experiences I have had with teams that get lost in unit testing, and it really can lower the code quality and simultaneously the velocity if not approached carefully. Some key quotes from the article: "If you find your testers splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity." This is all too common. Especially with inexperienced developers. Management requires 40, 60, 80, or 100% test coverage blindly, without thinking about whether it makes sense to test that particular part of the code, and furthermore doesn't take into account readability, or in my experience the pain of over-abstracting something. Little is worse when trying to debug a program, and dealing with over abstraction hell to the point where all you can read is the tests, and trying to read the source code has become entirely over complicated, all in the name of keeping the code in a format where its testable, at the cost of it being understandable. Developers that have a lot of experience with system design and software architecture are in a much better place to write appropriate tests while still maintaining understandable source code, but if I had my choice between an over complicated codebase with 80% test coverage, or a more simple codebase with 0% test coverage, I would choose the simple codebase every time. "There’s something really sloppy about this ‘fail fast’ culture in that it encourages throwing a bunch of pasta at the wall without thinking much... in part due to an over- confidence in the level of risk mitigation that unit tests are achieving." In modern development shops that do a lot of TDD, the tests are relied on way too much. Testing of any sort is not a silver bullet. But you find people, even in pretty big, mainstream development shops of large internet properties, relying almost solely on this. Then they pass their 'finished work' over to operations to be deployed, and when something breaks because there was not a test that counted how many file descriptors were used, the developers are always so quick to say 'well, all the tests pass, so its an operations problem now'. "However, many supposedly agile nerds put processes and JUnit ahead of individuals and interactions." In the article he comments on how someone told him that debugging isn't what you do in front of a debugger (obviously debatable) but that its also when you're staring at your ceiling, or discussing the inner workings of a program or algorithm with a counterpart. This is so key, and this is why pair programming is often helpful if you get into a good rhythm with someone. Thinking intrinsically about how software works is the take away here, and all too often we see people rely on tests as a silver bullet and the end result can be code that is over complicated, over confident, and when deployed is an operational nightmare. These sort of things often have a giant net loss in revenue, due to the net loss in a teams velocity to ultimately produce code that works rapidly. When developers lean on tests less (but still employ them where it counts) you'll find easier to maintain code, written by people that will step up to the plate and be responsible for that code. Obviously there are exceptions to this, and there are shops that have the right balance, and maintain high quality understandable code, while maintaining high velocity. Personally, in over 10 years of being in this industry, almost all examples of this that I have witnessed are open source projects that are peer reviewed.
• weatherlight 13y ago
  "Object orientation slowly took the world by storm.." That sentence doesn't even make sense.
  • deleted 13y ago
    [deleted]
• yeukhon 13y ago
  Some code are very hard to test unless you spend a great deal amount of time in making a test driver. I once wrote some 1000 unit tests all by myself for a project I was working on. It was painful to keep updating code, no matter how much time I put into abstraction. At some point a big feature blows things away and tests start to fail. Some systems are hard to test. Anything involving spawning processes doing unit-testing is hell. I can mock/stub but it takes a lot of effort to do that. Nowadays I go with functional tests as much as possible. If I can't even pass my functional tests how on earth do I ship my code upstream. Sometimes unit-tests don't reflect problem in the real world and the hours i have to spend on making unit tests passing can be used to make a few more functional tests more robust. Does anyone know unit test coverage for software like puppet/chef/ansible. A year ago when I checked ansible the number of unit tests was very little. Most of the time functional tests look more promising then unittesting in that kind of complex, process/system interactive software.
• chiph 13y ago
  > 1.5 Low-Risk Tests Have Low (even potentially negative) Payoff  That's the key point for me -- does this test add sufficient value to the test suite to justify it's creation and existence? If it's checking to see if a value got copied over from one object to another, then no. If it checks some logic, then probably yes. Is the code under test used in many many places? Then test the hell out of it.
• neurobro 13y ago
  Regarding the point about a test conveying little information if it never fails: That's true if it has never failed in the past and will never fail in the future. E.g. maybe it really is tautological or it's testing legacy code that has been locked down forever. But if it does fail at some point in the future after having passed in dozens of iterations, that failure will convey a lot of information. Every time it passes, the information-theoretic value of that future failure increases. Unless running the test involves some onerous cost, there is no reason to delete it. Just hide it in a folder where you won't have to look at it.
• stcredzero 13y ago
  Another client of mine also had too many unit tests. I pointed out to them that this would decrease their velocity, because every change to a function should require a coordinated change to the test. This is one thing I have never seen replicated outside of Smalltalk projects: Automated refactorings of code automatically applied those refactorings to the tests.
• thrownaway2424 13y ago
  This guy totally misses the point of TDD. You write the test first because it is the first client of the interface you believe you have finished designing. While writing the test, half the time you will realize that the API sucks and you need to change it.
  • illumen 13y ago
    Also, for pair programming it gives the pair a common goal to aim towards. The people know the inputs, and outputs up front. So one person doesn't just stare into space whilst the other person codes the function, and then says "this is what I wanted it to do". It also helps reduce code, since you only write code to pass tests. You do not write code for some general theoretical use case that is never required.
  • TillE 13y ago
    Exactly. TDD is the only decent way I've seen for quickly designing a good interface, from the outside in. The test suite that you generate along the way is a nice bonus.
• programminggeek 13y ago
  A good design is going to provide more value over time than a good test suite. A lot of developers spend a lot of time writing tests for terrible design on prototype codebases. Those prototypes eventually are thrown into production full of bugs and the answer tends towards "more tests!" instead of "better design!" Most developers just keep building on top of the same codebase until it is a total mess, long after the team has learned a bunch and should refactor or throw away the code and build something designed for production use. There is really not enough discussion about when he right time to do thorough testing and when a project transitions from prototype to production quality. I realize things like software design, planning, and thoughtfulness are heresy in the agile world of 2 week sprints, but most of these things are our own fault due to a lack of planning and craftsmanship. Complain all you want, but in general it's our own fault.
  • tigroferoce 13y ago
    > Most developers just keep building on top of the same codebase until it is a total mess, long after the team has learned a bunch and should refactor or throw away the code and build something designed for production use. That's exactly where a good test suite comes to save you. If you have tests you can refactor as much as you want until you get the code you want. If you don't you will never have enough time to refactor the code base because the effort of manually testing will be too high. And as result your code will rot over time.
    • programminggeek 13y ago
      Yes, but if you are leaning on your test suite to verify a huge refactor, you are unlikely to make the kind of sweeping design changes that are sometimes needed. For example, say your ORM is causing major trouble and you need to replace everything from the controller back to the database. Heck, maybe you decouple the whole backend out into an API. The only tests that help you are the end to end tests. The rest of your test suite for your backend needs to be thrown away as much as your backend itself does.
    • YZF 13y ago
      More often than not the unit tests will have assumptions about the design in them so they will actually stand in the way of refactoring rather than facilitate it. If you have class A with unit tests this is basically a design decision reflected right there. If having class A is bad design fixing that requires dealing with tests + class A and the tests are a hindrance rather than help.
• ejain 13y ago
  Hard to argue with this logic: 1. Poorly designed unit tests are a waste. 2. Most code is poorly designed. 3. Therefore most unit tests are a waste. To be fair, there's no shortage of high level advice on how to write good tests (e.g. "each change in the code should break just one test"), but it's hard to find examples of how to do so in practice (unless you're writing yet another calculator app)...
  • huherto 13y ago
    >Hard to argue with this logic: > ... Agree > but it's hard to find examples of how to do so in practice yeah, I think that it is because every architecture/design/component/layer seem to need a different testing approach. You really have to look at the specific case to come up with a good strategy.
• gregors 13y ago
  short version "programmers can mess up unit tests" let me suggest: http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Tests/dp/0321503627 http://www.amazon.com/Growing-Object-Oriented-Software-Guide...
• lgunsch 13y ago
  James Coplien and Bob Martin have a debate of TDD and unit testing. There is a video of it on YouTube: http://www.youtube.com/watch?v=KtHQGs3zFAM http://www.youtube.com/watch?v=KtHQGs3zFAM
• K0nserv 13y ago
  In my opinion there are two things that makes unit test really useful and they are API design and refactoring security. The article didn't talk about either of these. Writing unit tests creates a mindset where the API for a class /method/module will be designed in a decoupled way to ensure that unit testing is possible. Writing unit tests force the programmer into creating software that is decoupled and modular. You unit test suite is also a formal specification and documentation of what the program is suposed to do. This drastically reduces concerns and issue when refactoring the code base because if the tests are still passing the changed code has, with high probability, not introduced bugs. This of course relies somewhat on the tests actually being meaningfu and not tautological. As pointed out in the article striving for 100% test coverage is not a good measurement of quality. Some things simply don't need to and shouldn't be tested.
• dllthomas 13y ago
  Positively brain-damaged. It's not like unit tests are a magic bullet, or applicable to every situation, or nowhere deserving of criticisms, but both the criticisms and recommendations here seem poorly founded at best and frequently harmful.
• rqebmm 13y ago
  I do most of my work for mobile, and in that space unit tests generally feels like this: http://xkcd.com/1319/ http://xkcd.com/1319/ The vast majority of your typical app is made up of untestable UI, or framework methods and API calls that should be doing their own testing. Now, INTEGRATION testing on the other hand...
• bphogan 13y ago
  Unit testing is supposed to help you design better code. It's a design process that helps you find where you might be tightly coupling things. Are you mocking an external library? Ok, but how? Are you mocking the library directly? Hmmm, maybe you've lashed your code too tightly to that external library. Maybe that means you need some intermediate layer that you do control, so that when Braintree gets bought by Paypal you can change your payment processor by changing the middleware you built. A unit test tends to expose those tight couplings so that you and your team can decide if you want to keep going down that road or not. "If it's too hard to test, maybe you should look at the design." Best advice I ever got on TDD. It's a tool, not a dogma. Use it like a tool to write better code, and get regression tests as a side effect.
• Revex 13y ago
  I can't seem to justify writing tests. It is time consuming and cumbersome. I figure as long as I document reasonably well, and the design is straight forward there is no sense is wasting my time to write tests. I'd prefer to get something done instead of supposedly helping my future self(or my employers).
  • chadcf 13y ago
    Really depends on what your app is doing and how critical it is to a business. For a complex application that has been in production for years, whose original developers weren't very talented and now are long gone, and your business relies heavily on customer satisfaction and bugs can cause lost customers and thus revenue as well as potentially exposing protected data, tests are absolutely critical. On the other hand, a hobby project used by a few people or internally at your company where bugs can be discovered in use and aren't usually a big deal, eh. And of course, there are a lot of levels between the two extremes...
• bennyg 13y ago
  I usually only write unit tests for external open source libraries I am the author of, or ones for other libraries when I can think of a test case they aren't already checking for.
• shitgoose 13y ago
  Question to TDD crowd - should we write test cases to test the test cases? If your test cases are always correct and don't need verification, then why do not you write code that is always correct in first place?
  • drpre 13y ago
    Something something turtles all the way down. When you write a unit test, you are writing code that expresses intended behavior in a different way than the original implementation. The probability of making exactly corresponding errors (i.e. the implementation is wrong but the test incorrectly checks it and erroneously passes) is lower than the probability of making errors in the original implementation independently of any tests. If your test is incorrect, chances are good that it is wrong in a different way than your implementation, and ideally this will trigger a failure that will lead you to recognize your mistake and fix it. If you do not feel that first-degree tests offer enough confidence in the correctness of your code, then by all means write tests for the tests. But that many will find this idea absurd demonstrates that the cost/benefit ratio diminishes the more meta you get. (EDIT: Especially since integration and other tests also help contribute to the confidence that the code is correct.) Alternatively, if your unit test is so similar to the implementation itself that corresponding errors (between the implementation and the test) are likely, then it is probably a poorly written test and of little value.
• S_A_P 13y ago
  Unit testing along with every development methodology d'jour is that it fails to account for the fact that some devs are good and some, quite frankly suck. There really just isn't any way around it. Unit testing does solve a subset of development problems. It also creates a whole new set of problems when managers expect perfect software because it was tdd'd. They most important trait I've seen in good developers is a deep understanding of the problem domain in which they are working. If you don't know the how and why of what you are doing nothing can save you.
• methodin 13y ago
  Unit tests suck if you are not writing proper testable code to begin. If your classes and objects are not written to be separate from each other to a decent degree any change has widespread effects. When you are properly componentizing your code, then while refactoring the innerworkings of classes without affecting their method footprints, unit testing becomes and invaluable tool.
• InclinedPlane 13y ago
  A few things many coders don't worry about enough when it comes to testing: 1: Most test code is of lower quality than production code. 2: Tests often contain assumptions and those assumptions are usually not checked against any external reference. Whatever is assumed within a test is de facto law. Consider mocks, what ensures that mocks have the same behavior as the objects or services they're mocking? Mostly it's just an untested assumption. 3: By far one of the biggest class of errors in coding is omission. And there tests barely protect you at all. If you forget to implement a particular feature/aspect of code chances are that you're going to forget to implement a test for it as well. The way to solve these problems is with thorough code review and integration/beta testing. Ultimately you get into a "who tests the testers?" problem. Which, most of the time, is answered with the resounding noise of crickets. Tests need code review. Tests need owners. Tests need to be challenged. Tests need justifications. A lot of the critical rigor surrounding testing is eroded by common practices which encourage a distinct lack of rigor around test writing (TDD I'm looking at you). Tests aren't magic they're just code, they'll tell you what you ask but if you're not asking the right questions the result is just GIGO. Too many devs think that unit tests are cruise control for quality. They're not. Doing testing right should be just as rigorous and just as difficult as implementing features.
  • chris_mahan 13y ago
    And since it's impossible to have error-free code, and tests are code, therefore it is impossible to have error-free test code. Should there be tests for the tests? And tests for those tests? (Ohhh I see where you're going...)
    • InclinedPlane 13y ago
      Turtles all the way down, right? The point being that ultimately you need to have processes other than automated tests to ensure code, product, and test quality. Otherwise you're just shifting the problem around. Bad tests can be just as hazardous as bad code, if not more so, since they can easily waste lots of development resources which could have been used more productively.
• davvid 13y ago
  An interesting debate: "Jim Coplien and Bob Martin debate TDD": http://www.youtube.com/watch?v=KtHQGs3zFAM http://www.youtube.com/watch?v=KtHQGs3zFAM
• cranklin 13y ago
  Unit tests may cost time initially, but saves more time later.
• einhverfr 13y ago
  From the article: >In most businesses, the only tests that have business value are those that are derived from business requirements. This 10000 times over. I write lots of tests for financial software. My tests are always longer than my code. However, my tests are always written to business/legal requirements and never to the code in a financial software context. Moreover I try very hard to ensure the tests can be run on production software safely, so we know for sure whether or not things are actually working as expected in the field. For application frameworks it is different. The tests are written to the specification, not to the code. But if I had a dollar for every time I have seen a test case that shouldn't be there, because it tested some non-requirement, or worse (some behavior of a dependency).....
• lectrick 13y ago
  I disagree with many things in this article. For example, he says that you should throw away tests that haven't failed in a year. I say simply disable them and put them in a separate test suite, maybe called a validation suite. If the code that they test is changed, you can re-run those tests again to ensure no expected functionality is broken.
• magicroundabout 13y ago
  The argument seems confused. 'Few developers admit that they do only random or partial testing and many will tell you that they do complete testing for some assumed vision of complete. Such visions include notions such as: "Every line of code has been reached," which, from the perspective of theory of computation, is pure nonsense in terms of knowing whether the code does what it should.' It is as if he had never heard of 'use cases', which in fact are not mentioned anywhere in the text - a pretty glaring omission. The portion of the code that is reached is irrelevant as long as you have covered all of the use cases of the system. Ideally code that is not run for any of the use cases should be removed but it does not prevent the software from being fit for purpose.
• daxfohl 13y ago
  My feeling is that unit testing is so popular among developers because it's developers that get to decide the requirements. Don't really feel like working on that mundane piece of new functionality? Then spend a couple hours doing code golf and figuring out how to test this four-line async piece of code. Nevermind that the level of abstraction required to do so will add twelve lines and five new parameters to your method and ripple all the way through your architecture. But hey you can still charge the 8 hours for that because "unit tests add value". IMO unit tests are a business requirement and should be handled as such. Unit tests that go beyond what is required should be deleted, just YAGNI.
• daxfohl 13y ago
  "Premature optimization" has become something of an anathema in the coding world, why isn't "premature testing" so scrutinized? Back in the 90's everything was over-optimized because, well, it made for more interesting work than coding to requirements and it showed off how smart you were. It seems like unit testing is now the "interesting" thing to show off your intelligence—"how can I abstract this function to allow me to inject a mock that asynchronously...?" And the sublime thing is that unit test will always be there to show off the original writer's intelligence, just like the crazy cool optimization in the 90's. And while it works perfectly with the developer's idea of how things should work, there's a disparity between that and the real world, and as soon as that disparity is revealed, it's suddenly a lot more work to wade through all the cruft than it would have been if the requirement had just been implemented in the most straightforward fashion. A few years from now "premature testing" will be a thing.
  • daxfohl 13y ago
    Makes a lot of sense really, and the parallels are there: "First, design without (optimization/testability) in the most straightforward fashion such that it works and the code can be reviewed in a straightforward manner. Only then, decide if some increased (optimization/testability) is worth the extra complexity and implement as necessary." Pragmatic not dogmatic.
• cbsmith 13y ago
  Okay, I'm not familiar with this paper, and I've only read the history intro so far, but it was bad enough I had to stop and comment. It's just wrong on many different levels... For starters, the point about determining which function is going to be called being determined at runtime... That's a function of dynamic binding/function selection, not object orientation. You can have OO software that doesn't have any runtime polymorphism, and you can have non-OO software that is entirely based on dynamic dispatch.... and assuming the logic is such that you really can't know how it will be resolved until runtime, the problem is no better that static code, because the static code effectively becomes an branch based on a dynamically calculated runtime value NO MATTER THE CODING STYLE. I could go on... but I'd have to write as much as the author has.
• daxfohl 13y ago
  Much unit testing seems to paradoxically exist to justify OOP than to really test functionality. You'll never need to swap databases (and your DB abstraction layer probably wouldn't isolate all those problems anyway), but hey it lets you substitute mocks that let you verify that your implementation of the mock operates according to the code you happen to have written. So let's add that extra layer of abstraction, break all our "go to definition" IDE actions, because now we can prove that when we change DBs it's the DB that is wrong because it doesn't correspond to the mock we hacked together.
• sivanmz 13y ago
  Objects can substantially reduce and simplify testing, if you make guarantees about them during construction, keep them immutable and abide by Demeter's Law.