12 ms·
Why Most Unit Testing is Waste (2014) [pdf]
- nuggien 10y agoPrevious discussion: https://news.ycombinator.com/item?id=7353767 https://news.ycombinator.com/item?id=7353767
- shilman 10y agoHilarious: They informed me that they had written their tests in such a way that they didn't have to change the tests when the functionality changed.
- maxxxxx 10y agoWe have a test team that requires the product to stay backwards compatible with tests, not the other way.
- vdnkh 10y agoAbandoning unit tests was a thing many companies were proud to tell me while I was interviewing earlier this year. I always thought that was ridiculous - but i suppose their products didn't have so many users, or thought they could tolerate some bad behavior. I'm happy I ended up at a place where we're big on tests and even have SDETs embedded in within our team. Besides being useful, I often use a unit test as my main method of completing a feature. I also believe it's a code smell when it's hard to write a test. Maybe it's a regional thing - I've heard it said that here in nyc were much more strict with testing (a carryover from financial roots).
- brightball 10y agoI'm all for testing until it gets in the way of a proper solution. I've spent more time fixing tests because we changed something on purpose than I ever have because they caught something we missed. I've also spent time working with developers who cared more about their tests than their code or customers. Like anything else, all things in moderation.
- usrusr 10y ago> I've spent more time fixing tests because we changed something on purpose than I ever have because they caught something we missed. That's why I love the "dump results into a single large string and compare" style of testing. Declaring a given output as known good (until proven otherwise) is a bit of a gamble, because yes, you do think harder when you are writing individual asserts. But when something changes it is very easy to tell a bad, unexpected change in the output from a good one and replacing the previous known good copy with the new one is just a few keystrokes away. A comfortable anything-to-JSON tool and a good diff viewer should be part of any testing arsenal.
- pacala 10y agoThe article top recommendations are strong: * Keep system level integration tests [for up to a year]. * Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value. In my experience, an overarching test layering strategy works best in providing maximum coverage for minimum friction: Add tests bottom-up, at each layer testing the core functionality of that layer, using directly the layers beneath. The most valuable tests are the system-level tests, which, hopefully, exercise a large swath of the underlying codebase. This reduces to the author's recommendation for most cases. Some people are proud to ditch UnitTesting[TM] to avoid the unit test cargo cult, in particular, the proliferation of "decoupled" tests via overuse of mocking libraries. There is very little value in "unit tests" that reproduce the code flow via mocking asserts, alas a lot of codebases that are heavy on unit testing degenerate into gratuitous mock fests, which become incredibly onerous to work with.
- xahrepap 10y agoI'm currently at a phase where I'm just not as worried about TDD or Unit testing. I've realized that most of my buggy code is in integration points. For example, I'm working on an internal project that creates VMs with some provider (be it Virtual Box, AWS, etc) and then deploys a user defined set of docker container to it. I've found that I don't have bugs in situations I would typically test using mocking/stubbing/etc in traditional unit tests. I usually need to have the real AWS service with the docker service running to get any value out of the test. And at that point it's more work to mock anything else than it is to just start up the app embedded and do functional testing that way. I'm becoming more of a fan of verifying my code with some good functional tests in areas that feel like high risk and then some contract testing for APIs other apps consume. Then if I find myself breaking areas or manually testing areas often I fill those in with automated tests.
- lokedhs 10y agoThe beginning of the essay discusses traditional bottom-up programming to top-down programming (using object orientation). I have written a very large amount of Java code in my career, but after having spent a lot of (personal) time on a Common Lisp project (web application) I can safely say it's still possible to build modern applications using a bottom-up approach. I recommend people try it, it can be quite refreshing.
- taeric 10y agoAny recommended reading to describe the bottom-up approach you are taking?
- lokedhs 10y agoUnfortunately there isn't much that I know of. There are several frameworks out there, but I built my own. It only requires very little code to get something that mimics, for example, Ruby on Rails. As I mentioned in another reply, I intend to do some writeups on this stuff, but unfortunately it doesn't have as high priority as it should. But at least a video or two should be doable soon enough.
- aidenn0 10y agoI recently also spent some time on a CL webapp; do you have any details about what you did?
- lokedhs 10y agoWell, the project is a Slack-like application. It's open source, so you're welcome to have a look at the code: https://github.com/cicakhq/potato https://github.com/cicakhq/potato I have written a few blog posts about the architecture, but unfortunately not too much on the web part. I intend to write some more, and also make some videos showing how nice it is to develop a web application in the same process as the server is running.
- em3rgent0rdr 10y agoCopied from end: In summary: • Keep regression tests around for up to a year — but most of those will be system-level tests rather than unit tests. • Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value. • Except for the preceding case, if X has business value and you can test X with either a system test or a unit test, use a system test — context is everything. • Design a test with more care than you design the code. • Turn most unit tests into assertions. • Throw away tests that haven’t failed in a year. • Testing can’t replace good development: a high test failure rate suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth • If you find that individual functions being tested are trivial, double-check the way you incentivize developers’ performance. Rewarding coverage or other meaningless metrics can lead to rapid architecture decay. • Be humble about what tests can achieve. Tests don’t improve quality: developers do.
- taeric 10y agoThis set of takeaways seems quite solid to me. In particular, I've pushed the "remove tests that haven't failed in years" before, but never had any traction. Are there teams that actually do this?
- apsdsm 10y agoI really hope not. That would be kind of like saying "this pre-flight checklist hasn't failed for years, so let's just skip them before we take off."
- BraveNewCurency 10y agoGreat analogy!
- taeric 10y agoHa! I always assume some good faith in the judgement of the people that would be removing things. Didn't realize that would be withheld here. So, in short, no. It is not just a time bound thing. However, if you have not had a test fail in years, it is a good time to audit the test to make sure it is still relevant. If it isn't, get rid of it. It is this kind of logic that insists on the pre flight video that everyone ignores. We can, and should, do better.
- dewiz 10y agoIs it possible to do TDD without unit tests? I ask because a place where I interviewed had no concepts of mocking, injecting, and would just test the whole chain of classes. But dev leads would insist on TDD. Hmm?
- bbcbasic 10y agoNo I think they are conflating TDD with 'having automated tests'
- twunde 10y agoYes. You could do BDD with something like cucumber or codeception. It's also possible that the tests use a test database without mocks. You'll generally see tests without mocks or injections in older codebases where the codebase was not built with injection in mind and adding mocks would man rewriting the entire application.
- crdoconnor 10y agoYep: https://en.wikipedia.org/wiki/Acceptance_test-driven_development https://en.wikipedia.org/wiki/Acceptance_test-driven_develop... I much prefer this approach. The looser coupling gives you more freedom to undo your initial architectural mistakes. You also can't effectively do unit test driven development on big balls of mud. I dislike Gherkin-based languages though. The syntax design was not particularly well thought through.
- partycoder 10y agoThere are things that a machine can do in a much better and reliable way than a human. Comparing lists of items is one of them (e.g: outputs of functions to be verified against expected results). And that's the entire point of using computers in the first place. Otherwise we can go back to pen and paper and process forms by hand. Does it make more sense for a human to do all the aspects of the testing by hand? Of course not. Nobody has budget for that. It's much better to automate as much testing as possible so testers can focus in higher level tasks. Like the risk assessment involved in marking a build as releasable. Then, unit testing encourages people to construct their software for verification. This software construction paradigm in itself is enough of a benefit even if unit tests are absent. Construction for verification diminishes coupling, and encourages developers to separate deterministic logic from logic depending on unreliable processes that require error handling. Doing this frequently trains you to become a better developer. Unreliable processes can be mocked and error handling can be tested in a deterministic way.
- zby 10y agoI have not read the whole article - but I don't think he argues against automated testing in general. Unit testing is only one type of automated testing.
- bluejekyll 10y agoI generally try to keep my HN comments positive, but this is total Bullshit, yes, with a capital 'B'. Unit tests are not albotrosses around the neck of your code, they are proof that the work that you just did is correct and you can move on. After that they become proof that any refactor of your code was correct, or if the test fails and doesn't make sense, that the expectations of your test were incorrect. When you go to connect things up after that, and they don't work, you can look at the tests to verify that at least the units of code are working properly. I am no TDD fan, but I do believe that writing your code in a why that makes it easy to test generally also improves the API and design of the entire system. If it's unit testable, then it has decent separation of concerns, if not, then there may be something wrong (and yes this applies to all situations). I use this methodology for client/server interactions as well where I can run the client code in one thread and the server in another, with no sockets, to simulate their functioning together (thus abstracting out an entire area of potential fault that can be tested in isolation from network issues). The article/paper raises good points about making sure that the tests are not just being written for the sake of code-coverage, but to say they are useless is just sloppy. Utilize the testing pyramid [1], if you adhere properly to that, everything about your system will be better. I have a serious question, given that this was written by a consultant, is it possible that tests get in the way of completing a project in a timely manner, thus causing a conflict of interest in terms of testing? [1] - http://martinfowler.com/bliki/TestPyramid.html http://martinfowler.com/bliki/TestPyramid.html
- je42 10y agoYes, that.
- partycoder 10y agoAnd they are also "executable documentation". If you as a developer want to know how to use something, unit tests can be of help. Then, writing tests for the sake of code coverage, is like judging take a driving license exam and being judged on the streets you visited or how much gas you used. Tests need to be written with logic verification in mind. I am deeply sad this article got upvoted. People should have a sense of responsibility when giving visibility to stuff like this.
- MichaelBurge 10y agoIf I'm writing Perl or something, I'll write unit tests just to verify the code runs in the basic cases. I like Haskell because I can skip most of the unit tests. Integration tests are still good, and some unit tests like "confirm that test data are equal under serialization and then deserialization" help with development speed. But I can usually refactor vast swathes of code all I want without having to worry about breaking anything. If you do write unit tests and your test passes on the first try, make sure you change the output a little bit to ensure it fails. It's more common than you'd think to accidentally not run a test.
- soaringmonchi 10y agoI couldn't agree more. I am doing a lot of nodejs development these days and it's usually a pain to verify that the code works the way it should. Writing a quick bunch of unit tests is very superior to loading your code in an interactive console and manually fiddling with inputs / outputs. These unit tests often break at the first refactoring and we rewrite them at that point. I'm a bit surprised that people argue that unit tests facilitate "aggressive" refactorings. Aggressive refactoring to me means architecture changes and unit tests almost never survive those. Integration testing is what ensures proper functioning in those situations for me.
- henrik_w 10y agoThere was also a follow-up article [1]. My take on the two articles is that he argues that integration tests should be able to replace unit tests in most cases. However, in my own experience, both kinds of tests have their palces. Why unit tests are good: - You get well-tested parts that you can use in your integration tests, so that the integration tests truly catch the problesm that couldn't be caught at a lower level. This makes trouble-shooting easier. - Decoupled design - one of the key advantages of TDD - Rapid feedback. Not all integration tests can be run as quickly as unit tests. - Easier to set up a specific context for the tests. There are more details in the blog post I wrote as a response [2]. [1] http://rbcs-us.com/documents/Segue.pdf http://rbcs-us.com/documents/Segue.pdf [2] https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-is-waste/ https://henrikwarne.com/2014/09/04/a-response-to-why-most-un...
- maskull 10y agoIf the code is decoupled it's easier to reason about. The tricky part is breaking things down to the right level of granularity.
- planetjones 10y agoThrow away tests that haven't failed in a year. That just a ridiculous point IMO. Written properly tests act as perfect docimentation for the system. So just because a screen or a process hasn't changed in a year, so therefore the tests have been consistently passing does not mean you throw the tests away! You never know when someone will need to maintain that piece of the system.
- WalterBright 10y agoI'm not giving up unit tests anytime soon. I've had excellent success using them - success being much faster development time and far fewer bugs found in the field.
- bengalister 10y agoAt my company, we (developers) tend to develop mainly automated functional tests and we test our product in blackbox mode. I also prefer favouring functional tests and found that actually many integration tests become useless. For unit tests, I keep that for pieces of code that are used thoroughly in different contexts, i.e, actually code that could be turned into an external library and/or code that is hard to test in integration/functional tests => it makes non-regression much faster to test. For blackbox mode, I am not that convinced that it is the proper strategy, especially when the product is being built incrementally. The typical example is when an entity state is being modified in a UC and the function to test the state is not yet developped. I'd prefer in that case to have the test verify that the state has been properly updated directly in DB.
- jacquesm 10y agoOperative word 'most', and then only when done by someone who doesn't understand the goal of unit testing. Any tool can be abused, including testing. I became a 'convert' after having to clean up a fairly large mess. Without first writing a bunch of test code there would have been no way whatsoever to re-factor the original code. That doesn't mean I'm a religious test writer and that there is 150% test code for each and every small program I write. But unit testing when done properly is certainly not wasteful, especially not in dynamic languages and in very low level functions. The sooner you break your code after making changes the quicker you can fix the bug and close the black box again. It's all about mental overhead and trust. Unit tests are like the guardrails on the highway they allow you to drive faster confident that there is another layer that will catch you in case something goes wrong rather than that you'll end up in the abyss.
- fsloth 10y agoMy kneejerk reaction is that this is BS but the text has good points and can function as an anecdote on how not to do things. "... Large functions for which 80% coverage was impossible were broken down into many small functions for which 80% coverage was trivial. This raised the overall corporate measure of maturity of its teams in one year, because you will certainly get what you reward. Of course, this also meant that functions no longer encapsulated algorithms. It was no longer possible to reason about the execution context of a line of code in terms of the lines that precede and follow it in execution," Unit tests which break code are stupid. Refactoring is good, but just splitting a large function into smaller pieces does nothing to improve the value of code unless it's done so that there is an understanding of the algorithm available and communicated. Everything can be abused if not used craftly.
- avip 10y agoOk let's stop writing UT and see what happens. Wait... we've already tried that, and we know the result pretty well: 1. In dynamic languages, simple type mismatches, wrong variable names etc. are now caught in "top level system level test". Yes these are bugs that should have been caught by a compiler had we had one. 2. There's no documentation as to how something should work, or what functionality a module is trying to express. 3. No one dares to refactor anything ==> Code rottens ==> maintenance hell. 4. Bugs are caught by costly human beings (often used to execute "system level tests") instead of pieces of code. 5. When something does break in those "top level system tests", no one has a clue where to look, as all the building blocks of our system are now considered equally broken. 6. It's scary to reuse existing modules, as no one knows if they work or not, outside the scope of the specific system with which they were previously tested. Hence re-invention, code duplication, and yet another maintenance hell. Did I fail to mention something? UT cannot assert the correctness of your code. But it will constructively assert its incorrectness.
- misja111 10y agoThere is a difference between unit tests and integration/ system/ component tests. I think the author is talking about pure unit tests, i.e. the ones that test only one specific piece of code and mock everything else away. Too fine grained unit tests actually make refactoring more difficult, because a refactor typically touches so much code that they have to be rewritten. But component tests do provide the safety net that you had in mind when you wrote your reaction.
- zmmmmm 10y ago> 3. No one dares to refactor anything ==> Code rottens ==> maintenance hell. In my experience, with enough poorly written unit tests and a hard line policy on unit test coverage (which often results in poorly written tests), the unit tests often turn out to be as much a barrier to code maintenance as the regular code is. I'm an advocate of judicious unit testing, but taking it too far is worse than not doing it at all if you have integration or system / functional tests IMHO.
- dzhiurgis 10y ago
- kriro 10y agoJust to add another perspective. There is a (pretty sad but real) business case for unit tests (don't need be good just need to exist :( ). In some b2b/enterprise fields it is an excellent sales pitch to be able to say we have X% test-coverage or 500 unit tests OMG. The people that make the buy decisions want to sleep well no matter if it is based on sound reasoning or not. Test coverage is almost a "feature" (in the bad enterprise IT sense). Sounds less risky to buy the product with more test coverage. Less risk I can understand it's like buying insurance...which is great since insurance turns risk into a budgetable item...yes please sell me this awesome software with that high test coverage. /cynic
- lazyfunctor 10y agoI've read this advice couple of times "convert your unit tests to assertions". What does it actually mean? Say in the context of web dev, you add assertions to the code and when they fail you log an exception and move on? Any links related to it will be helpful.
- majewsky 10y agoAssertions are usually only run in development or test mode. So if you run an integration test, the unit tests happen inline through the assertions.
- kmiroslav 10y agoI'm really annoyed there is no date and no way to date when this was written. Coplien has been around for a long time, this could have been written yesterday or thirty years ago.
- mercer 10y agoThe creation date of the pdf is August 5th, 2014. That doesn't necessarily mean anything, but it might :-).
- dang 10y agoGoogling and searching on Twitter both suggest 2014, which is also when the previous HN thread was posted, so we'll go with that.
- twic 10y ago> this could have been written yesterday or thirty years ago An astute observation which reflects poorly on our industry!
- testingnut 10y agoPeople start writing tests like they start coding - badly. They forgot all the DRY principles and don't architect their tests to reduce their coupling on the code they are trying to test. The result is tests that are a drag on additional new functionality. Even these are better than no tests as you can refactor the tests to reduce the coupling. The debate about whether UT or system tests or something in the middle is better is missing the point. A test should be understandable at any level. 5+ mocks per test generally doesn't help the next guy understand what you are trying to test. If you can abstract your system behind an API to drive and test it, you'll have much longer lasting tests that are more business focused and importantly are clearer for the next person to understand. I can see great value in identifying the slow and rarely failing tests and running them after the quick / more information producing tests. Aee there any CI support for such things? I know TeamCity can run failing tests first...
- fahrradflucht 10y agoThat would be true also if the next one trying to understand what you are trying to test is actually a girl.
- testingnut 10y agoPeople start writing tests like they start coding - badly. They forget all the DRY principles and don't architect their tests to reduce their coupling on the code they are trying to test that they learned to do as coders. Scrappy test coding doesn't save time. The result is tests that are a drag on new functionality. Even these are better than no tests as you can refactor the tests to reduce the coupling. The debate about whether UT or system tests or something in the middle is better is missing the point. A test should be understandable at any level. 5+ mocks per test generally doesn't help the next guy understand what you are trying to test. If you can abstract your system behind an API to drive and test it, you'll have much longer lasting tests that are more business focused and importantly are clearer for the next person to understand. Being able to system test without undefined behaviour isn't easy - you have to be able to get to and reset to a known state. Containers / images help here a lot. I can see great speedup in categorising the slow and rarely failing tests and running them after the quick / more information producing tests. Aee there any CI support for such things? I know TeamCity can run failing tests first...
- csours 10y agoWatching the Dragon Innovations videos on Design for Manufacturing [1] earlier, Bill Drislane said that anything you can afford to test must not fail. I think the corollary is that if you cannot afford to test something you cannot rely on it. 1. https://www.youtube.com/watch?v=zCnTUOxMl_4&list=PLNTXUUIxHyNwrlAh2ZkaMTSBrgk86wC-a&index=14 https://www.youtube.com/watch?v=zCnTUOxMl_4&list=PLNTXUUIxHy...
- perfunctory 10y agoExcuse me. Did you read the article?
- dang 10y agoThis comment breaks the HN guidelines. Please (re-)read them: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. We detached this comment from https://news.ycombinator.com/item?id=11799791 https://news.ycombinator.com/item?id=11799791 and marked it off-topic.
- heisenbit 10y agoJames O. Coplin has a long of experience and is steeped in theory and practice. His list of publications is about 200 entries long: https://sites.google.com/a/gertrudandcope.com/info/Publications https://sites.google.com/a/gertrudandcope.com/info/Publicati... Unit tests are not free as they are also code that much is obvious. Coplin however delves also into less obvious aspects of impact of unit tests on design and also the organizational aspects. Ultimately coding patterns are going to reflect the incentives that govern the system. Software development is a lot about trade-offs. There is plenty to be learned here how to do it. A addendum by him can be found here: http://rbcs-us.com/documents/Segue.pdf http://rbcs-us.com/documents/Segue.pdf but the meat is in the 2014 article.
- LoSboccacc 10y agoone thing that is really imposed by unit testing is the ability of exercise single part of code independently. it really brings out code smells: if you need mocks injected everywhere instead of being able to use dependency injection cleanly, it shows. if you have code paths that can only be triggered within events, it shows, etc. having "wasteful" unit testing is more an investment for the future: when users came with real bugs, the ability of reproducing their step in code and fixing that in a no-regression suite is invaluable, but requires your app to be testable in the first place, lacking which you are stuck with manually testing stuff or even worse coughselenium cough
- ryanbrunner 10y agoA lot of times, the argument for why a lot of things are "code smells" turns into circular logic: Code that is hard to test is poorly architected. Poorly architected means it's hard to test. Example. Take this code (I'm reusing an example from this thread since I think it represents typical "well-factored" code, and isn't an obtuse example to prove a point): class FooBuilder def create(object) if FooValidator.valid?(object) FooFactory.save(object) else raise "Can't save an invalid object" end end end It's reasonably easy to intuit the purpose of this code, and what exactly makes for a valid object (since the validator is an explicit dependency of this class.) I've often seen this called "poorly architected code", however, since an isolated unit test needs to depend heavily on mocks to implement, and you end up with something like this: class ObjectBuilder def initialize(foo_validator, foo_factory) @foo_validator = foo_validator @foo_factory = foo_factory end def create(object) if @foo_validator.valid?(object) @foo_factory.save(object) else raise "Invalid - can't save object" end end end From the perspective of a coder coming in and trying to understand what's happening here, this code is much more difficult to understand, despite being more "testable". What makes a foo valid? How do I know what a "foo_factory" is? I suppose I could assume that the class defined in foo_factory.rb is probably one - but I can't actually be sure. The code is more extensible, for sure, but in a way that probably doesn't matter. I can pass in any validator I want! Amazing! Except, in 99% of cases, I'm going to have one way of validating something. The same goes for saving. I would posit that at least 90% of the time that I see dependency injection in a codebase, it's there solely to aid testing and almost never adds practical (as in, actually being used and not just theoretical) value to a codebase.
- yason 10y agoThe final points at the end of the article are gold. They match my own experiences exactly.
- jondubois 10y agoI share a similar sentiment about unit tests. When you have a lot of code with fast changing requirements, unit testing can be a HUGE waste of time. In my previous job, we spent much more time writing and fixing unit tests than actually adding new features. I was operating at like 1/10th productivity. It might make sense if you're working for a huge corporation with a LOT at stake. Unit tests then become a form of risk management - It forces employees to think REALLY LONG AND HARD about each tiny change that they make. It's good if the company doesn't trust their employees basically. I MUCH prefer integration tests. I find that when you test a whole API/service end-to-end (covering all major use cases), you are more likely to uncover issues that you didn't think about, also, they're much easier to maintain because you don't have to update integration tests every time you rename a method of a class or refactor private parts of your code. About the argument regarding using unit tests as a form of documentation engine; that makes sense but in this case you should keep your unit tests really lightweight - Only one test per method (no need to test unusual argument permutations) - At that point, I wouldn't even regard them as 'tests' anymore, but more like 'code-validated-documentation'; because their purpose then is not to uncover new issues, but rather to let you know when the documentation has become out of date. I think if you're a small startup and you have smart people on your team (and they all understand the framework/language really well and they follow the same coding conventions), then you shouldn't even need unit tests or documentation - Devs should be able to read the code and figure it out. Maybe if a particular feature is high-risk, then you can add unit tests for that one, but you shouldn't need 100% unit test coverage for every single class in your app.
- n72 10y agoUnit tests function as a kind of REPL for me and allow me to code considerably faster than without them. Without them, it takes me considerable time each time I want to test the smallest code change since in order to get my app to a testable state I have to click around in the UI, enter a few values in inputs, etc. This is just a waste of time. Moreover, there's a slightly costly context switch which happens when I go from coding the feature to setting up my app to test the feature. With judicious mocking, however, I save a ton of time getting my app to a state where I can actually test the functionality I'm coding and do away with that context switch.
- ryanbrunner 10y agoThat's great, and I do the same. But you really need to ask yourself what value keeping those tests around in your repository is adding. In some cases, absolutely they will have value, but I don't think there's anything wrong with saying at least some of the tests you write were there as a tool to help you build. For example, I almost always do "gold master" testing when refactoring a large unit or module of code (test the big picture input / output given a few cases without regard for fine-grained tests within, refactor away as long as you can keep the tests green). It's an amazing way of refactoring as it acts almost like a safety harness - you have immediate feedback when you've done something wrong and changed the behaviour of the class. After the refactoring is done, however, those tests are almost useless, as they don't test the purpose of a class but just dumbly look at the input and output. I think a lot of the tests done via TDD should be looked at in the same way.
- erikb 10y agoThe funny thing about unit tests is that it's actually possible to write unit tests that don't really help you at all and that this is the way how many unit tests are written (a pessimistic me would say "most"), but nobody thought that would be possible. The initial people who came up with the idea thought about writing down the execution of a usecase, or a small part of that, as a test. Then they ran their code against it while developing it. That gave them insight into the the usecase as well as the API and the implementation. This insight could then be used to improve tests, API and implementation. But most professionals aren't about making quality. They are about paying their rent. So when they started to learn unit tests, they just wrote their code as always, and then tried to write tests, no matter how weird or unreasonable, to increas the line coverage of their test suite. The proudest result for them is not to have a much more elegant implementation, but to find the weird test logic that moved them from 90% coverage to 91%. I believe that's how you get a lot of clutter in your unit tests. However what is described in the document are sometimes example of people really trying, but that are just early in their development. Of course when you learn to do something by a new method you will first do crappy, inefficient stuff. The idea here is how much do you listen to feedback. If that team that broke their logic to get higher coverage learned that this was bad, then they probably adapted after some time, and then they did exactly what unit tests are there for.
- sametmax 10y agoYes, I was the bit of a shock the first time I realized how misleading coverage was. Being executed doesn't mean tested. Tested doesn't mean it works. Working doesn't mean it does what you want. Doing what you want doesn't mean it will do what the client wants. Doing what's the client want now doens't mean it will do keep doing it in the future. But anybody who edited a 100 000 lines long project, ran the unit test, and saw red lines, fixed them, ran it again and then saw green now the feeling : it's great. You are way more confident.
- madgar 10y ago> But anybody who edited a 100 000 lines long project, ran the unit test, and saw red lines, fixed them, ran it again and then saw green now the feeling : it's great. You are way more confident. This is the key insight: unit tests are about feelings more than anything else. People get so defensive in unit testing threads because eliminating unit tests means eliminating their source of confidence.
- keithnz 10y agoI once had a talk with Kent Beck way way back in the day (early 2000s) about how much unit tests there should be, etc... and I think its nicely captured by his reply here http://stackoverflow.com/questions/153234/how-deep-are-your-unit-tests http://stackoverflow.com/questions/153234/how-deep-are-your-... A lot of people seem to miss many of Kent's subtly, but intentionally phrased advice. Unit Tests are a liability, so use them responsibly and as little as possible, but not at the expense of removing confidence in your software. Also, delete tests that aren't doing you any favors.
- misja111 10y agoThanks for the link! I never knew that Kent Beck, one of the fathers of TDD, said such a thing already in 2008. And in the mean while herds of developers have started to chase goals like 90% test coverage, thinking that this is the way that guys like Kent Beck do TDD. If only they would have known earlier!
- maxxxxx 10y agoThe problem is that to follow his advice you need judgement. From my experience more and more devs want strict self-imposed rules that need to be followed so they don't have to think.
- powera 10y agoWhy are so many people saying that if some developers write bad unit tests, then all unit tests are pointless and a waste of time? Yes, I've seen thousand line files of boilerplate unittests that don't actually say anything useful about the system. I've also written unit tests that tell me in 2 minutes rather than in 3 weeks that somebody has broken my code. If your standard for a system of testing is that it guarantees that people can only write good code, you're insane.
- mokn80 10y ago"Throw away tests that haven’t failed in a year" What a load of crap. We have a test that asserts creating a new user doesnt fail, it hasnt failed for over a year, but the day it does..
- dtheodor 10y agoWe often see criticism against unit testing that is based on facts such as tests are badly written, unmaintainable, incomprehensive, and they drag development down. Which may very well be true! But I am amazed at the conclusion: That because tests are badly written, writing tests is a bad thing. No! Any code can be badly written, it doesn't mean that writing code is a bad thing. Tests, like any other piece of code, also need to be designed and implemented well. And this is something you need to learn and get experience with. As to whether well-written unit tests are worth it, I cannot imagine how someone could efficiently maintain a codebase of any size without unit tests. Every little code change is a candidate to break the whole system without them, especially in dynamic languages.
- V-2 10y ago"Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it. (Trillion is not used rhetorically here, but is based on the different possible states given that the average object size is four words, and the conservative estimate that you are using 16-bit words)." It's the most bizarre reasoning I've seen for a while. Well, yes, of course, and my string trimming function will always have coverage of essentially 0% no matter what I do, since strings can be of any length, and there's way more possible strings than atoms in the universe... we clearly have a different concept of "functionality" though "When developers write code they insert about three system-affecting bugs per thousand lines of code. If we randomly seed my client’s code base — which includes the tests — with such bugs, we find that the tests will hold the code to an incorrect result more often than a genuine bug will cause the code to fail! Some people tell me that this doesn’t apply to them since they take more care in writing tests than in writing the original code. First, that’s just poppycock." Of course, because in reality it's not about being more careful or attentive. It's about the fact that tests don't (or aren't supposed to) contain any logic! (Meaning conditional statements of any sort, loops, etc.) It's logic that breeds an overwhelming majority of bugs. If your tests contain any, it means they're written badly. "even if it were true that the tests were higher quality than the code because of a better process or increased attentiveness, I would advise the team to improve their process so they take the smart pills when they write their code instead of when they write their tests" No, it's not about "smart pills" (this condescending tone is so annoying), it's about the fact that - otherwise than with my tests - I can't keep logic out of my production code. "Most programmers believe that source line coverage, or at least branch coverage, is enough. No. From the perspective of computing theory, worst-case coverage means investigating every possible combination of machine language sequences, ensuring that each instruction is reached..." ...oh God, enough. I don't know the author, but it's quite clear he's some CS professor rather than a real-life full time software dev Remember this article? http://blog.triplebyte.com/who-y-combinator-companies-want http://blog.triplebyte.com/who-y-combinator-companies-want https://phaven-prod.s3.amazonaws.com/files/image_part/asset/1669427/kXHIcd9WPsdYClR3CsnR9lipkDk/medium_who_yc_co_want.png https://phaven-prod.s3.amazonaws.com/files/image_part/asset/... - that's who YC startups want. When I saw it for the first time, I wondered why "academic programmers" rated the worst.
- scotty79 10y agoI think I know why I don't enjoy writing test. Most of my enjoyment from programming comes from feeling of power when I am able to write concise code that does a lot for me. Testing spoils the fun as now I need to write another piece of code for each, single thing that my original piece of code is doing. I am no longer a wizard casting fireball into a room. I'm also the guy that has to go over the corpses and poke each one with a stick strong enough and for long enough to check if they are absolutely totally dead.
- xtreme 10y agoExtending that analogy, I enjoy having a robot that will tell me if all the enemies are dead. Then I can try out different spells to my heart's content without having that nagging doubt about their effectiveness.
- scotty79 10y agoExcept it's not a robot. It's a robot per body. Sure some components are common but you still have to build and grease each bugger yourself and adjust them whenever you change the spell in any meaningful way because enemies will drop differently. I'm too lazy for that. I'll just walk with the people I should get through the room and kill the enemies that are meaningfully not dead. Besisdes, after this room, there's another and who has the time. From time to time some of the people I lead will die due to my sloppiness but such is life. It's not like I'm leading royal family through this dungeon.
- ensiferum 10y agoWow so much fail. Where do these mickeymouse programmers repeatedly spring up? "Oh, I'm a supermanprogrammer and I can get it just right the first time because I use my brain!" Sure I believe that there are people who can do this and get the code right by just thinking it through. But I have a question for even these people (and their organizations). - What happens when these people leave and the new junior dev becomes the maintainer? - What happens when the code is migrated/reused somewhere else? - How do you make sure your code works at least the same as it did before after you: * modify it * update some dependent component somewhere, say some open source library that your code uses internally. These have bugs too you know... Simply you don't unless you write the tests. The real value of testing (unit testing, regression testing, system testing) comes once you have that nice test suite and you can automate it and make sure that on every change nothing breaks. This is a beautiful thing to have, because simply no human can understand these software systems fully to make sure that some innocent looking change doesn't break something. These things unfortunately happen. Sure testing is hard if you make it so. I said before that I get an average ratio of about 2/5 of actual code/unit testing code. This doesn't mean that "this is so bothersome let's not do it", what it should mean is that you design your system with testability in mind. A primary architectural goal in any design should be testability How easy it is to write unit tests for your classes and methods. The easier it is, the smoother the unit tests are to write and maintain and the better code quality you will have in the end. Also note that when you design your software to be testable you get a few other abilities for free such as reusability. One of the most important starting points is to find the right separation of truly orthogonal components and define their interfaces right. Way too often I see code where the engineer didn't understand this and clumped several unrelated concepts together. This really makes the testing hard and painful.
- Const-me 10y ago> the new junior dev becomes the maintainer this shouldn’t happen. Maintaining a complex project is the job for a senior developer. > What happens when the code is migrated/reused somewhere else someone else need to carefully test what’s he/she reusing. > The real value of testing comes once you have that nice test suite and you can automate it Sure there’s _some_ value, especially for dynamic languages and/or for developers who don’t use asserts and other runtime checks. But that value isn’t free. Not only they’re expensive to develop, they also make very expensive to change things: you change a minor thing in the code, and suddenly dozens of tests are broken. If your code is implementing some 25-years old spec that’s not going to change for another 40 years, unit testing will likely pay off. If you’re creating something new and rapidly changing, using a statically-typed language that you know how to use correctly, and you understand the value of asserts — unit tests are IMO waste of time.
- deleted 10y ago[deleted]
- tux1968 10y agoNot sure what this says about unit testing in general, but Perl 6 as a language is specified by a test suite. Ie. the only official language definition is the test suite: http://doc.perl6.org/language/faq#What_is_the_Perl_6_spec%3F http://doc.perl6.org/language/faq#What_is_the_Perl_6_spec%3F
- ebbv 10y agoThere's actually a section of this where he basically says that a test that always passes is not useful (provides no information) and a test that fails sometimes is useful (provides lots of information.) I'm not sure exactly what failure of reasoning lead to this conclusion but it's totally bogus. They are both useful. The ideal case is that your codebase is entirely made up of code that never fails and tests that always pass. Obviously sometimes you are going to have tests that fail and introduce bugs that cause tests that used to pass to fail. But that's the reason that you write those tests, to find those problems. The author gives the silly example of a method that always sets x to 5, and a test that calls it and makes sure x is now 5. That seems like a bad test but anyone who's actually done work as a developer understands why it isn't. If you skip the tests that are simple and straight forward and seem like a waste of time and only write more complicated tests then you will have a hard time reasoning what failed when the complicated test fails. Was your x = 5 method faulty? You don't think so but you don't have proof since it wasn't tested. Having the test, as silly as it seems, lets you know that method is working. Anyone who has been on a team that skips easy/simple tests knows what a mistake it is. And if you don't, you will eventually.
- snarfy 10y agoWith modern compilers and debuggers, a good set of integration tests is all you need. You can refactor large parts of the code base and if the integration tests break, thanks to debuggers etc you can easily identify the breaking changes within minutes. There is no need for 100% code coverage in unit tests to catch these same changes. If you do need it your tooling is lacking. Writing code is expensive. If you have more test code than real code it means you value correctness over features. If I can skip unit tests entirely and have a 95% functioning system, I'm not sure that 5% is worth an extra 500% or so lines of code needed in unit tests for 100% code coverage. Unit tests might seem more important in dynamic languages like JavaScript, but that really just points to poor tooling.
- svckr 10y agoWow. > That means that tests have to be at least as computationally complex as code. My BS sense is tingling. No matter how complex the code, in the end it comes down to comparing the output of a function (or state after execution) against what you expected. Granted, OO regularly leads to design where unit testing goes straight to hell and after way to many lines of test-setup you essentially only test that your mocking framework works. But spare me these incorrect blanket statements – they don't help – thank you.
- OJFord 10y ago> When I look at most unit tests — especially those > written with JUnit — they are assertions in disguise. Very often they're assertions not at all in disguise. Python's `unittest`, for example. I'd always assumed the point was to run these assertions at 'test-time', prior to distribution, and not have that code in the 'real' program. Besides, most (?) of the time we probably want to fail more gracefully than that. (Okay we could `except AssertionError`, but typically it's going to be better to return something else, or raise and handle a more specific exception.)
- wongarsu 10y ago>I'd always assumed the point was to run these assertions at 'test-time', prior to distribution, and not have that code in the 'real' program. That's the point of assertions, as they are usually not included in release mode builds in compiled languages. In Python you could just write an assert function that does nothing if some global "release" variable is set to true. You only gain something from unit-tests if they test something that can't be equally well tested by writing the assertions into the regular code and running the software with typical input.
- tigershark 10y agoI think that the author never understood why unit tests have been introduced. Nowadays, with BDD slowly gaining momentum,unit tests are even more imprescindible and they really do add business value. In the past TDD used correctly added business value, but BDD takes it one step further helping to write more valuable tests. When he speaks about testing all the registers states and all the possible combinations of inputs it is apparent that he is missing the point. The only thing that a well written unit test should accomplish is to make sure that a requirement is implemented correctly. And it must be written while designing the code. When I write some new feature I try to understand the requirements and model them into scenarios. At the same time I start writing the code to implement the feature required that is complete only when the respective scenario is green. During this workflow I change multiple times both the code and the tests and, contrary to what the author experienced, my velocity is not hampered by the tests, but it's actually increased. This happens because I can improve my design thanks to the tests that force me to see the problem from the users point of view. I don't really think that a person that only focuses on writing code without writing any scenarios can really tackle the problem in the best way.
- kevan 10y ago>The only thing that a well written unit test should accomplish is to make sure that a requirement is implemented correctly. A common side effect of writing these tests is discovering missing or conflicting cases. A recent example for me in eCommerce is our algorithm for determining if a particular product can be added to your cart. Things start simple (is it in stock?), but they get complicated really quickly. Is there a dropship vendor with stock? Do we expect to receive more stock in the next 24 hours? Is it discontinued? Clearance? Does it look like we'll run out in the next day and accidentally oversell the stock we have? (realtime inventory is a little fuzzy still). When we unit test the module that handles all of this we don't care about internal state or exhaustive input testing, we care about whether the user sees "In Stock" or "Out of Stock" for the situations we encounter
- kabdib 10y agoMandate from Above: "All code that is checked in must have seventy percent code coverage" Developer Response: Keep tests the way they are. Puff up shipping code by adding layers that just check and transpose arguments; enough of these layers and you get your 70% metric of code that's basically guaranteed not to have issues, and check in. My response, trying to take that stuff and port it: Rip out about 75% of the junk code while cursing clueless management and developers who, ultimately, didn't write very good tests (none of the tests ran anyway, because they required an elaborate lab setup that involved undocumented APIs, installs of binaries meant for OSes that were nearly end-of-life, a SQL server and a bunch of other shit infrastructure. Beware test environment creep).
- sivanmz 10y agoIf I write testable code, coverage comes naturally and I don't worry about it. I do think it's misguided to focus on coverage. To me unit testing is a development tool. I write my class inside a test suite and extract it when ready.