10 ms·

Most Unit Testing Is Waste (2014) [pdf]

192 points by fagnerbrack 7y ago

dep_b 7y ago
Doing a typical "front-end" mobile application I write the following tests: * Integration tests that test the API I get a ton of value out of it though it's sometimes hard to guarantee a certain state at the API's end. But I use them while building the API's but they also can detect any kind of problem after an API upgrade etcetera * Unit tests that test data transformations This is stuff like date formatting, building headers into a request given certain input, building more complex ViewModels that take a few structs as input and turn them into something that reflects the actual process happening in the view. They're valuable while building the logic, help separating the logic since you need to make it testable and also help a ton if you find a bug since a bug simply means adding another test case to see if something goes wrong and then fix it. I don't think I even get to 30% code coverage but I think what I cover is super valuable and the other 70% is usually mostly CRUD boiler plate.
- hhjinks 7y ago
  I personally find testing incoming APIs a waste. Sometimes I write tests to verify that I treat the incoming data correctly if I transform it in fancy ways, but other than that, you should trust the API contract. It's almost like writing tests for the frameworks you're using. Granted, that's from a unit test perspective. Integration tests of APIs are invaluable. I wish we had them already where I currently work.
  - Mandatum 7y ago
    > but other than that, you should trust the API contract As someone who's been working in corporate integration for the past 6 years.. Never trust a contract. Be in WSDL, OpenAPI spec, Word documents or otherwise. I've worked with large tech vendors, I've worked with finance, I've worked with large consultancies. The only people that seem to get it right, is the people you don't want to work with because it's soul-crushing - think HL7 et al.
    - hhjinks 7y ago
      The point is you can't write unit tests for broken API contracts, so either you trust that the contract is upheld, or you've got bigger problems on your hands.
      - Mandatum 7y ago
        Do you mean integration tests? I don't think API contracts should be a part of a unit test. A unit test should be self-contained, unless of course there is a tight coupling to an API.. Which I would steer clear from. That's why I value integration tests over most thing. I can see immediately that something is broken at a high level, what business impacts it has and explain what systems are affected.
revskill 7y ago
I agree. Unit Test is for testing API usage, not implementation. We can use it to figure out the API design good or not. But the problem, is API change is very often, and we don't want to change both implementation and test just for the sake of API change. Integration test is enough in most case. Even API is the implementation details of an abstraction.
desc 7y ago
When I started programming, I'd think up lots of clever ways to avoid repeating things. Ten years on, that code is a nightmare to maintain because changing the behaviour for just one call site of hundreds is next to impossible, because everything gets funnelled through one extremely DRY group of modules. Redundancy vs. Dependencies: it's the dependencies which kill you. Redundancy is often a good thing. If you state an algorithm only once, as the implementation, then the next programmer only knows what it does, not whether it's correct. This is, in my opinion, the main value of unit tests: state the algorithm twice, once as implementation and once as expectations, and if they don't agree then something's wrong. While the odds of a bug existing in any line of code haven't changed, the odds that the exact same bug exists in both sides are much lower. Any bugs which survive that are probably in design rather than implementation, ie. my mental model of what the module should achieve is wrong somehow. Catching that is a job for integration tests. I definitely agree with the 80/20 rule here. 100% code coverage is neither necessary nor desirable, and 20% is fine if it's the most valuable 20%.
dang 7y ago
A thread from 2017: https://news.ycombinator.com/item?id=15591190 https://news.ycombinator.com/item?id=15591190 2016: https://news.ycombinator.com/item?id=12666454 https://news.ycombinator.com/item?id=12666454 2016: https://news.ycombinator.com/item?id=11799272 https://news.ycombinator.com/item?id=11799272 2014: https://news.ycombinator.com/item?id=7353767 https://news.ycombinator.com/item?id=7353767
WrtCdEvrydy 7y ago
This is very interesting, when you contrast it to [https://dl.acm.org/citation.cfm?id=3106270 https://dl.acm.org/citation.cfm?id=3106270]. I'm always weary of arguments that rely on 'unit tests are more complex than the code'. If that's true, then it's correct. Tests have to encapsulate the software contract that's being enforced and should be more complex than the underlying code.
biesnecker 7y ago
My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.
- ChrisCinelli 7y ago
  I have a 3 part article on testing that I need to post... Regarding wasting time on 50% tests... it depends what your system is. In most of the systems (the no mission critical ones, no general libraries) I like to use the 80%/20% rule. I always ask: what are the 20% of the tests that deliver 80% of the value? You learn with experience but if you do not have it, this is a simple way to learn (as a team). During the sprint, have people write down: 1 - The tests that actually failed and found a real problem before the code was shipped. 2 - The tests you wrote for the bugs that were found in production during the sprint. If you haven't written the test yet, ask: what kind of test would have found this bug? Have people in the team share in the retrospective meeting. If you need more time at the beginning you can hold a meeting just for bug sharing. Discuss about the patterns. Commit to write more of these kind of tests. In a few sprint, you should see regression bugs going down. If you also take a look at the tests than never fail, you will also have a sense of the bugs you should not write. Let me know how it goes!
  - Pistos2 7y ago
    > If you also take a look at the tests than never fail, you will also have a sense of the bugs you should not write. Just checking: Did you mean "the tests you should not write"?
- sametmax 7y ago
  My take on that is to make tests using a unit test lib, but a bit more macro. Midway between unit tests and e2e. It's not pure, it's not best practices, but it's great value.
- chongli 7y ago
  I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.
yodon 7y ago
I've generally preferred integration tests to unit tests but Storybook has completely changed how I write React components, for the better.
- notus 7y ago
  Agreed on integration tests over unit tests. I get the most bang for my buck with them. Storybook is pretty nice. I use it for developing smaller components, more of the building blocks of the application. Have you ever checked out Kent C Dodd's react-testing-library? It was a refreshing approach to testing react components IMO and it has gained quite a bit of adoption.
  - yodon 7y ago
    The docs for react-testing-library look like they offer a great deal of wisdom in a tiny package. I find my React components tend to be simple enough that I'm generally far more concerned about whether the CSS works than the JS, leading me to probably keep my focus on Storybook for the moment but I suspect I'll be reaching for react-testing-library before too long. Thanks for the heads up about it.
rileymat2 7y ago
> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. I feel like there is some subtlety here that most will miss. Specifically, "and consider". That is the part we are really bad at.
- alkonaut 7y ago
  The tests that never failed for years are usually unit tests for fundamental things. Say your app has a math library at the bottom as in my example. It has thousands of years for things like vector additions, matrix multiplication and so on. And of course these never fail - they have no dependencies and we never modify it anymore. But we might modify it, say to plug in some other library to do math in 5 or 10 years. I don’t want to delete them for that reason. Luckily this kind of low level test usually takes very little time - we have a 15k test suite and the 5k “fundamental” tests take seconds while the whole integration level takes hours.
- acchow 7y ago
  Those tests are helpful when you do major architectural changes, which may not happen every year. Tests also help in general development. Make some changes, then fix the tests. If you are running tests locally on your machine (and offline) how can you be sure that every time a test fails locally the failure is logged?
  - kazinator 7y ago
    ... and you don't always know what is a "major architectural change" that's going to break one of these tests, so you can't just relegate those tests into a category that is only run for major architectural changes.
  - war1025 7y ago
    There's a difference between tests that haven't broken because they are in stable code, and tests that haven't failed because they are testing tautologies or even just completely failing to break when the underlying code is broken. I've lost count of how many times I've gone into an old test suite at work and found that the tests were still passing even though the code they were testing had completely changed or been removed. Sometimes tests are written very poorly. The codebase benefits from their removal.
sulam 7y ago
I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a class may take on (true!) but then followed up with espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely”. He also implied at the very beginning that somehow this was driven by the switch from FORTRAN to OOP, as if FORTRAN avoided this combinatorial explosion (magic!) and it’s only because of polymorphism that we live in the “unit testing is good” world. The logical contradictions eventually overcame me.
- danmaz74 7y ago
  I saw cases where there really was a waste of stupid unit tests, and no significant tests for the interaction of those units, so I can see where the author is coming from. But my conclusion, after many years of developing with different levels of testing, is that it's a good idea to: * Have at least one unit test for each nontrivial unit, but no unit tests for the trivial ones. But, in these unit tests, use mocking and stubbing only when using real objects isn't feasible, or very inconvenient - isolation is nice to have, but generally overrated in my experience * Create as many unit tests as necessary to feel confident about the edge cases for the most critical units * Implement higher level tests covering the main use cases for the interaction of all the units
- hdfbdtbcdg 7y ago
  Ironically modern fortran is object orientated...
- cryptica 7y ago
  For the same number of lines of test code, an integration test case traverses through much more source code than a unit test case. Not only that, but it also checks that the units are interacting properly and passing each other the right data and returning the right data. Good integration tests can also fee us from static typing because the integration tests implicitly check that the interface contracts between all the different units are met internally. It also verifies many other factors which affect the robustness of the software. For example, they can identify race conditions and prevent them in the future.
lumost 7y ago
The utility of unit tests greatly depends on the type of software you're building. Building a DB without tests is difficult and unlikely to win many customers - particularly when news of that gnarly data race comes out. Whereas a crud webapp where everyone can see what's working and what's not is a bit hit or miss. At the very least testing helps communicate to a code reviewer "hey this thing does what I think it does".
ChrisMarshallNY 7y ago
I've never tested 100% of my code. But I've also never written a unit test that didn't expose bugs. That's mainly because I know what parts of the code will be dicey, and focus unit tests on them.
- Nasrudith 7y ago
  In practice I personally tend to do the reverse. If I encounter a bug I create a unit test to replicate its circumstances and use it to ensure that the bug stays dead and gone.
- aey 7y ago
  That’s funny. I only write tests to prove that the program does what it’s meant to do and not to find bugs. The goal is to prevent a future programmer, including myself, from breaking any of the declared properties.
  - falsedan 7y ago
    Tests don’t prove anything, they just show the intended behaviour. Only full verification proofs show correctness. Too often I’ve seen “tests show it’s correct” suites horribly fail to provide value when the behaviour changes in a unit of business logic and the brittleness needs to be unwound and replaced with robust assumptions.
  - ChrisMarshallNY 7y ago
    For me, it’s all about maximizing quality; which is usually about bugs. For example, I’m currently writing an ONVIF driver. This needs to address hundreds of devices. I only have about ten to test against. They feature a wide range of implementation, but they are, by no means, full coverage. I’m gonna need to mock up some crazy, nonexistent devices to stress a few particular parts of the library. If I can make it work with them, then chances are better than even that it will work with devices I can’t directly test. I use test harnesses to ensure functionality. My test harnesses tend to be huge; often, a lot bigger than actual shipping implementations.
  - deleted 7y ago
    [deleted]
thisgoodlife 7y ago
Most insurance is waste
- Toine 7y ago
  If your house didn’t burn in the last two years, consider not insuring it anymore.
ed-209 7y ago
"The cross product of those paths with the possible state configurations of all global data (including instance data which, from a method scope, are global) and formal parameters is indeed very large. And the cross product of that number with the possible sequencing of methods within a class is countably infinite." Sounds more like a condemnation of OOP than of unit testing, and I do genuinely feel sorry for the unit testing, OOP purists out there. I prefer to design more functional methods, which operate on parameters and injected config instead of instance state and/or globals (cringe). Incidently, this approach makes full coverage attainable. "Large functions for which 80% coverage was impossible were broken down into many small functions for which 80% coverage was trivial. ... Of course, this also meant that functions no longer encapsulated algorithms. It was no longer possible to reason about the execution context of a line of code in terms of the lines that precede and follow it in execution" I can reason about such an implentation MUCH more effectively by glancing at the small bit of higher level code which integrates everything (and as mentioned above by foresaking instance state and polymorphism). This strikes me as a bit like advocating a flat directory structure cause it's important to be able to see all your files at once.
- ereyes01 7y ago
  Yeah a lot of the 90s/early 2000s OOP stuff I learned in school seemed to always result in really tightly coupled systems and bespoke webs of tests and fixtures that strung along weird dependency chains in unwieldy spaghetti piles that did no good. Following TDD has helped me land at decoupled functional interfaces like the ones you've described, and it all scales and composes so nicely, yet stays very tractable. Robert Martin sketched 2 diagrams in [1] that elegantly illustrate these two different design patterns and how testable usually means composable and more tractable: [1] http://blog.cleancoder.com/uncle-bob/2017/03/03/TDD-Harms-Architecture.html http://blog.cleancoder.com/uncle-bob/2017/03/03/TDD-Harms-Ar...
  - MetalGuru 7y ago
    I wish he gave actual examples to illustrate what he was talking about. So you have an additional API layer the tests hit to call the functions you’re testing? Do endpoints map to classes? Modules? So aren’t you just tightly coupling this API to your service? When the service changes, you still have to update the API. Can you explain to me how this solves the problem?
the_arun 7y ago
Unit Tests = The assurance/confidence that you get, if something changes logic, this test will break & you will know it. Aside from just limiting bugs - this is more powerful.
- war1025 7y ago
  Having a robust test suite is very nice. Having an overly pedantic test suite is not. Things like getters / setters that use the same library you've used 1000 other places have no need to be tested again. Add tests for the base concepts and trust that they'll continue working based on those.
- mobjack 7y ago
  95% of the time when a unit test fails for me, it is the test that need fixing instead of my code. It hardly inspires confidence.
  - winrid 7y ago
    Is that because you're not writing tests while you're adding code and just breaking old tests?
  - qznc 7y ago
    To me this suggests that there is no clear understanding what your units are supposed to do. My second theory would be that the unit tests are written by the inexperienced developers while the good developers write the other code.
    - mobjack 7y ago
      The tests will involve 5 mocked dependencies just to test a simple if statement. They test the implemention and are a waste of time. Isolated units that are well tested usually don't need to be updated in my experience so they rarely break.
      - jrhurst 7y ago
miccah 7y ago
> If you find your testers splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity. The emphasis here should be on the reason for splitting up functions. Long, complex functions can be difficult to understand, and removing a few lines in exchange for a (well named) function call is very beneficial for the reader. The opportunity for testing comes from this delegation. A function call is a contract, and the test ensures it complies. Now the reader can comprehend what the code is doing at a higher level; trusting the sub functions do what they intend.
- tsss 7y ago
  It's quite the opposite: The whole point of the architecture is often to facilitate testing. If we didn't have to write tests we could do without a lot of that boilerplate.
njharman 7y ago
I did not read the article. I did read all comments Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages with loose or poor type systems. Many unit tests are not written well. testing more than the interface/contract. Or full of complexity boilerplate and mock. Which means code needs fixing.
- cryptica 7y ago
  Unit tests don't guarantee that your code units are well designed. You can still write unit tests for spaghetti code. In fact, sometimes it encourages it because it encourages antipatterns like dependency injection; because developers get the urge to inject mock objects into the units from the unit tests. Just think about it at a high level. If your module depends on another module, is it really always necessary that this module be substitutable for another module that has the same interface (e.g. another similar module or a mock)? 99% of the time, the answer is no! In fact it's often a problem because this wrongly assumes that if a class exposes the required interface, then it is compatible. There is more to compatibility than just interface; if the submodule maintains its own state (OOP), then its behaviour can change in a way which could break the dependent logic. You can't have Dependency Injection for everything because it's never so simple; these customizable parts of the code have to be designed very carefully.
  - deleted 7y ago
    [deleted]
  - jrhurst 7y ago
    I actually don't get where you are going with this breakdown. Can you give me an example.
    - cryptica 7y ago
      I mean when you have a class which, instead of importing/including/requiring its dependencies directly, expects instances of various classes to be passed into its constructor for example. So the class cannot do its job unless you pass its dependencies into its constructor... This is an antipattern which unit tests tend to encourage because it allows you to decouple the unit logic from its dependencies and it makes it easy to inject mocks in the place of dependencies. A unit test should only test a single class so that means you need to mock out all other classes which your class depends on. Mocking out dependencies in the test code is difficult or not feasible and that's why developers often resort to Dependency Injection in their source code but as mentioned before, it is an antipattern. Also, I noticed that a lot of developers confuse unit tests with integration tests but the definition is quite clear: If your test covers more than one class without mocks, then it's an integration test. Integration testing does not necessarily mean end-to-end testing. It could just be a single class with its internal dependencies.
deleted 7y ago
[deleted]
devin 7y ago
Absent from this discussion is the use of generative testing. At the unit level they can automatically enumerate a variety of scenarios the programmer would likely overlook.
- steve_adams_86 7y ago
  I just discovered this, only it was called 'property based testing'. Do you recommend any resources for learning to use it effectively? I haven't dug in yet, but so far it hasn't clicked when thinking about it. I like the idea a lot, but I guess I don't see where it would be most useful for my own code.
quality_theater 7y ago
One of the things I dislike the most about unit tests is how it's used for "quality theater". Some examples: - Trying to use coverage percentage metric as a sign of quality. As if a simple percentage means anything about the quality of the tests. It's as useless as using lines of code as a way to measure progress. - Not recognizing that useless unit tests are harmful for the code maintenance. It makes refactoring code into better structure difficult and developers just give up because it's too much work to fix the tests that weren't even providing any value in the first place. This ignorance is expressed in statements like how there's a testing "pyramid" with unit tests at the bottom and end to end tests at the top. Which is a nice sounding soundbite and image, but is useless. Forget pyramids, just write tests where it makes sense. - Code review comments that try to look smart with "where's the unit test?" Then the developer doesn't want a long drawn out fight about how a unit test would be useless here since there's a huge crowd just cargo cult yelling "code coverage!" "unit tests are good!" "pyramid!" So the developer just writes the stupid test to get the code merged. This is also an example of how harmful code reviews can sometimes be, when there's a popular stupid idea, code reviews perpetuate it because the developers who know better just get tired of fighting the same fight over and over again. I really hate useless unit tests. I've seen tests that setup a mock, configure it to return a value when called, call the code using the mock, then check the returned value is the mocked value. This tested absolutely nothing! I've seen unit tests that verify every line of the method was called, completely pointless. The point is supposed to be to verify that for a given input, it has a given output, not lock it into a specific implementation by verifying every line of code ran a certain way. There is an idea for structuring software "functional core, imperative shell". Write the software this way and the natural place for unit tests and integration tests becomes obvious. But nope, the industry is all about unit test coverage percentage, stupid pyramids, looking good in code reviews. It's all quality theater, not actual focus on quality.
- reallydontask 7y ago
  > I really hate useless unit tests. I've seen tests that setup a mock, configure it to return a value when called, call the code using the mock, then check the returned value is the mocked value. This tested absolutely nothing! I've had a similar disagreement about the validity of such a test. The other mocking issue I've seen, is people mocking blackbox third-party APIs over which they have absolutely no control, which sometimes leads to passing unit tests, failing integration tests and head scratching all around.
kstenerud 7y ago
After 15 years of writing unit tests, I haven't reached the same conclusion as the author. What I've found in my time is that unit testing can be good, but like anything it's not a panacea. It requires discipline, and like normal code, it has code smells. Black box unit tests are the most likely to be good tests, and white box unit tests are the most likely to be bad tests. The more you depend on the inner workings of a function in order to test it, the more likely it is that you are coupling your test to the implementation rather than the purpose of the unit being tested. Once you tie to the implementation, refactoring becomes a LOT harder, because changes will break the tests even if they don't break the functionality. Mocks are also a major source of trouble, and more likely to be a code smell. If your tests are using a mock to test how many times your unit called it, either your tests are bad or your architecture is wrong. There are three main kinds of code: - Code that fetches data - Code that stores data - Code that transforms data Mocks are necessary when you mix these. If you have a function that opens a DB connection, fetches data, transforms the data, and then stores the data, you now have an extra problem to deal with (the database), when all you wanted to do was test the transformation. Things would be far easier if you separated the transformation out, tested that in isolation, and then integrated that encapsulated functionality with fetching/storing code. This also improves separation of concerns and code duplication, since now your fetching/storing code can be generalized and also tested in isolation. Actually I lie. There is a fourth kind of code: code that modifies state. This is the evilest, smelliest code around, and it's also something that unfortunately we can't get completely away from. But we can manage it, by isolating state, reducing the need for or scope of the state, and providing "configuration object" function entry points to make testing these monstrosities less nasty. Code coverage is not just a measure of quality, but also of waste. If your code is not being called, then one of three things is happening: 1. It's error checking code for another API it's calling, which you normally shouldn't be writing tests for (unless that API is known to be buggy and you need to guard against it). 2. It's not contributing to the goals of the program, and can be taken out. 3. It does contribute to the goals of the program, in which case you need a test for it. You can't reach 100% code coverage because of (1). But you absolutely should check WHICH code is covered in your tests because of (2) and (3). Anything higher than 80% coverage is pure luck, and tells you nothing about quality or wastage. In many cases, even 60-70% is sufficient.
dmitriid 7y ago
Kent Beck (yes, the Kent Beck): “I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence” https://stackoverflow.com/a/153565 https://stackoverflow.com/a/153565
nickthemagicman 7y ago
I don't just code for a single project. I create libraries when I write a piece of code that I know will be useful down the line. And for those libraries that are going to be reused throughout many applications I think unit testing is gold.
fogetti 7y ago
The thing is that most people don't understand that the code needs to be tested anyway. If you don't test it, you are just an idiot who has no proof that the code does what it claims to do. Now because you have to test it anyway, you can just simply spend the same time writing a unit test instead of executing the code with manually configured non-repeatable test cycles, a.k.a clicking through the UI, or sending Postman requests, etc. Also it's kind of selfish not to make something repeatable by others. And as someone pointed out before me, unit tests are not about testing at all. It's a documentation about the system, and what it's supposed to do. Also it's the way to stop the next person who works on the code to ruin something by not knowing the business rules.
- phereford 7y ago
  To add to this sentiment, tests also give teams confidence that is needed. It gives engineers confidence when developing new features because the test suite will break if you introduce a regression. It gives engineers confidence during deployment/CD. While the tangible benefits of unit tests are very important, there are other intangible benefits that are equally important.
  - didericis 7y ago
    I’d say that confidence is a tangible factor. The single most important reason to test your code seems to be that it allows you to refactor your code while ensuring it still meets the original outside expectations. The author’s main gripe seems to be about tightly coupled tests that make refactoring larger systems more difficult, and about prioritizing meeting arbitrary metrics (lines of codes covered) rather that thinking critically about the actual benefits. This is in line with his emphasis on integration tests determined by the business. I think that same thought process can be applied to internal components of code as well, and you can empirically determine the quality of your approach (roughly) by evaluating how often your tests change. If nearly every change breaks a test, that probably means they’re low value/too tightly coupled. Excessive mocking seems to be the biggest source of evil in that regard.
  - closeparen 7y ago
BaronSamedi 7y ago
This paper is emblematic of a serious problem in the software development field: lack of empirical research. It is unsurprisingly filled with opinion and anecdote, with little mention of research in the area. And in the one actual study cited, "Does Test-Driven Development Really Improve Software Design Quality?", the author mis-characterizes the findings. Not that there is a huge set of research on unit testing to refer to--there isn't--and hence the problem. "Software development has been characterized from its origins by a serious want of empirical facts tested against reality, which provide evidence of the advantages or disadvantages of the different methods, techniques or tools used in the development of software systems."[1] In my view, the best thing we could do is to adopt "Evidence-Based Software Engineering"[2] as other disciplines have. This is more likely to have a major positive impact than the newest and hottest language, tool, or technique. [1] Reviewing 25 Years of Testing Technique Experiments. https://www.researchgate.net/publication/220277637_Reviewing_25_Years_of_Testing_Technique_Experiments https://www.researchgate.net/publication/220277637_Reviewing... [2] Evidence-Based Software Engineering. https://dl.acm.org/citation.cfm?id=999432 https://dl.acm.org/citation.cfm?id=999432
- Frost1x 7y ago
  I agree entirely. The more tendency there is to push development as an "engineering" discipline, the more evidence based research needs to be used to solidify or disprove certain development dogmas repeated over and over again, which in many cases, seemingly have no empirical evidence beyond a few anecdotal success cases. This is the norm in the industry, "you're doing it wrong, the best way is this way..."--based on what evidence that pertains to this case or has proven generalized success applicable here? In most cases, it's someone's successful anecdotal experiences which worked for the specific cases they were involved with. That doesn't mean those approaches can be abstracted away and generalized to all cases, but many in this industry do that regularly and critique others' approaches based on that. It becomes this competitive ego contest: well my work was at x, y, z solving q and was successful--making me an authority on a, b, c's problems solving p because x, y, z is a leader (appeal to authority fallacy)... etc. It's one thing to treat development as much of an art (which to me, it very well still is) but once you start treating it as a concrete discipline, you need to provide the lyme, cement, aggregates, water... evidence and studies showing approaches and how they faired across controls and varied cases.
kissgyorgy 7y ago
> If you find your testers splitting up functions to support the testing process, you’re destroying your system Why Most Unit Testing is Waste 4 architecture and code comprehension along with it. Test at a coarser level of granularity. This is very much in line with DHH's opinion about testing in general (the discussion "TDD is dead" is about this). He said that he doesn't want to split logic for the sake of testing, so HE drives the design NOT tests. I very much agree with this. I think a human can design better code (meaning that's easier to use, so easier to consume by other humans) than any automated process (like TDD). I noticed his blog post quotes this PDF: https://dhh.dk/2014/tdd-is-dead-long-live-testing.html https://dhh.dk/2014/tdd-is-dead-long-live-testing.html
chrisweekly 7y ago
Obligatory(?) "Unit testing without integration testing" pic: https://www.reddit.com/r/ProgrammerHumor/comments/5pbl2q/two_unit_tests_but_no_integration_test/ https://www.reddit.com/r/ProgrammerHumor/comments/5pbl2q/two...
- qznc 7y ago
  We have this one in our office: https://twitter.com/dave1010/status/613601365529657344 https://twitter.com/dave1010/status/613601365529657344
HankB99 7y ago
tl;dr I made it to "Testing does not increase quality; programming and design do" and quit. It seems clear that the author has a strong opinion on this and perhaps that has been formed by exposure to unit tests done wrong. I suppose his article is worth reviewing and asking if any of my unit tests suffer from the problems he identified. I think most of his complaints relate to badly applied unit tests and I think we can all agree that any methodology can be badly applied. That does not provide grounds to condemn the methodology. Having used Fortran (and Macro-11) as the first languages I employed professionally, I do not recall any thrust for unit testing at that time. Maybe it was just the shop I worked in. More recently I have used unit tests for Go, C/C++, Python, Perl, Java, shell scripts and probably some I'm forgetting. I wouldn't consider coding anything w/out some kind of unit test. Back to the quote "Testing does not increase quality; programming and design do", I disagree vehemently with the claim that testing does not increase quality. While true that one cannot 'test in' quality, I find that designing code to be testable provides higher quality results. This is particularly true in languages such as shell scripts that often start small and grow until they are hundreds of lines of in-line code. Testable code is generally better partitioned and structured than it would otherwise be. A second benefit to unit testing is immediate feedback and completion of parts of the system. I get a feeling of accomplishment when I complete something that passes its tests and prefer that to deferring satisfaction until the entire thing works. Finally, if I test the bits in isolation, I can provide them data for all of the corner cases I think could cause trouble and make sure they work for a wider range of inputs than could easily be done during integration testing. When I do get to the point where I put the bits together, I have a much higher success rate with integration testing.
msclrhd 7y ago
When implementing lexers and parsers for language specifications, I have taken the approach of testing each token (keyword, number, etc.) for the lexer for each EBNF symbol. This repeats tests for common keywords and other constructions, but means I can see that the lexer handles all the tokens in each EBNF symbol. In this way, these tests are more BDD (behavioural driven development) in style, but without things like gherkin/cucumber. With this approach, I do full coverage tests for a given symbol/token, and just check the basic symbol/token case elsewhere. I do a similar thing with the parser tests -- have a test for each valid complete symbol production, and then tests for as many error cases that the parser can handle for that symbol. With that base, I can add additional tests for bugs and additional error recovery cases I implement. I never see unit tests as a waste as they provide a set of regression tests that are invaluable for refactoring and making other changes to the code, like implementing new features or better error handling.
he0001 7y ago
I run a one-man project and I write test first unit tests for all of my code. This has enabled me to advance the code with new functions without breaking current API’s. The tests are not in the way and usually doesn’t need to change because I change the implementation. It has been tremendous for productivity because I don’t need to worry about if stuff breaks. I can trust my tests for changing those and so far they have always done that.
derangedHorse 7y ago
This article seems to ignore the fact that unit tests are typically made for other programmers down the line as a form of documentation, as an informal description of business rules, and as a way to gauge whether new code will break the product in a very obvious way.
henrik_w 7y ago
I wrote a response to this article (and the follow-up) here: https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-is-waste/ https://henrikwarne.com/2014/09/04/a-response-to-why-most-un...

192 points by fagnerbrack 7y ago

Doing a typical "front-end" mobile application I write the following tests: * Integration tests that test the API I get a ton of value out of it though it's sometimes hard to guarantee a certain state at the API's end. But I use them while building the API's but they also can detect any kind of problem after an API upgrade etcetera * Unit tests that test data transformations This is stuff like date formatting, building headers into a request given certain input, building more complex ViewModels that take a few structs as input and turn them into something that reflects the actual process happening in the view. They're valuable while building the logic, help separating the logic since you need to make it testable and also help a ton if you find a bug since a bug simply means adding another test case to see if something goes wrong and then fix it. I don't think I even get to 30% code coverage but I think what I cover is super valuable and the other 70% is usually mostly CRUD boiler plate.

hhjinks 7y ago
I personally find testing incoming APIs a waste. Sometimes I write tests to verify that I treat the incoming data correctly if I transform it in fancy ways, but other than that, you should trust the API contract. It's almost like writing tests for the frameworks you're using. Granted, that's from a unit test perspective. Integration tests of APIs are invaluable. I wish we had them already where I currently work.
- Mandatum 7y ago
  > but other than that, you should trust the API contract As someone who's been working in corporate integration for the past 6 years.. Never trust a contract. Be in WSDL, OpenAPI spec, Word documents or otherwise. I've worked with large tech vendors, I've worked with finance, I've worked with large consultancies. The only people that seem to get it right, is the people you don't want to work with because it's soul-crushing - think HL7 et al.
  - hhjinks 7y ago
    The point is you can't write unit tests for broken API contracts, so either you trust that the contract is upheld, or you've got bigger problems on your hands.
    - Mandatum 7y ago
      Do you mean integration tests? I don't think API contracts should be a part of a unit test. A unit test should be self-contained, unless of course there is a tight coupling to an API.. Which I would steer clear from. That's why I value integration tests over most thing. I can see immediately that something is broken at a high level, what business impacts it has and explain what systems are affected.

revskill 7y ago

I agree. Unit Test is for testing API usage, not implementation. We can use it to figure out the API design good or not. But the problem, is API change is very often, and we don't want to change both implementation and test just for the sake of API change. Integration test is enough in most case. Even API is the implementation details of an abstraction.

desc 7y ago

When I started programming, I'd think up lots of clever ways to avoid repeating things. Ten years on, that code is a nightmare to maintain because changing the behaviour for just one call site of hundreds is next to impossible, because everything gets funnelled through one extremely DRY group of modules. Redundancy vs. Dependencies: it's the dependencies which kill you. Redundancy is often a good thing. If you state an algorithm only once, as the implementation, then the next programmer only knows what it does, not whether it's correct. This is, in my opinion, the main value of unit tests: state the algorithm twice, once as implementation and once as expectations, and if they don't agree then something's wrong. While the odds of a bug existing in any line of code haven't changed, the odds that the exact same bug exists in both sides are much lower. Any bugs which survive that are probably in design rather than implementation, ie. my mental model of what the module should achieve is wrong somehow. Catching that is a job for integration tests. I definitely agree with the 80/20 rule here. 100% code coverage is neither necessary nor desirable, and 20% is fine if it's the most valuable 20%.

dang 7y ago

A thread from 2017: https://news.ycombinator.com/item?id=15591190 https://news.ycombinator.com/item?id=15591190 2016: https://news.ycombinator.com/item?id=12666454 https://news.ycombinator.com/item?id=12666454 2016: https://news.ycombinator.com/item?id=11799272 https://news.ycombinator.com/item?id=11799272 2014: https://news.ycombinator.com/item?id=7353767 https://news.ycombinator.com/item?id=7353767

WrtCdEvrydy 7y ago

This is very interesting, when you contrast it to [https://dl.acm.org/citation.cfm?id=3106270 https://dl.acm.org/citation.cfm?id=3106270]. I'm always weary of arguments that rely on 'unit tests are more complex than the code'. If that's true, then it's correct. Tests have to encapsulate the software contract that's being enforced and should be more complex than the underlying code.

biesnecker 7y ago

My experience with testing is like that old adage about advertising, "I know I'm wasting 50% of my money but I don't know which 50%." Most of it is a waste but it's hard to know in advance which test will stop an engineer in a couple of years from altering some fundamental contract and bringing the system down.

ChrisCinelli 7y ago
I have a 3 part article on testing that I need to post... Regarding wasting time on 50% tests... it depends what your system is. In most of the systems (the no mission critical ones, no general libraries) I like to use the 80%/20% rule. I always ask: what are the 20% of the tests that deliver 80% of the value? You learn with experience but if you do not have it, this is a simple way to learn (as a team). During the sprint, have people write down: 1 - The tests that actually failed and found a real problem before the code was shipped. 2 - The tests you wrote for the bugs that were found in production during the sprint. If you haven't written the test yet, ask: what kind of test would have found this bug? Have people in the team share in the retrospective meeting. If you need more time at the beginning you can hold a meeting just for bug sharing. Discuss about the patterns. Commit to write more of these kind of tests. In a few sprint, you should see regression bugs going down. If you also take a look at the tests than never fail, you will also have a sense of the bugs you should not write. Let me know how it goes!
- Pistos2 7y ago
  > If you also take a look at the tests than never fail, you will also have a sense of the bugs you should not write. Just checking: Did you mean "the tests you should not write"?
sametmax 7y ago
My take on that is to make tests using a unit test lib, but a bit more macro. Midway between unit tests and e2e. It's not pure, it's not best practices, but it's great value.
chongli 7y ago
I'll say that any unit test for a bug which would have been caught by a more sophisticated type system is a waste. I don't know how much time people spend writing such "obsolete tests", but I doubt it's insubstantial.

yodon 7y ago

I've generally preferred integration tests to unit tests but Storybook has completely changed how I write React components, for the better.

notus 7y ago
Agreed on integration tests over unit tests. I get the most bang for my buck with them. Storybook is pretty nice. I use it for developing smaller components, more of the building blocks of the application. Have you ever checked out Kent C Dodd's react-testing-library? It was a refreshing approach to testing react components IMO and it has gained quite a bit of adoption.
- yodon 7y ago
  The docs for react-testing-library look like they offer a great deal of wisdom in a tiny package. I find my React components tend to be simple enough that I'm generally far more concerned about whether the CSS works than the JS, leading me to probably keep my focus on Storybook for the moment but I suspect I'll be reaching for react-testing-library before too long. Thanks for the heads up about it.

rileymat2 7y ago

> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. I feel like there is some subtlety here that most will miss. Specifically, "and consider". That is the part we are really bad at.

alkonaut 7y ago
The tests that never failed for years are usually unit tests for fundamental things. Say your app has a math library at the bottom as in my example. It has thousands of years for things like vector additions, matrix multiplication and so on. And of course these never fail - they have no dependencies and we never modify it anymore. But we might modify it, say to plug in some other library to do math in 5 or 10 years. I don’t want to delete them for that reason. Luckily this kind of low level test usually takes very little time - we have a 15k test suite and the 5k “fundamental” tests take seconds while the whole integration level takes hours.
acchow 7y ago
Those tests are helpful when you do major architectural changes, which may not happen every year. Tests also help in general development. Make some changes, then fix the tests. If you are running tests locally on your machine (and offline) how can you be sure that every time a test fails locally the failure is logged?
- kazinator 7y ago
  ... and you don't always know what is a "major architectural change" that's going to break one of these tests, so you can't just relegate those tests into a category that is only run for major architectural changes.
- war1025 7y ago
  There's a difference between tests that haven't broken because they are in stable code, and tests that haven't failed because they are testing tautologies or even just completely failing to break when the underlying code is broken. I've lost count of how many times I've gone into an old test suite at work and found that the tests were still passing even though the code they were testing had completely changed or been removed. Sometimes tests are written very poorly. The codebase benefits from their removal.

sulam 7y ago

I got progressively more and more frustrated with this article, largely because he keeps making statements about the impossibility of covering all states a class may take on (true!) but then followed up with espousing more use of integration and system tests, which are clearly combinatorial harder to test “completely”. He also implied at the very beginning that somehow this was driven by the switch from FORTRAN to OOP, as if FORTRAN avoided this combinatorial explosion (magic!) and it’s only because of polymorphism that we live in the “unit testing is good” world. The logical contradictions eventually overcame me.

danmaz74 7y ago
I saw cases where there really was a waste of stupid unit tests, and no significant tests for the interaction of those units, so I can see where the author is coming from. But my conclusion, after many years of developing with different levels of testing, is that it's a good idea to: * Have at least one unit test for each nontrivial unit, but no unit tests for the trivial ones. But, in these unit tests, use mocking and stubbing only when using real objects isn't feasible, or very inconvenient - isolation is nice to have, but generally overrated in my experience * Create as many unit tests as necessary to feel confident about the edge cases for the most critical units * Implement higher level tests covering the main use cases for the interaction of all the units
hdfbdtbcdg 7y ago
Ironically modern fortran is object orientated...
cryptica 7y ago
For the same number of lines of test code, an integration test case traverses through much more source code than a unit test case. Not only that, but it also checks that the units are interacting properly and passing each other the right data and returning the right data. Good integration tests can also fee us from static typing because the integration tests implicitly check that the interface contracts between all the different units are met internally. It also verifies many other factors which affect the robustness of the software. For example, they can identify race conditions and prevent them in the future.

lumost 7y ago

The utility of unit tests greatly depends on the type of software you're building. Building a DB without tests is difficult and unlikely to win many customers - particularly when news of that gnarly data race comes out. Whereas a crud webapp where everyone can see what's working and what's not is a bit hit or miss. At the very least testing helps communicate to a code reviewer "hey this thing does what I think it does".

ChrisMarshallNY 7y ago

I've never tested 100% of my code. But I've also never written a unit test that didn't expose bugs. That's mainly because I know what parts of the code will be dicey, and focus unit tests on them.

Nasrudith 7y ago
In practice I personally tend to do the reverse. If I encounter a bug I create a unit test to replicate its circumstances and use it to ensure that the bug stays dead and gone.
aey 7y ago
That’s funny. I only write tests to prove that the program does what it’s meant to do and not to find bugs. The goal is to prevent a future programmer, including myself, from breaking any of the declared properties.
- falsedan 7y ago
  Tests don’t prove anything, they just show the intended behaviour. Only full verification proofs show correctness. Too often I’ve seen “tests show it’s correct” suites horribly fail to provide value when the behaviour changes in a unit of business logic and the brittleness needs to be unwound and replaced with robust assumptions.
- ChrisMarshallNY 7y ago
  For me, it’s all about maximizing quality; which is usually about bugs. For example, I’m currently writing an ONVIF driver. This needs to address hundreds of devices. I only have about ten to test against. They feature a wide range of implementation, but they are, by no means, full coverage. I’m gonna need to mock up some crazy, nonexistent devices to stress a few particular parts of the library. If I can make it work with them, then chances are better than even that it will work with devices I can’t directly test. I use test harnesses to ensure functionality. My test harnesses tend to be huge; often, a lot bigger than actual shipping implementations.
- deleted 7y ago
  [deleted]

thisgoodlife 7y ago

Most insurance is waste

Toine 7y ago
If your house didn’t burn in the last two years, consider not insuring it anymore.

ed-209 7y ago

"The cross product of those paths with the possible state configurations of all global data (including instance data which, from a method scope, are global) and formal parameters is indeed very large. And the cross product of that number with the possible sequencing of methods within a class is countably infinite." Sounds more like a condemnation of OOP than of unit testing, and I do genuinely feel sorry for the unit testing, OOP purists out there. I prefer to design more functional methods, which operate on parameters and injected config instead of instance state and/or globals (cringe). Incidently, this approach makes full coverage attainable. "Large functions for which 80% coverage was impossible were broken down into many small functions for which 80% coverage was trivial. ... Of course, this also meant that functions no longer encapsulated algorithms. It was no longer possible to reason about the execution context of a line of code in terms of the lines that precede and follow it in execution" I can reason about such an implentation MUCH more effectively by glancing at the small bit of higher level code which integrates everything (and as mentioned above by foresaking instance state and polymorphism). This strikes me as a bit like advocating a flat directory structure cause it's important to be able to see all your files at once.

ereyes01 7y ago
Yeah a lot of the 90s/early 2000s OOP stuff I learned in school seemed to always result in really tightly coupled systems and bespoke webs of tests and fixtures that strung along weird dependency chains in unwieldy spaghetti piles that did no good. Following TDD has helped me land at decoupled functional interfaces like the ones you've described, and it all scales and composes so nicely, yet stays very tractable. Robert Martin sketched 2 diagrams in [1] that elegantly illustrate these two different design patterns and how testable usually means composable and more tractable: [1] http://blog.cleancoder.com/uncle-bob/2017/03/03/TDD-Harms-Architecture.html http://blog.cleancoder.com/uncle-bob/2017/03/03/TDD-Harms-Ar...
- MetalGuru 7y ago
  I wish he gave actual examples to illustrate what he was talking about. So you have an additional API layer the tests hit to call the functions you’re testing? Do endpoints map to classes? Modules? So aren’t you just tightly coupling this API to your service? When the service changes, you still have to update the API. Can you explain to me how this solves the problem?

the_arun 7y ago

Unit Tests = The assurance/confidence that you get, if something changes logic, this test will break & you will know it. Aside from just limiting bugs - this is more powerful.

war1025 7y ago
Having a robust test suite is very nice. Having an overly pedantic test suite is not. Things like getters / setters that use the same library you've used 1000 other places have no need to be tested again. Add tests for the base concepts and trust that they'll continue working based on those.
mobjack 7y ago
95% of the time when a unit test fails for me, it is the test that need fixing instead of my code. It hardly inspires confidence.
- winrid 7y ago
  Is that because you're not writing tests while you're adding code and just breaking old tests?
- qznc 7y ago
  To me this suggests that there is no clear understanding what your units are supposed to do. My second theory would be that the unit tests are written by the inexperienced developers while the good developers write the other code.
  - mobjack 7y ago
    The tests will involve 5 mocked dependencies just to test a simple if statement. They test the implemention and are a waste of time. Isolated units that are well tested usually don't need to be updated in my experience so they rarely break.
    - jrhurst 7y ago

miccah 7y ago

> If you find your testers splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity. The emphasis here should be on the reason for splitting up functions. Long, complex functions can be difficult to understand, and removing a few lines in exchange for a (well named) function call is very beneficial for the reader. The opportunity for testing comes from this delegation. A function call is a contract, and the test ensures it complies. Now the reader can comprehend what the code is doing at a higher level; trusting the sub functions do what they intend.

tsss 7y ago
It's quite the opposite: The whole point of the architecture is often to facilitate testing. If we didn't have to write tests we could do without a lot of that boilerplate.

njharman 7y ago

I did not read the article. I did read all comments Unit tests for the most part aren't about "testing". They are developer tool. To verify rthat modifications (refactoring, additions, bug fixes, etc) doesn't break contracts etc. Oh and showing that your code is a codependent mess of poorly isolated spagehtti, if your unit tests are hard to write, the code under test is a mess. Inittests are more useful in languages with loose or poor type systems. Many unit tests are not written well. testing more than the interface/contract. Or full of complexity boilerplate and mock. Which means code needs fixing.

cryptica 7y ago
Unit tests don't guarantee that your code units are well designed. You can still write unit tests for spaghetti code. In fact, sometimes it encourages it because it encourages antipatterns like dependency injection; because developers get the urge to inject mock objects into the units from the unit tests. Just think about it at a high level. If your module depends on another module, is it really always necessary that this module be substitutable for another module that has the same interface (e.g. another similar module or a mock)? 99% of the time, the answer is no! In fact it's often a problem because this wrongly assumes that if a class exposes the required interface, then it is compatible. There is more to compatibility than just interface; if the submodule maintains its own state (OOP), then its behaviour can change in a way which could break the dependent logic. You can't have Dependency Injection for everything because it's never so simple; these customizable parts of the code have to be designed very carefully.
- deleted 7y ago
  [deleted]
- jrhurst 7y ago
  I actually don't get where you are going with this breakdown. Can you give me an example.
  - cryptica 7y ago
    I mean when you have a class which, instead of importing/including/requiring its dependencies directly, expects instances of various classes to be passed into its constructor for example. So the class cannot do its job unless you pass its dependencies into its constructor... This is an antipattern which unit tests tend to encourage because it allows you to decouple the unit logic from its dependencies and it makes it easy to inject mocks in the place of dependencies. A unit test should only test a single class so that means you need to mock out all other classes which your class depends on. Mocking out dependencies in the test code is difficult or not feasible and that's why developers often resort to Dependency Injection in their source code but as mentioned before, it is an antipattern. Also, I noticed that a lot of developers confuse unit tests with integration tests but the definition is quite clear: If your test covers more than one class without mocks, then it's an integration test. Integration testing does not necessarily mean end-to-end testing. It could just be a single class with its internal dependencies.

deleted 7y ago

[deleted]

devin 7y ago

Absent from this discussion is the use of generative testing. At the unit level they can automatically enumerate a variety of scenarios the programmer would likely overlook.

steve_adams_86 7y ago
I just discovered this, only it was called 'property based testing'. Do you recommend any resources for learning to use it effectively? I haven't dug in yet, but so far it hasn't clicked when thinking about it. I like the idea a lot, but I guess I don't see where it would be most useful for my own code.

quality_theater 7y ago

One of the things I dislike the most about unit tests is how it's used for "quality theater". Some examples: - Trying to use coverage percentage metric as a sign of quality. As if a simple percentage means anything about the quality of the tests. It's as useless as using lines of code as a way to measure progress. - Not recognizing that useless unit tests are harmful for the code maintenance. It makes refactoring code into better structure difficult and developers just give up because it's too much work to fix the tests that weren't even providing any value in the first place. This ignorance is expressed in statements like how there's a testing "pyramid" with unit tests at the bottom and end to end tests at the top. Which is a nice sounding soundbite and image, but is useless. Forget pyramids, just write tests where it makes sense. - Code review comments that try to look smart with "where's the unit test?" Then the developer doesn't want a long drawn out fight about how a unit test would be useless here since there's a huge crowd just cargo cult yelling "code coverage!" "unit tests are good!" "pyramid!" So the developer just writes the stupid test to get the code merged. This is also an example of how harmful code reviews can sometimes be, when there's a popular stupid idea, code reviews perpetuate it because the developers who know better just get tired of fighting the same fight over and over again. I really hate useless unit tests. I've seen tests that setup a mock, configure it to return a value when called, call the code using the mock, then check the returned value is the mocked value. This tested absolutely nothing! I've seen unit tests that verify every line of the method was called, completely pointless. The point is supposed to be to verify that for a given input, it has a given output, not lock it into a specific implementation by verifying every line of code ran a certain way. There is an idea for structuring software "functional core, imperative shell". Write the software this way and the natural place for unit tests and integration tests becomes obvious. But nope, the industry is all about unit test coverage percentage, stupid pyramids, looking good in code reviews. It's all quality theater, not actual focus on quality.

reallydontask 7y ago
> I really hate useless unit tests. I've seen tests that setup a mock, configure it to return a value when called, call the code using the mock, then check the returned value is the mocked value. This tested absolutely nothing! I've had a similar disagreement about the validity of such a test. The other mocking issue I've seen, is people mocking blackbox third-party APIs over which they have absolutely no control, which sometimes leads to passing unit tests, failing integration tests and head scratching all around.

kstenerud 7y ago

After 15 years of writing unit tests, I haven't reached the same conclusion as the author. What I've found in my time is that unit testing can be good, but like anything it's not a panacea. It requires discipline, and like normal code, it has code smells. Black box unit tests are the most likely to be good tests, and white box unit tests are the most likely to be bad tests. The more you depend on the inner workings of a function in order to test it, the more likely it is that you are coupling your test to the implementation rather than the purpose of the unit being tested. Once you tie to the implementation, refactoring becomes a LOT harder, because changes will break the tests even if they don't break the functionality. Mocks are also a major source of trouble, and more likely to be a code smell. If your tests are using a mock to test how many times your unit called it, either your tests are bad or your architecture is wrong. There are three main kinds of code: - Code that fetches data - Code that stores data - Code that transforms data Mocks are necessary when you mix these. If you have a function that opens a DB connection, fetches data, transforms the data, and then stores the data, you now have an extra problem to deal with (the database), when all you wanted to do was test the transformation. Things would be far easier if you separated the transformation out, tested that in isolation, and then integrated that encapsulated functionality with fetching/storing code. This also improves separation of concerns and code duplication, since now your fetching/storing code can be generalized and also tested in isolation. Actually I lie. There is a fourth kind of code: code that modifies state. This is the evilest, smelliest code around, and it's also something that unfortunately we can't get completely away from. But we can manage it, by isolating state, reducing the need for or scope of the state, and providing "configuration object" function entry points to make testing these monstrosities less nasty. Code coverage is not just a measure of quality, but also of waste. If your code is not being called, then one of three things is happening: 1. It's error checking code for another API it's calling, which you normally shouldn't be writing tests for (unless that API is known to be buggy and you need to guard against it). 2. It's not contributing to the goals of the program, and can be taken out. 3. It does contribute to the goals of the program, in which case you need a test for it. You can't reach 100% code coverage because of (1). But you absolutely should check WHICH code is covered in your tests because of (2) and (3). Anything higher than 80% coverage is pure luck, and tells you nothing about quality or wastage. In many cases, even 60-70% is sufficient.

dmitriid 7y ago

Kent Beck (yes, the Kent Beck): “I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence” https://stackoverflow.com/a/153565 https://stackoverflow.com/a/153565

nickthemagicman 7y ago

I don't just code for a single project. I create libraries when I write a piece of code that I know will be useful down the line. And for those libraries that are going to be reused throughout many applications I think unit testing is gold.

fogetti 7y ago

The thing is that most people don't understand that the code needs to be tested anyway. If you don't test it, you are just an idiot who has no proof that the code does what it claims to do. Now because you have to test it anyway, you can just simply spend the same time writing a unit test instead of executing the code with manually configured non-repeatable test cycles, a.k.a clicking through the UI, or sending Postman requests, etc. Also it's kind of selfish not to make something repeatable by others. And as someone pointed out before me, unit tests are not about testing at all. It's a documentation about the system, and what it's supposed to do. Also it's the way to stop the next person who works on the code to ruin something by not knowing the business rules.

phereford 7y ago
To add to this sentiment, tests also give teams confidence that is needed. It gives engineers confidence when developing new features because the test suite will break if you introduce a regression. It gives engineers confidence during deployment/CD. While the tangible benefits of unit tests are very important, there are other intangible benefits that are equally important.
- didericis 7y ago
  I’d say that confidence is a tangible factor. The single most important reason to test your code seems to be that it allows you to refactor your code while ensuring it still meets the original outside expectations. The author’s main gripe seems to be about tightly coupled tests that make refactoring larger systems more difficult, and about prioritizing meeting arbitrary metrics (lines of codes covered) rather that thinking critically about the actual benefits. This is in line with his emphasis on integration tests determined by the business. I think that same thought process can be applied to internal components of code as well, and you can empirically determine the quality of your approach (roughly) by evaluating how often your tests change. If nearly every change breaks a test, that probably means they’re low value/too tightly coupled. Excessive mocking seems to be the biggest source of evil in that regard.
- closeparen 7y ago

BaronSamedi 7y ago

This paper is emblematic of a serious problem in the software development field: lack of empirical research. It is unsurprisingly filled with opinion and anecdote, with little mention of research in the area. And in the one actual study cited, "Does Test-Driven Development Really Improve Software Design Quality?", the author mis-characterizes the findings. Not that there is a huge set of research on unit testing to refer to--there isn't--and hence the problem. "Software development has been characterized from its origins by a serious want of empirical facts tested against reality, which provide evidence of the advantages or disadvantages of the different methods, techniques or tools used in the development of software systems."[1] In my view, the best thing we could do is to adopt "Evidence-Based Software Engineering"[2] as other disciplines have. This is more likely to have a major positive impact than the newest and hottest language, tool, or technique. [1] Reviewing 25 Years of Testing Technique Experiments. https://www.researchgate.net/publication/220277637_Reviewing_25_Years_of_Testing_Technique_Experiments https://www.researchgate.net/publication/220277637_Reviewing... [2] Evidence-Based Software Engineering. https://dl.acm.org/citation.cfm?id=999432 https://dl.acm.org/citation.cfm?id=999432

Frost1x 7y ago
I agree entirely. The more tendency there is to push development as an "engineering" discipline, the more evidence based research needs to be used to solidify or disprove certain development dogmas repeated over and over again, which in many cases, seemingly have no empirical evidence beyond a few anecdotal success cases. This is the norm in the industry, "you're doing it wrong, the best way is this way..."--based on what evidence that pertains to this case or has proven generalized success applicable here? In most cases, it's someone's successful anecdotal experiences which worked for the specific cases they were involved with. That doesn't mean those approaches can be abstracted away and generalized to all cases, but many in this industry do that regularly and critique others' approaches based on that. It becomes this competitive ego contest: well my work was at x, y, z solving q and was successful--making me an authority on a, b, c's problems solving p because x, y, z is a leader (appeal to authority fallacy)... etc. It's one thing to treat development as much of an art (which to me, it very well still is) but once you start treating it as a concrete discipline, you need to provide the lyme, cement, aggregates, water... evidence and studies showing approaches and how they faired across controls and varied cases.

kissgyorgy 7y ago

> If you find your testers splitting up functions to support the testing process, you’re destroying your system Why Most Unit Testing is Waste 4 architecture and code comprehension along with it. Test at a coarser level of granularity. This is very much in line with DHH's opinion about testing in general (the discussion "TDD is dead" is about this). He said that he doesn't want to split logic for the sake of testing, so HE drives the design NOT tests. I very much agree with this. I think a human can design better code (meaning that's easier to use, so easier to consume by other humans) than any automated process (like TDD). I noticed his blog post quotes this PDF: https://dhh.dk/2014/tdd-is-dead-long-live-testing.html https://dhh.dk/2014/tdd-is-dead-long-live-testing.html

chrisweekly 7y ago

Obligatory(?) "Unit testing without integration testing" pic: https://www.reddit.com/r/ProgrammerHumor/comments/5pbl2q/two_unit_tests_but_no_integration_test/ https://www.reddit.com/r/ProgrammerHumor/comments/5pbl2q/two...

qznc 7y ago
We have this one in our office: https://twitter.com/dave1010/status/613601365529657344 https://twitter.com/dave1010/status/613601365529657344

HankB99 7y ago

tl;dr I made it to "Testing does not increase quality; programming and design do" and quit. It seems clear that the author has a strong opinion on this and perhaps that has been formed by exposure to unit tests done wrong. I suppose his article is worth reviewing and asking if any of my unit tests suffer from the problems he identified. I think most of his complaints relate to badly applied unit tests and I think we can all agree that any methodology can be badly applied. That does not provide grounds to condemn the methodology. Having used Fortran (and Macro-11) as the first languages I employed professionally, I do not recall any thrust for unit testing at that time. Maybe it was just the shop I worked in. More recently I have used unit tests for Go, C/C++, Python, Perl, Java, shell scripts and probably some I'm forgetting. I wouldn't consider coding anything w/out some kind of unit test. Back to the quote "Testing does not increase quality; programming and design do", I disagree vehemently with the claim that testing does not increase quality. While true that one cannot 'test in' quality, I find that designing code to be testable provides higher quality results. This is particularly true in languages such as shell scripts that often start small and grow until they are hundreds of lines of in-line code. Testable code is generally better partitioned and structured than it would otherwise be. A second benefit to unit testing is immediate feedback and completion of parts of the system. I get a feeling of accomplishment when I complete something that passes its tests and prefer that to deferring satisfaction until the entire thing works. Finally, if I test the bits in isolation, I can provide them data for all of the corner cases I think could cause trouble and make sure they work for a wider range of inputs than could easily be done during integration testing. When I do get to the point where I put the bits together, I have a much higher success rate with integration testing.

msclrhd 7y ago

When implementing lexers and parsers for language specifications, I have taken the approach of testing each token (keyword, number, etc.) for the lexer for each EBNF symbol. This repeats tests for common keywords and other constructions, but means I can see that the lexer handles all the tokens in each EBNF symbol. In this way, these tests are more BDD (behavioural driven development) in style, but without things like gherkin/cucumber. With this approach, I do full coverage tests for a given symbol/token, and just check the basic symbol/token case elsewhere. I do a similar thing with the parser tests -- have a test for each valid complete symbol production, and then tests for as many error cases that the parser can handle for that symbol. With that base, I can add additional tests for bugs and additional error recovery cases I implement. I never see unit tests as a waste as they provide a set of regression tests that are invaluable for refactoring and making other changes to the code, like implementing new features or better error handling.

he0001 7y ago

I run a one-man project and I write test first unit tests for all of my code. This has enabled me to advance the code with new functions without breaking current API’s. The tests are not in the way and usually doesn’t need to change because I change the implementation. It has been tremendous for productivity because I don’t need to worry about if stuff breaks. I can trust my tests for changing those and so far they have always done that.

derangedHorse 7y ago

This article seems to ignore the fact that unit tests are typically made for other programmers down the line as a form of documentation, as an informal description of business rules, and as a way to gauge whether new code will break the product in a very obvious way.

henrik_w 7y ago

I wrote a response to this article (and the follow-up) here: https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-is-waste/ https://henrikwarne.com/2014/09/04/a-response-to-why-most-un...