8 ms·

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock dow

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful to your project. If you build square wheels for example, you may not realize that they're not designed correctly until you attach them to the car and realize that the car doesn't function well with them. It makes no sense to preemtively write unit tests for a component which has a very high likelihood of not being in its final desired state; you're just giving yourself more work to refactor the unit tests over and over; or worse, you're afraid of refactoring the tests and so you lie to yourself thinking that square wheels are fine. Integration tests are by far the most useful tests to have at the beginning and middle stages of the project; integration tests allow you to keep your focus on the real goals of the project and not get stuck on designing the perfect square wheel.

rypskar 8y ago
What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.
- twodave 8y ago
  The basic rule I go by is whether or not the test is self-contained. For example if you are making a trip to a database or hitting a 3rd party dll then it’s an integration test. I thought this was pretty widely accepted.
  - jondubois 8y ago
    Wikipedia disagrees with that definition: "Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method.[2] Unit tests are short code fragments[3] created by programmers or occasionally by white box testers during the development process. It forms the basis for component testing.[4] Ideally, each test case is independent from the others. Substitutes such as method stubs, mock objects,[5] fakes, and test harnesses can be used to assist testing a module in isolation. Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended." https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing
    - twodave 8y ago
      I don’t think that definition necessarily is in conflict with my point.
  - rypskar 8y ago
    I would say that for most CRUD applications it is much easier to have the tests run against an sqlite database instead of mocking it, so would let unit tests update the database and write tests to check if the data in the database match what I would expect from the input. If you change how you store the data, it isn't a side effect of changes you are doing but the main effect so the test could be rewritten anyway. Some call that an integration test
    - hopler 8y ago
      Mocking is a last resort and usually the wrong choice and used incorrectly. In memory fakes like sqlite (or better, running your real database in a small local inmemory footprint) is usally the way to get fast, correct, stable tests.
    - hopler 8y ago
      Mocking is a last resort and usually the wrong choice and used incorrectly. In memory fakes like sqlite (or better, running your real database in a small local inmemory footprint) is usually the way to get fast, correct, stable tests.
    - twodave 8y ago
      I wouldn’t mock a data layer unless I was unit testing (for example) response codes on a route depending on the data layer. In those cases you aren’t interested in the data as much as how it is handled. I like to be able to point an integration test at a “clean” database in order to test a data layer. In this way you can have separate tests to basically ensure your API responses and database queries are all valid given your database is in a valid state and the API request is valid. With this kind of setup I rarely actually have to run an application anymore to feel it is in good working condition, and that saves me a LOT of time.
- jondubois 8y ago
  Unit tests should test individual subcomponents of your system in isolation. If your project is to build a car from scratch, then the unit cannot be the car. If your project is to build an autonomous fleet of self-driving cars, then from your project's perspective, the car could be a unit; the project to build the autonomous car would have different units from the project which manages the fleet of cars.
  - rypskar 8y ago
    Exactly, context matter. Most cases when some is writing for or against unit testing they miss the part about finding what they define as an unit. If the class need other classes to work, the whole set can be seen as one unit. Who didn't want to test a private method before they see that a unit is more than a function?
    - Jach 8y ago
      I like to call it the 'system under test' / SUT. Then I can go on to say a unit test is a test that's fast and if it fails points to a specific problem in the SUT. Smaller SUTs obviously help satisfy both. If setting up collaborators in a bigger SUT leads to either unacceptable runtimes or too much brittleness from many things being able to go wrong, it's a signal you're not getting the value of unit testing in that place. (The test itself may or may not be valuable, other types of testing have their own value propositions.)
  - jbergens 8y ago
    This means that the unit tests will miss a lot of possible errors. Nothing wrong with that, just use integration testing or functional tests to cover the other part.
- adrianhel 8y ago
  It does not matter. Are your tests preventing defects? Is it possible to run new or affected tests quickly? Are your tests reasonably easy to maintain? When I am testing an application that is basically a layer between my app and a database, it does not make sense to mock the database. What then am I actually testing. When I am testing a package that is pure logic, it makes sense to testing it in isolation. When I am building a user interface that is in flux, the best tests are manual. My definition of a unit test: The smallest piece of functionality that can be tested in isolation and provide value. This does not test a damned thing: assert.equal(dbQuery, "SELECT * FROM users")
adrianN 8y ago
IMHO unit tests are replacements for an interface contract specification language. You encode the requirements for the interface of your unit (whatever that may be) into tests. Writing unit tests early means that you think about the interface early. This way you can find awkward or wrong requirements early. Of course they're no replacement for integration tests, but they're supposed to be much cheaper to write, to run, and to change than integration tests.
- Cpoll 8y ago
  > interface contract specification language Sounds a bit like C#'s code contracts. A team I was on tried it some years ago, and I think they were a great substitute for a bulk of the unit tests we wrote. It didn't catch on, unfortunately.
deleted 8y ago
[deleted]
snidane 8y ago
Very good comment about SW testing in general. In my experience the only time when unit tests were useful was when they were integration tests at the same time. For example when writing a data structure or algoritms or programming language/compiler. There is a clear boundary to such a system and one can reason about it in terms of input and output black box. As such the test for this system is an integration test, but due to its isolation it looks so simple that someone would call it a unit test. Second example would be all TDD blog posts, where the author tests some trivial entity with perfect isolation. That confuses integration tests with unit tests again. The biggest mistake people do when they write tests (or library functions) too early is that they miss the target. The first part of writing code is not to produce a clean and maintainable system, but to get the requirements right. Listening to people seldomly works, it is usually better to observe people interacting with some prototype and infer requirements from that. In words of Henry Ford, the goal is to find that the requirements are for a 'car' and not just for 'faster horses'. People who fossilize the requirements too early miss the target completely in my experience.
mannykannot 8y ago
More generally, the idea that you can conquer complexity by writing only small components overlooks the fact that the complexity is found in the interactions of the components. Modularity is necessary, but not sufficient, for success in software development.
austincheney 8y ago
It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise way or it doesn't. Everything else is extraneous, though you may have to test the application in wildly different ways with wildly different means of automation to validate all feature support. Feature tests are executed by running the application with a given input and configuration and comparing the result against a known expected output. Provided sufficient feature coverage this is enough to test for regression. While the feature tests are running each test runs against a clock so that dramatic swings in performance can be qualified with numbers. When that is not enough, as in the case of accessibility, have people (actual users) perform feature tests. Other problems like bad design, redundant features, or poor code quality are qualified using code reviews and validation tools, like a linter.
- JoeAltmaier 8y ago
  In a complex system a feature test can detect a problem but not diagnose it. For that a unit test is desirable. The hole you get into is, lots of units are changed and a feature quits working and everybody says 'its not me!' and it doesn't get fixed.
  - austincheney 8y ago
    > and everybody says 'its not me!' and it doesn't get fixed. Test automation is not a remedy to prop up broken leadership. I would appoint an arbitrary owner of the defect and that person would visit with other people as necessary to remove or dismiss various functionality from blame one by one. Once the appropriate collision of changes is discovered it will become more clear how to address resolution.
    - JoeAltmaier 8y ago
      ...and they would do that by writing individual unit tests.
humanrebar 8y ago
> The purpose of unit tests is to lock down the project's source code once it's essentially complete Good unit tests (like doctests) also provide superior documentation and examples. Sure, worrying about documentation too soon is bad, but sometimes when doing early research, getting details clarified and written down is paramount and tests, maybe even unit tests, are a nice tool to have when you want it.
th456789123 8y ago
unit tests is not only to catch regression and it does not lock down anything. I found that doing TDD actually helps you design better, it naturally force you to write easily testable code which translate into more readable and maintainable code. Also well written UT prove very useful as living documentation. When you put the same care in them as you do in production code, you can very easily add features not needing to reverse-engineer the all thing. UT is one of thing things that may seem counter-intuitive but if you stick to it for a little while you see lots of positive effects. A bit like forcing yourself to hold the neck properly when learning the guitar, it slows you down at 1st, hurts a little, but if you do it you'll play better.
mabbo 8y ago
> The purpose of unit tests is to lock down the project's source code I disagree slightly. If you're writing unit tests such that the code itself can't change without breaking the tests, perhaps the tests have been too tightly coupled. The end goal of unit tests is to secure the expected functionality of the unit of code such that the tests will fail if the functionality stops working. But if changes to the code (refactoring, changing the ordering of certain lines, etc) causes the tests to break, then what was tested wasn't the functionality but the implementation. You might as well write a test that loads each source file as a string and verifies it equals what you've written. An example: my favorite set of tests I've written lately were for some classes to handle deduplication of a large set of records. One implementation wrote to disk (for data sets too large to fit into memory) while the other did it purely in memory. Since their functionality was supposed to be identical, I wrote one set of tests that both ran on. The tests could not be specific to the code written because it had to work on two sets of code. If we wrote a third implementation, the same tests should work for it as well. (But what you said about integration tests: 100% agree)
SkyPuncher 8y ago
I agree. The project I'm working on hovers around 70% coverage. As far as I can tell, we'd likely loose productivity pushing for any better coverage. We get some bugs/issues that could be identified by better coverage, but most of our "bugs" stem from real world issues or differing views on functionality. Getting that right takes a ton of effort that simply isn't worth it right now.
- dtech 8y ago
  Yep, testing definitely has a hugely decreasing marginal value. I personally like the approach of writing a small amount of "obvious" tests to detect large functional breakage, and add a test for a bug once it's been identified aka a regression test. Imo it's maximum test value per effort. Of course it depends on how impactful a bug could be.
didibus 8y ago
Very well said, I'm in your camp as well. The major issue is no one ever learns about or talks about the "Unit" in unit tests. But a Unit is supposed to be a component, albeit a very small one, in OOP, that normally translates to a Class. Your unit is a given Class, assuming its API doesn't change, the unit tests should continue working. If you havn't settled on an API for that unit yet, the tests are going to be coupled to a quick moving target, and that, in my opinion, is a waste of time. That said, people don't really design their Classes as unit in a bigger set, with self-contained API and SOLID principles. That's the first mistake. And that then leads to tests which assert not a Unit, but some random piece of logic within a Unit, or against things that are not Units of their own, but already tightly coupled and leaky containers. Which ends up both with bad designed software that is inflexible and quick to rot, as well as poor tests which prevent you from making any meaningful design improvements to it, without all the tests breaking.
brlewis 8y ago
It's true that obsessing over 100% code coverage is a bad thing. It's false that the purpose of unit tests is to lock down the project's source code. Unit tests have multiple purposes, but that isn't one of them. The main purpose of unit tests is to allow developers to quickly reach confidence that code works, be it a code change or new code. There is no such thing as writing unit tests too early.
stcredzero 8y ago
The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. This is only true for development environments which have particular weaknesses. If your tools for code changes and refactoring easily cover changing your unit tests while you are refactoring, then you can use unit tests for rapid development. The use of Unit Tests in present day "agile" methods actually originates in an environment where this was the norm. (Extreme Programming, as developed at the Chrysler C3 project in VisualWorks Smalltalk, using the phenomenal Refactoring Browser.) Your statement above would be 180 degrees turned around in an environment like that. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful to your project. Reading this just makes me sigh. Not all environments have the same cost/benefit factors in all parts of the edit/test/debug cycle. Not all environments and languages have the same costs for doing similar development operations. As a matter of fact, these are generally very, very different, even across projects using the same language. Change those cost/benefit factors, and the pronouncements you are making become more valid or less valid, depending. Not understanding that those cost/benefit factors can change and change exactly what your best practices are -- this is one of the most pervasive big misunderstandings that fosters poor management of programming projects. (EDIT: I think a lot of the annoyance many people have with "agile" methods, stems from practices which work well in one environment migrated to another environment where they don't work well at all!)

rypskar 8y ago

What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.

twodave 8y ago
The basic rule I go by is whether or not the test is self-contained. For example if you are making a trip to a database or hitting a 3rd party dll then it’s an integration test. I thought this was pretty widely accepted.
- jondubois 8y ago
  Wikipedia disagrees with that definition: "Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method.[2] Unit tests are short code fragments[3] created by programmers or occasionally by white box testers during the development process. It forms the basis for component testing.[4] Ideally, each test case is independent from the others. Substitutes such as method stubs, mock objects,[5] fakes, and test harnesses can be used to assist testing a module in isolation. Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended." https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing
  - twodave 8y ago
    I don’t think that definition necessarily is in conflict with my point.
- rypskar 8y ago
  I would say that for most CRUD applications it is much easier to have the tests run against an sqlite database instead of mocking it, so would let unit tests update the database and write tests to check if the data in the database match what I would expect from the input. If you change how you store the data, it isn't a side effect of changes you are doing but the main effect so the test could be rewritten anyway. Some call that an integration test
  - hopler 8y ago
    Mocking is a last resort and usually the wrong choice and used incorrectly. In memory fakes like sqlite (or better, running your real database in a small local inmemory footprint) is usally the way to get fast, correct, stable tests.
  - hopler 8y ago
    Mocking is a last resort and usually the wrong choice and used incorrectly. In memory fakes like sqlite (or better, running your real database in a small local inmemory footprint) is usually the way to get fast, correct, stable tests.
  - twodave 8y ago
    I wouldn’t mock a data layer unless I was unit testing (for example) response codes on a route depending on the data layer. In those cases you aren’t interested in the data as much as how it is handled. I like to be able to point an integration test at a “clean” database in order to test a data layer. In this way you can have separate tests to basically ensure your API responses and database queries are all valid given your database is in a valid state and the API request is valid. With this kind of setup I rarely actually have to run an application anymore to feel it is in good working condition, and that saves me a LOT of time.
jondubois 8y ago
Unit tests should test individual subcomponents of your system in isolation. If your project is to build a car from scratch, then the unit cannot be the car. If your project is to build an autonomous fleet of self-driving cars, then from your project's perspective, the car could be a unit; the project to build the autonomous car would have different units from the project which manages the fleet of cars.
- rypskar 8y ago
  Exactly, context matter. Most cases when some is writing for or against unit testing they miss the part about finding what they define as an unit. If the class need other classes to work, the whole set can be seen as one unit. Who didn't want to test a private method before they see that a unit is more than a function?
  - Jach 8y ago
    I like to call it the 'system under test' / SUT. Then I can go on to say a unit test is a test that's fast and if it fails points to a specific problem in the SUT. Smaller SUTs obviously help satisfy both. If setting up collaborators in a bigger SUT leads to either unacceptable runtimes or too much brittleness from many things being able to go wrong, it's a signal you're not getting the value of unit testing in that place. (The test itself may or may not be valuable, other types of testing have their own value propositions.)
- jbergens 8y ago
  This means that the unit tests will miss a lot of possible errors. Nothing wrong with that, just use integration testing or functional tests to cover the other part.
adrianhel 8y ago
It does not matter. Are your tests preventing defects? Is it possible to run new or affected tests quickly? Are your tests reasonably easy to maintain? When I am testing an application that is basically a layer between my app and a database, it does not make sense to mock the database. What then am I actually testing. When I am testing a package that is pure logic, it makes sense to testing it in isolation. When I am building a user interface that is in flux, the best tests are manual. My definition of a unit test: The smallest piece of functionality that can be tested in isolation and provide value. This does not test a damned thing: assert.equal(dbQuery, "SELECT * FROM users")

adrianN 8y ago

IMHO unit tests are replacements for an interface contract specification language. You encode the requirements for the interface of your unit (whatever that may be) into tests. Writing unit tests early means that you think about the interface early. This way you can find awkward or wrong requirements early. Of course they're no replacement for integration tests, but they're supposed to be much cheaper to write, to run, and to change than integration tests.

Cpoll 8y ago
> interface contract specification language Sounds a bit like C#'s code contracts. A team I was on tried it some years ago, and I think they were a great substitute for a bulk of the unit tests we wrote. It didn't catch on, unfortunately.

deleted 8y ago

[deleted]

snidane 8y ago

Very good comment about SW testing in general. In my experience the only time when unit tests were useful was when they were integration tests at the same time. For example when writing a data structure or algoritms or programming language/compiler. There is a clear boundary to such a system and one can reason about it in terms of input and output black box. As such the test for this system is an integration test, but due to its isolation it looks so simple that someone would call it a unit test. Second example would be all TDD blog posts, where the author tests some trivial entity with perfect isolation. That confuses integration tests with unit tests again. The biggest mistake people do when they write tests (or library functions) too early is that they miss the target. The first part of writing code is not to produce a clean and maintainable system, but to get the requirements right. Listening to people seldomly works, it is usually better to observe people interacting with some prototype and infer requirements from that. In words of Henry Ford, the goal is to find that the requirements are for a 'car' and not just for 'faster horses'. People who fossilize the requirements too early miss the target completely in my experience.

mannykannot 8y ago

More generally, the idea that you can conquer complexity by writing only small components overlooks the fact that the complexity is found in the interactions of the components. Modularity is necessary, but not sufficient, for success in software development.

austincheney 8y ago

It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever. In my mind you only need 1 kind of testing: feature tests. Does the application provide the expected output for a given input and/or configuration. The application does all that it claims to do in a very precise way or it doesn't. Everything else is extraneous, though you may have to test the application in wildly different ways with wildly different means of automation to validate all feature support. Feature tests are executed by running the application with a given input and configuration and comparing the result against a known expected output. Provided sufficient feature coverage this is enough to test for regression. While the feature tests are running each test runs against a clock so that dramatic swings in performance can be qualified with numbers. When that is not enough, as in the case of accessibility, have people (actual users) perform feature tests. Other problems like bad design, redundant features, or poor code quality are qualified using code reviews and validation tools, like a linter.

JoeAltmaier 8y ago
In a complex system a feature test can detect a problem but not diagnose it. For that a unit test is desirable. The hole you get into is, lots of units are changed and a feature quits working and everybody says 'its not me!' and it doesn't get fixed.
- austincheney 8y ago
  > and everybody says 'its not me!' and it doesn't get fixed. Test automation is not a remedy to prop up broken leadership. I would appoint an arbitrary owner of the defect and that person would visit with other people as necessary to remove or dismiss various functionality from blame one by one. Once the appropriate collision of changes is discovered it will become more clear how to address resolution.
  - JoeAltmaier 8y ago
    ...and they would do that by writing individual unit tests.

humanrebar 8y ago

> The purpose of unit tests is to lock down the project's source code once it's essentially complete Good unit tests (like doctests) also provide superior documentation and examples. Sure, worrying about documentation too soon is bad, but sometimes when doing early research, getting details clarified and written down is paramount and tests, maybe even unit tests, are a nice tool to have when you want it.

th456789123 8y ago

unit tests is not only to catch regression and it does not lock down anything. I found that doing TDD actually helps you design better, it naturally force you to write easily testable code which translate into more readable and maintainable code. Also well written UT prove very useful as living documentation. When you put the same care in them as you do in production code, you can very easily add features not needing to reverse-engineer the all thing. UT is one of thing things that may seem counter-intuitive but if you stick to it for a little while you see lots of positive effects. A bit like forcing yourself to hold the neck properly when learning the guitar, it slows you down at 1st, hurts a little, but if you do it you'll play better.

mabbo 8y ago

> The purpose of unit tests is to lock down the project's source code I disagree slightly. If you're writing unit tests such that the code itself can't change without breaking the tests, perhaps the tests have been too tightly coupled. The end goal of unit tests is to secure the expected functionality of the unit of code such that the tests will fail if the functionality stops working. But if changes to the code (refactoring, changing the ordering of certain lines, etc) causes the tests to break, then what was tested wasn't the functionality but the implementation. You might as well write a test that loads each source file as a string and verifies it equals what you've written. An example: my favorite set of tests I've written lately were for some classes to handle deduplication of a large set of records. One implementation wrote to disk (for data sets too large to fit into memory) while the other did it purely in memory. Since their functionality was supposed to be identical, I wrote one set of tests that both ran on. The tests could not be specific to the code written because it had to work on two sets of code. If we wrote a third implementation, the same tests should work for it as well. (But what you said about integration tests: 100% agree)

SkyPuncher 8y ago

I agree. The project I'm working on hovers around 70% coverage. As far as I can tell, we'd likely loose productivity pushing for any better coverage. We get some bugs/issues that could be identified by better coverage, but most of our "bugs" stem from real world issues or differing views on functionality. Getting that right takes a ton of effort that simply isn't worth it right now.

dtech 8y ago
Yep, testing definitely has a hugely decreasing marginal value. I personally like the approach of writing a small amount of "obvious" tests to detect large functional breakage, and add a test for a bug once it's been identified aka a regression test. Imo it's maximum test value per effort. Of course it depends on how impactful a bug could be.

didibus 8y ago

Very well said, I'm in your camp as well. The major issue is no one ever learns about or talks about the "Unit" in unit tests. But a Unit is supposed to be a component, albeit a very small one, in OOP, that normally translates to a Class. Your unit is a given Class, assuming its API doesn't change, the unit tests should continue working. If you havn't settled on an API for that unit yet, the tests are going to be coupled to a quick moving target, and that, in my opinion, is a waste of time. That said, people don't really design their Classes as unit in a bigger set, with self-contained API and SOLID principles. That's the first mistake. And that then leads to tests which assert not a Unit, but some random piece of logic within a Unit, or against things that are not Units of their own, but already tightly coupled and leaky containers. Which ends up both with bad designed software that is inflexible and quick to rot, as well as poor tests which prevent you from making any meaningful design improvements to it, without all the tests breaking.

brlewis 8y ago

It's true that obsessing over 100% code coverage is a bad thing. It's false that the purpose of unit tests is to lock down the project's source code. Unit tests have multiple purposes, but that isn't one of them. The main purpose of unit tests is to allow developers to quickly reach confidence that code works, be it a code change or new code. There is no such thing as writing unit tests too early.

stcredzero 8y ago

The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. This is only true for development environments which have particular weaknesses. If your tools for code changes and refactoring easily cover changing your unit tests while you are refactoring, then you can use unit tests for rapid development. The use of Unit Tests in present day "agile" methods actually originates in an environment where this was the norm. (Extreme Programming, as developed at the Chrysler C3 project in VisualWorks Smalltalk, using the phenomenal Refactoring Browser.) Your statement above would be 180 degrees turned around in an environment like that. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful to your project. Reading this just makes me sigh. Not all environments have the same cost/benefit factors in all parts of the edit/test/debug cycle. Not all environments and languages have the same costs for doing similar development operations. As a matter of fact, these are generally very, very different, even across projects using the same language. Change those cost/benefit factors, and the pronouncements you are making become more valid or less valid, depending. Not understanding that those cost/benefit factors can change and change exactly what your best practices are -- this is one of the most pervasive big misunderstandings that fosters poor management of programming projects. (EDIT: I think a lot of the annoyance many people have with "agile" methods, stems from practices which work well in one environment migrated to another environment where they don't work well at all!)