7 ms·

It's important to note that having high test coverage doesn't make code good. Unit tests will actually make bad code even worse because it will be even more dif

It's important to note that having high test coverage doesn't make code good. Unit tests will actually make bad code even worse because it will be even more difficult to change the underlying logic (because the tests lock all the poor implementation details into place). Tests have nothing to do with code quality. All they do is verify that the code works. I would argue that the simpler and therefore the better your code is, the less you need to rely on tests to verify that it works. Fewer edge cases means fewer tests. I'm a big fan of integration tests though because they lock down the code based on high level features and not based on implementation details. If you ever have to rewrite a decent portion of a system (e.g. due to changing business requirements) it is deeply satisfying if your integration tests are still passing afterwards (e.g. with only minor changes to the test logic to account for the functionality changes).

humanrebar 7y ago
Most of this is a series of false choices. In fact - tests can help show code quality improvements do not break anything - you can have integration tests and unit tests at the same time; in fact, it is more of a spectrum than two rigid categories - it's often possible to have simple code and test it Generally speaking the more specific the question, the less controversial the choices are. It's typically not all that interesting to argue about how to test a particular algorithm, data structure, or service. The hard part in all of this, from an engineering perspective, is just talking to folks, promoting good teamwork, actually showing the value of less obvious things (a passing test suite), and knowing what to do when technology choices become toxic to the product or team.
- barrkel 7y ago
  Most interesting refactorings change the boundaries and count of abstractions, which usually does break unit tests. Unit tests are great at the leaves of the call graph, and things which are almost leaves because their dependencies aren't at any real risk of change. The further into the stack you go, the more brittle they get.
  - humanrebar 7y ago
    The point is that I use all kinds of tests all the time. It works fine. You aren't required to join a tribe and debate about abstract test styles. Look at the current problem and come up with good answers to the questions: - How do we know it works? - How will we know it still works in a year? ...you don't always need the best answers, even. Most projects should start with honest answers and work from there.
geezerjay 7y ago
> Unit tests will actually make bad code even worse because it will be even more difficult to change the underlying logic (because the tests lock all the poor implementation details into place). This statement is patently false, unless for some reason a project includes unit tests themselves as the production code, which would be highly unusual. At most, unit tests must be refactored along with the code, but that's the standard operating procedure.
- rement 7y ago
  I actually just finished a ticket related to this. It took me significantly longer than necessary because I also had to go through all the poorly written tests.
- cryptica 7y ago
  But while you're building a new system/subsystem, it doesn't make sense to write unit tests for units of code which have a high likelihood of being deleted 1 month from now due to evolving project requirements. It's like if you were building a smartphone; it wouldn't make sense to screw all the internal components into place one by one unless you were sure that all the components would end up fitting perfectly together inside the case. While building the prototype, you may decide to move some components around, downsize some components, trim some wires and remove some low-priority components to make room for others. In this case, unit tests are like screws.
  - gnator 7y ago
    The problem is prototypes end up being production code in the real world. Writing tests is about managing risk. You should write some basal level of unit test as you go to validate your logic as you go. That basal level is determined by the team or individual tolerance of risk
    - v-erne 7y ago
      Who said anything about production ? Haven't You seen requirements changing even before first prototype is ready? I had a meeting literally today where client's CFO and head accountant throw out my team's week of work because they forgot about key requirements (and this have happened third time this year)
benjiweber 7y ago
I see this opinion a lot from people who haven't seen tests and code written by people experienced with TDD. The tests should not end up that coupled to the code. The implementation structure and the test structure end up somewhat different when refactoring every time the tests are green. When listening to the feedback from the tests & code. With the skills to spot the refactoring opportunities. Oftentimes people seem to equate unit testing with a 1:1 correspondence of test and implementation with high coupling between the two. These sort of tests resist refactoring, rather than enabling it. With good tests you can pivot the implementation and tests independently. Recommend https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM and https://vimeo.com/83960706 https://vimeo.com/83960706 on TDD
- ishjoh 7y ago
  In my experience, your statement is true when writing library code or tests that don't need to mock lots of objects. Unfortunately, Unit testing becomes highly coupled when testing classes in the standard web architecture. A service class you're testing can depend on other service classes, a DAO, and potentially other web services, so now you're left mocking all those other classes if you want to create a Unit test instead of an integration test. Since the external dependencies have been mocked out, now the Unit test is higly coupled to the implementation and is a PITA to change the implmentation of the test or the code implementation. I suspect that's why OP prefers integration testing, as it helps keep the test less coupled from the implementation.
  - Boxxed 7y ago
    In my experience, if your tests require lots of mocks then that's a sign that IO is coupled too tightly to application logic. Refactoring your code so this isn't the case isn't always obvious, but it's a breath of fresh air and really cleans up the interfaces.
    - BeetleB 7y ago
      This is my experience as well. I learned the lesson the one time I was allowed to write unit tests at work. It was on an existing code base without tests. I had to significantly refactor code to make it testable, and one of the lessons I learned from the experience is to isolate I/O from the main business logic that I'm testing. In the pre-test code, the functions were littered with PrintConsole statements that would take a string and a warning level (the Console was an object that was responsible for printing strings on a HW console). I made sure my main business logic was never aware of the Console object. I made an intermediate/interface class that handled all I/O, and mocked that class. Instead, the function now had LogMessage, LogWarning, LogError functions of the interface class that took a string. The function had no idea where these messages could go - it could go to the console, it could be logged to a file, it could be sent as a text message. It didn't care. Now when we needed to make changes to how things were printed, none of our business logic functions, nor their tests, were impacted. In this case at least, attempting to unit test led to less coupled code.
jacques_chester 7y ago
> It's important to note that having high test coverage doesn't make code good. Sure, but low test coverage doesn't make it good either. Coverage is a metric and like any metric, it (1) needs to be assessed with judgement and (2) becomes useless when it's used as a target rather than a measurement. > Tests have nothing to do with code quality. All they do is verify that the code works. Well to start, Fowler notes a distinction between external and internal quality. External quality is "does it work from the end user perspective?", which can be verified by tests -- you note integration tests in this role (acceptance tests, feature tests, user tests, behaviour tests, whatever you choose to call them). In the external quality case, verifying that the code works is a large fraction of quality. Your argument, I think, is that internal quality is unaffected by testing. I don't agree: in my experience the needs of simple testing create constant design pressure on the production code, most of which makes it easier to create future changes. Though as noted at the top of the thread: expertise still matters. Writing better tests and better production code are skills.
jasode 7y ago
>Tests have nothing to do with code quality. I didn't downvote your comment but I vehemently disagree. Mission-critical code such as NASA flight guidance, avionics, and low-level libraries like SQLite depend on a suite of tests to maintain software quality. (I wrote a previous comment on this.[0]) We also want the new software that commands self-driving cars to have thousands of tests that cover as many scenarios as possible. I don't have inside knowledge of either Waymo or Tesla but it seems like common sense to assume those software programmers rely on a massive suite of unit tests to stress test their cars' decision algorithms. One can't write software with that level complexity that has life-&-death consequences without relying on numerous tests at all layers of the stack. Yes, the cars will still have bugs and will sometimes make the wrong decision but their software would be worse without the tests. High quality software relies on both lower-level unit tests and higher-level integration tests. Or put another way, both "black box" and "white box" testing strategies are used. [0] https://news.ycombinator.com/item?id=15592392 https://news.ycombinator.com/item?id=15592392
- commandlinefan 7y ago
  I think what he means is that just because you have tests (and even if you have high code coverage) doesn't meant that your code is high quality. They're correlated, but I've actually seen code with high test coverage... whose tests never made any assertions.
  - jasode 7y ago
    >tests [...] doesn't meant that your code is high quality. They're correlated, Yes, if they're correlated, that contradicts the absolutist statement of "Tests have nothing to do with code quality." Trying to improve code correctness is directly affecting code quality.
- oldmanhorton 7y ago
  Isn't this disagreement basically the same point made by Martin about different kinds of quality? SQLites tests don't say the code is architected well and reusable and modular and blah blah blah, it says that it works. When people talk about the quality of NASA code or SQLite, that feels more like external quality than internal quality.
Boxxed 7y ago
> I'm a big fan of integration tests though because they lock down the code based on high level features and not based on implementation details. I've found this to be a dangerous mindset. Integration tests are great, but they need a solid foundation of unit tests. Integration tests are slow, difficult to root-cause, complex to write and maintain, and also generally don't test all the various corners of the system. Testing is a pyramid, with unit tests at the bottom and integration tests somewhere in the middle. If your unit tests are based in implementation details, as you say, then that's probably a sign that a refactoring is in order (would love to be less blunt but it's tough with the absence of more details).
- deleted 7y ago
  [deleted]
msluyter 7y ago
Re: your last point, I recently rewrote parts of a billing system full of hairy logic and edge cases (and bugs). The initial MVP consisted of exactly replicating the existing invoicing logic. Due to the general complexity of the problem domain, I found myself rethinking and rewriting large portions of the system as I grew more familiar with the (undocumented, naturally) business requirements. In some cases I'd throw out entire modules and associated unit tests. After a while, I started relying more on integration tests which simply compared generated invoices between the two systems (and/or against golden files.) Having these made it extremely easy to refactor large portions of the system quickly without needing to refactor unit tests. (I still wrote unit tests, just less of them, more focused on the stabler parts of the system.) This has loosened the grip of the "every function must have a unit test" mantra in my mind, which... I dunno, somewhere along the way sort of became simply assumed. Some caveats to note, however. A) The code had minimal external dependencies (postgres). B) The integration tests ran very quickly against a local postgres database, only slightly slower than unit tests might, providing a quick dev feedback loop. C) While internally, the system was rather complex, the output was not. It was a simple CSV file that's trivial to parse/compare. Thus, I wouldn't overgeneralize from the above. In cases where there are lots of external dependencies, integration tests are slow, or where evaluating the test results is more tricky (ie, you need Selenium or whatnot), this approach wouldn't be as feasible.
Buttons840 7y ago
> Unit tests will actually make bad code even worse because it will be even more difficult to change the underlying logic Objectively false, if not having tests is better than having tests then delete the tests. Instant improvement. This fact leads to the conclusion that the value of having tests is greater than or equal to the value of not having tests, in all cases.
- discreteevent 7y ago
  Yes I have instantly improved code by removing micro level solpisistic tests that were tightly coupled to the implementation. These tests made it much slower to improve the quality of the code and had zero benifits because they only tested that the code did what it did and not what it is supposed to do.
- TheCoelacanth 7y ago
  You are missing the effect of loss-aversion and the cognitive bias towards keeping a test even if it adds negative value. Once a test has been added, it will tend to stick around even if it is worse than no test.
  - Buttons840 7y ago
    Good point, and I agree. Sunk cost fallacy and all that. I wouldn't consider those "objective", but I don't think it's worth arguing over the definition of that word when I think we otherwise agree. I also would agree that sometimes time has been wasted creating too many tests. Perhaps that time could have been spent to greater effect. I also think that even if, in retrospec, a test is very tightly coupled and specific to one implementation, that test still might have revealed bugs and may have helped the original author. If that test is now a burden, throw it away.
- Scarblac 7y ago
  You appear to be trying some sort of reduction ad absurdum, but in many cases work on some change to the software starts with deleting all the related tests because they're going to be irrelevant and changing them isn't worth the extra work. That that deletion is necessary means it apparently did make the code a bit worse. Also all the time they were in while the code wasn't being changed, they made running the application's tests slower.
Rooster61 7y ago
> Tests have nothing to do with code quality. All they do is verify that the code works. I would argue that the simpler and therefore the better your code is, the less you need to rely on tests to verify that it works. Fewer edge cases means fewer tests. While I won't argue that tests verify that the code works, the assertion that tests have nothing to do with code quality based on that premise is incorrect, and here's why. Some of the main types of poorly written code are 1) brittle code, which breaks easily when things are changed, such as dependency changes or changes in I/O, and 2) unreadable code, which decreases accurate understanding of what the code does and causes incorrect assumptions to be made, which yields bugs. Unit tests, over time, raise the alarm to these types of code smells. While a test might not yield much info for a short time after it is written, when the code ages and has to stand up to the test of changing code/environment around it, well written tests WILL highlight parts of the code that can be considered poorly written due to the two criteria above.

humanrebar 7y ago

Most of this is a series of false choices. In fact - tests can help show code quality improvements do not break anything - you can have integration tests and unit tests at the same time; in fact, it is more of a spectrum than two rigid categories - it's often possible to have simple code and test it Generally speaking the more specific the question, the less controversial the choices are. It's typically not all that interesting to argue about how to test a particular algorithm, data structure, or service. The hard part in all of this, from an engineering perspective, is just talking to folks, promoting good teamwork, actually showing the value of less obvious things (a passing test suite), and knowing what to do when technology choices become toxic to the product or team.

barrkel 7y ago
Most interesting refactorings change the boundaries and count of abstractions, which usually does break unit tests. Unit tests are great at the leaves of the call graph, and things which are almost leaves because their dependencies aren't at any real risk of change. The further into the stack you go, the more brittle they get.
- humanrebar 7y ago
  The point is that I use all kinds of tests all the time. It works fine. You aren't required to join a tribe and debate about abstract test styles. Look at the current problem and come up with good answers to the questions: - How do we know it works? - How will we know it still works in a year? ...you don't always need the best answers, even. Most projects should start with honest answers and work from there.

geezerjay 7y ago

> Unit tests will actually make bad code even worse because it will be even more difficult to change the underlying logic (because the tests lock all the poor implementation details into place). This statement is patently false, unless for some reason a project includes unit tests themselves as the production code, which would be highly unusual. At most, unit tests must be refactored along with the code, but that's the standard operating procedure.

rement 7y ago
I actually just finished a ticket related to this. It took me significantly longer than necessary because I also had to go through all the poorly written tests.
cryptica 7y ago
But while you're building a new system/subsystem, it doesn't make sense to write unit tests for units of code which have a high likelihood of being deleted 1 month from now due to evolving project requirements. It's like if you were building a smartphone; it wouldn't make sense to screw all the internal components into place one by one unless you were sure that all the components would end up fitting perfectly together inside the case. While building the prototype, you may decide to move some components around, downsize some components, trim some wires and remove some low-priority components to make room for others. In this case, unit tests are like screws.
- gnator 7y ago
  The problem is prototypes end up being production code in the real world. Writing tests is about managing risk. You should write some basal level of unit test as you go to validate your logic as you go. That basal level is determined by the team or individual tolerance of risk
  - v-erne 7y ago
    Who said anything about production ? Haven't You seen requirements changing even before first prototype is ready? I had a meeting literally today where client's CFO and head accountant throw out my team's week of work because they forgot about key requirements (and this have happened third time this year)

benjiweber 7y ago

I see this opinion a lot from people who haven't seen tests and code written by people experienced with TDD. The tests should not end up that coupled to the code. The implementation structure and the test structure end up somewhat different when refactoring every time the tests are green. When listening to the feedback from the tests & code. With the skills to spot the refactoring opportunities. Oftentimes people seem to equate unit testing with a 1:1 correspondence of test and implementation with high coupling between the two. These sort of tests resist refactoring, rather than enabling it. With good tests you can pivot the implementation and tests independently. Recommend https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM and https://vimeo.com/83960706 https://vimeo.com/83960706 on TDD

ishjoh 7y ago
In my experience, your statement is true when writing library code or tests that don't need to mock lots of objects. Unfortunately, Unit testing becomes highly coupled when testing classes in the standard web architecture. A service class you're testing can depend on other service classes, a DAO, and potentially other web services, so now you're left mocking all those other classes if you want to create a Unit test instead of an integration test. Since the external dependencies have been mocked out, now the Unit test is higly coupled to the implementation and is a PITA to change the implmentation of the test or the code implementation. I suspect that's why OP prefers integration testing, as it helps keep the test less coupled from the implementation.
- Boxxed 7y ago
  In my experience, if your tests require lots of mocks then that's a sign that IO is coupled too tightly to application logic. Refactoring your code so this isn't the case isn't always obvious, but it's a breath of fresh air and really cleans up the interfaces.
  - BeetleB 7y ago
    This is my experience as well. I learned the lesson the one time I was allowed to write unit tests at work. It was on an existing code base without tests. I had to significantly refactor code to make it testable, and one of the lessons I learned from the experience is to isolate I/O from the main business logic that I'm testing. In the pre-test code, the functions were littered with PrintConsole statements that would take a string and a warning level (the Console was an object that was responsible for printing strings on a HW console). I made sure my main business logic was never aware of the Console object. I made an intermediate/interface class that handled all I/O, and mocked that class. Instead, the function now had LogMessage, LogWarning, LogError functions of the interface class that took a string. The function had no idea where these messages could go - it could go to the console, it could be logged to a file, it could be sent as a text message. It didn't care. Now when we needed to make changes to how things were printed, none of our business logic functions, nor their tests, were impacted. In this case at least, attempting to unit test led to less coupled code.

jacques_chester 7y ago

> It's important to note that having high test coverage doesn't make code good. Sure, but low test coverage doesn't make it good either. Coverage is a metric and like any metric, it (1) needs to be assessed with judgement and (2) becomes useless when it's used as a target rather than a measurement. > Tests have nothing to do with code quality. All they do is verify that the code works. Well to start, Fowler notes a distinction between external and internal quality. External quality is "does it work from the end user perspective?", which can be verified by tests -- you note integration tests in this role (acceptance tests, feature tests, user tests, behaviour tests, whatever you choose to call them). In the external quality case, verifying that the code works is a large fraction of quality. Your argument, I think, is that internal quality is unaffected by testing. I don't agree: in my experience the needs of simple testing create constant design pressure on the production code, most of which makes it easier to create future changes. Though as noted at the top of the thread: expertise still matters. Writing better tests and better production code are skills.

jasode 7y ago

>Tests have nothing to do with code quality. I didn't downvote your comment but I vehemently disagree. Mission-critical code such as NASA flight guidance, avionics, and low-level libraries like SQLite depend on a suite of tests to maintain software quality. (I wrote a previous comment on this.[0]) We also want the new software that commands self-driving cars to have thousands of tests that cover as many scenarios as possible. I don't have inside knowledge of either Waymo or Tesla but it seems like common sense to assume those software programmers rely on a massive suite of unit tests to stress test their cars' decision algorithms. One can't write software with that level complexity that has life-&-death consequences without relying on numerous tests at all layers of the stack. Yes, the cars will still have bugs and will sometimes make the wrong decision but their software would be worse without the tests. High quality software relies on both lower-level unit tests and higher-level integration tests. Or put another way, both "black box" and "white box" testing strategies are used. [0] https://news.ycombinator.com/item?id=15592392 https://news.ycombinator.com/item?id=15592392

commandlinefan 7y ago
I think what he means is that just because you have tests (and even if you have high code coverage) doesn't meant that your code is high quality. They're correlated, but I've actually seen code with high test coverage... whose tests never made any assertions.
- jasode 7y ago
  >tests [...] doesn't meant that your code is high quality. They're correlated, Yes, if they're correlated, that contradicts the absolutist statement of "Tests have nothing to do with code quality." Trying to improve code correctness is directly affecting code quality.
oldmanhorton 7y ago
Isn't this disagreement basically the same point made by Martin about different kinds of quality? SQLites tests don't say the code is architected well and reusable and modular and blah blah blah, it says that it works. When people talk about the quality of NASA code or SQLite, that feels more like external quality than internal quality.

Boxxed 7y ago

> I'm a big fan of integration tests though because they lock down the code based on high level features and not based on implementation details. I've found this to be a dangerous mindset. Integration tests are great, but they need a solid foundation of unit tests. Integration tests are slow, difficult to root-cause, complex to write and maintain, and also generally don't test all the various corners of the system. Testing is a pyramid, with unit tests at the bottom and integration tests somewhere in the middle. If your unit tests are based in implementation details, as you say, then that's probably a sign that a refactoring is in order (would love to be less blunt but it's tough with the absence of more details).

deleted 7y ago
[deleted]

msluyter 7y ago

Re: your last point, I recently rewrote parts of a billing system full of hairy logic and edge cases (and bugs). The initial MVP consisted of exactly replicating the existing invoicing logic. Due to the general complexity of the problem domain, I found myself rethinking and rewriting large portions of the system as I grew more familiar with the (undocumented, naturally) business requirements. In some cases I'd throw out entire modules and associated unit tests. After a while, I started relying more on integration tests which simply compared generated invoices between the two systems (and/or against golden files.) Having these made it extremely easy to refactor large portions of the system quickly without needing to refactor unit tests. (I still wrote unit tests, just less of them, more focused on the stabler parts of the system.) This has loosened the grip of the "every function must have a unit test" mantra in my mind, which... I dunno, somewhere along the way sort of became simply assumed. Some caveats to note, however. A) The code had minimal external dependencies (postgres). B) The integration tests ran very quickly against a local postgres database, only slightly slower than unit tests might, providing a quick dev feedback loop. C) While internally, the system was rather complex, the output was not. It was a simple CSV file that's trivial to parse/compare. Thus, I wouldn't overgeneralize from the above. In cases where there are lots of external dependencies, integration tests are slow, or where evaluating the test results is more tricky (ie, you need Selenium or whatnot), this approach wouldn't be as feasible.

Buttons840 7y ago

> Unit tests will actually make bad code even worse because it will be even more difficult to change the underlying logic Objectively false, if not having tests is better than having tests then delete the tests. Instant improvement. This fact leads to the conclusion that the value of having tests is greater than or equal to the value of not having tests, in all cases.

discreteevent 7y ago
Yes I have instantly improved code by removing micro level solpisistic tests that were tightly coupled to the implementation. These tests made it much slower to improve the quality of the code and had zero benifits because they only tested that the code did what it did and not what it is supposed to do.
TheCoelacanth 7y ago
You are missing the effect of loss-aversion and the cognitive bias towards keeping a test even if it adds negative value. Once a test has been added, it will tend to stick around even if it is worse than no test.
- Buttons840 7y ago
  Good point, and I agree. Sunk cost fallacy and all that. I wouldn't consider those "objective", but I don't think it's worth arguing over the definition of that word when I think we otherwise agree. I also would agree that sometimes time has been wasted creating too many tests. Perhaps that time could have been spent to greater effect. I also think that even if, in retrospec, a test is very tightly coupled and specific to one implementation, that test still might have revealed bugs and may have helped the original author. If that test is now a burden, throw it away.
Scarblac 7y ago
You appear to be trying some sort of reduction ad absurdum, but in many cases work on some change to the software starts with deleting all the related tests because they're going to be irrelevant and changing them isn't worth the extra work. That that deletion is necessary means it apparently did make the code a bit worse. Also all the time they were in while the code wasn't being changed, they made running the application's tests slower.

Rooster61 7y ago

> Tests have nothing to do with code quality. All they do is verify that the code works. I would argue that the simpler and therefore the better your code is, the less you need to rely on tests to verify that it works. Fewer edge cases means fewer tests. While I won't argue that tests verify that the code works, the assertion that tests have nothing to do with code quality based on that premise is incorrect, and here's why. Some of the main types of poorly written code are 1) brittle code, which breaks easily when things are changed, such as dependency changes or changes in I/O, and 2) unreadable code, which decreases accurate understanding of what the code does and causes incorrect assumptions to be made, which yields bugs. Unit tests, over time, raise the alarm to these types of code smells. While a test might not yield much info for a short time after it is written, when the code ages and has to stand up to the test of changing code/environment around it, well written tests WILL highlight parts of the code that can be considered poorly written due to the two criteria above.