7 ms·

You don't want to test code. Coverage is mostly useless. You want to test functionalities: setup end-to-end tests. It will be hard, it will take time and it of

by arkh 7y ago

You don't want to test code. Coverage is mostly useless. You want to test functionalities: setup end-to-end tests. It will be hard, it will take time and it often is a solo endeavor for the first months. But when tested functionalities stop breaking for no reasons even when you refactor a lot of code, when "the project for which a dev put tests" starts being "the project we are confident to push to prod without breaking things" you should get some traction. And once you think all your functionalities are tested you can measure code coverage: what is not covered should be dead code you can remove.

sverhagen 7y ago
Pointing people only at end-to-end testing seems poor advice. There is value in a mix of all sorts of testing. I agree that testing functionality blows counting coverage by the line out of the water. But since a lot of code that implements algorithms, string manipulation, etc. can be tested just fine with unit tests, and since end-to-end tests are expensive, easily more brittle, require a lot more dependencies, it would be better to have a healthy mix of different types of tests.
- shkkmo 7y ago
  The comments was specifically about the use case of getting a large refactor merged. If you are simply adding code or making simple changes, then unit tests can have advantages. It is when you are making a large refactor that you need end-to-end testing to verify that nothing was broken.
adrianN 7y ago
Coverage is very useful in determining whether your tests actually do what you think they're doing. It's not very useful as a metric to be optimized. Writing good end-to-end tests that really cover the functionality as specified is very difficult, doing so blindly without tool support is doomed to fail. An even better tool than measuring coverage is setting up a mutation testing framework and adding tests until all mutations are caught.
- mmjaa 7y ago
  I agree, and I think there is an arrogance by those with the view that 'coverage testing is useless' which blinds them to the real purpose of code coverage testing: to tell you what you haven't tested yet. Coverage isn't supposed to tell you anything about the quality of the tests beyond what code simply has never been tested yet. And yet, people seem to think "code coverage testing" means something else entirely. The only purpose for paying attention to coverage metrics, as a manager, is to understand how much more work there is to be done on the tests - or, to determine code paths which never get executed and are therefore dead weight on the project.
  - arkh 7y ago
    > the real purpose of code coverage testing: to tell you what you haven't tested yet. I can get 100% code coverage with tests passing on: // Returns the result of A*B int mult(int A, int B) { return 4; } I won't know the functionality is not tested. Code coverage will just be an illusion of security.
    - mmjaa 7y ago
      Code coverage is not all you should be doing.
gmueckl 7y ago
Tests need to cover small units one at a time. I've been bitten too many times by tests that were too complex and attempted to test too many things at once. They invariably ended up doing nothing of value. Test setup becomes overly complex and prone to oversights. Writing another test for a variation of a problem that takes a slightly different code path becomes a chore. It also gets hard to spot that odd untested corner case hidden deep inside some complex logic. The proper way to think about this is to have unit tests that cover the nooks and crannies of the business logic and integration tests that make sure that interfaces are correct.
JohnBooty 7y ago
I can't get behind this. End to end tests are indeed important. But let's say you have an end to end test that covers A, B, C, D, E, and F. One day this test stops working. Well, what's the culprit? A? B? C? D? E? F? If you had more granular tests in addition to the end-to-end tests, you'd know. Now, I agree there's a possible downside here. Maybe there comes a time when C and D are no longer being used. It might not be obvious and C and D could linger in the code base, with corresponding dead weight in the test suite. This is not ideal. However, I would rather have this than a tough-to-debug test failure.
- arkh 7y ago
  > Well, what's the culprit? A? B? C? D? E? F? I honestly don't care. I know this merge is not touching production. I could have all of A, B, C, D, E and F test suites working individually if they break when put together they are all bad. The developer have tests so they know exactly how to reproduce the bug. Time for them to earn their salary.
- yabadabadoes 7y ago
  I see this over and over also in organizations I work in and even have to fight it in my own behavior.. If you run an end to end test automation and it fails and then you are afraid or otherwise hesitant of doing the process manually you are out of touch with the system and probably missing the forest for the trees when you work on subsystems. While I would like to have better end-to-end tests, I know that I will have to fight more problems of getting too far away from how the system works.
- andrewflnr 7y ago
  > Well, what's the culprit? A? B? C? D? E? F? Probably, the one you just changed. How often is it really going to be that hard to debug?
  - thfuran 7y ago
    I only changed L, which is nominally unrelated to all of those. You thought this was a clean, well-modularized codebase?

by arkh 7y ago

sverhagen 7y ago

Pointing people only at end-to-end testing seems poor advice. There is value in a mix of all sorts of testing. I agree that testing functionality blows counting coverage by the line out of the water. But since a lot of code that implements algorithms, string manipulation, etc. can be tested just fine with unit tests, and since end-to-end tests are expensive, easily more brittle, require a lot more dependencies, it would be better to have a healthy mix of different types of tests.

shkkmo 7y ago
The comments was specifically about the use case of getting a large refactor merged. If you are simply adding code or making simple changes, then unit tests can have advantages. It is when you are making a large refactor that you need end-to-end testing to verify that nothing was broken.

adrianN 7y ago

Coverage is very useful in determining whether your tests actually do what you think they're doing. It's not very useful as a metric to be optimized. Writing good end-to-end tests that really cover the functionality as specified is very difficult, doing so blindly without tool support is doomed to fail. An even better tool than measuring coverage is setting up a mutation testing framework and adding tests until all mutations are caught.

mmjaa 7y ago
I agree, and I think there is an arrogance by those with the view that 'coverage testing is useless' which blinds them to the real purpose of code coverage testing: to tell you what you haven't tested yet. Coverage isn't supposed to tell you anything about the quality of the tests beyond what code simply has never been tested yet. And yet, people seem to think "code coverage testing" means something else entirely. The only purpose for paying attention to coverage metrics, as a manager, is to understand how much more work there is to be done on the tests - or, to determine code paths which never get executed and are therefore dead weight on the project.
- arkh 7y ago
  > the real purpose of code coverage testing: to tell you what you haven't tested yet. I can get 100% code coverage with tests passing on: // Returns the result of A*B int mult(int A, int B) { return 4; } I won't know the functionality is not tested. Code coverage will just be an illusion of security.
  - mmjaa 7y ago
    Code coverage is not all you should be doing.

gmueckl 7y ago

Tests need to cover small units one at a time. I've been bitten too many times by tests that were too complex and attempted to test too many things at once. They invariably ended up doing nothing of value. Test setup becomes overly complex and prone to oversights. Writing another test for a variation of a problem that takes a slightly different code path becomes a chore. It also gets hard to spot that odd untested corner case hidden deep inside some complex logic. The proper way to think about this is to have unit tests that cover the nooks and crannies of the business logic and integration tests that make sure that interfaces are correct.

JohnBooty 7y ago

I can't get behind this. End to end tests are indeed important. But let's say you have an end to end test that covers A, B, C, D, E, and F. One day this test stops working. Well, what's the culprit? A? B? C? D? E? F? If you had more granular tests in addition to the end-to-end tests, you'd know. Now, I agree there's a possible downside here. Maybe there comes a time when C and D are no longer being used. It might not be obvious and C and D could linger in the code base, with corresponding dead weight in the test suite. This is not ideal. However, I would rather have this than a tough-to-debug test failure.

arkh 7y ago
> Well, what's the culprit? A? B? C? D? E? F? I honestly don't care. I know this merge is not touching production. I could have all of A, B, C, D, E and F test suites working individually if they break when put together they are all bad. The developer have tests so they know exactly how to reproduce the bug. Time for them to earn their salary.
yabadabadoes 7y ago
I see this over and over also in organizations I work in and even have to fight it in my own behavior.. If you run an end to end test automation and it fails and then you are afraid or otherwise hesitant of doing the process manually you are out of touch with the system and probably missing the forest for the trees when you work on subsystems. While I would like to have better end-to-end tests, I know that I will have to fight more problems of getting too far away from how the system works.
andrewflnr 7y ago
> Well, what's the culprit? A? B? C? D? E? F? Probably, the one you just changed. How often is it really going to be that hard to debug?
- thfuran 7y ago
  I only changed L, which is nominally unrelated to all of those. You thought this was a clean, well-modularized codebase?