4 ms·

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There a

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

gamegoblin 12y ago
This is why I like using a randomized framework such as QuickCheck (or its derivatives in non-Haskell languages, such as Java) along with typical unit testing. It's often easier to write than tedious unit tests, and it can catch a lot of funny corner cases you don't even think to test. Of course, this still isn't a guarantee. But it does make me feel better about my code.
- sargun 12y ago
  I love QuickCheck. It might be a little hard to do for Tetris.
  - gamegoblin 12y ago
    Definitely true, but with enough abstraction to small, composable, pure functions, you can get pretty far.
lucaspiller 12y ago
One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This ends up causing trouble if you ever want to change the implementation as you end up having to throw away half the tests, which means a lot of effort you spent to get 100% code coverage is now gone.
- canadev 12y ago
  I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.
  - aikah 12y ago
    You cant have it both ways in my opinion. High test coverage == testing every possible paths == looking at implementation details. If you are testing an algorithm(and games are full of these)you want it to be a 100% accurate therefore you dont have much choice.
    - canadev 12y ago
      This is sort of the conclusion I was coming to. I am glad to hear someone else express it :)
    - alkonaut 12y ago
      Tests should be based on the specification. If I want to change some internal implementation detail I should only have to verify that the current tests pass. If a e.g game contains a sorting in some place in the renderer, I can replace the quicksort with a mergesort as long as the renderer interface is still testing ok. The new sort algorithm may have new special case paths (even number of items vs odd for example) but it's not a concern of the renderer public interface. I may however have introduced a bug with an odd number of items here and the old code was 100% covered and now it isn't. So there is a potential problem and the 99% has actually helped spot it. If the sorting is a private implementation detail of the renderer then there is no other place to test it than to add a new test to the renderer component only because the sorting algo requires it for a code path. This is BAD. The proper action here is NOT to add tests to the renderer component to test the sorting code path, but instead to make the sorting visible and testable in isolation via its own public interface. So one of the positive things about requiring coverage is that if you do it right, it will lead to smaller and more decoupled modules of code. The bad thing is that if you do it wrong you will have your God classes and a bunch of tests coupled tightly to them.
rrrrob 12y ago
For example, 100% coverage won't help with the classic Windows Tetris bug where the score overflows at 32768.
- xxs 12y ago
  Yes, overflows and data races are bane for the coverage tests - just a false sense of security.
ygra 12y ago
»Our customers tend to be makers of aircraft or car parts. Both businesses have strict safety standards which involve coverage testing, and our tools help you produce the relevant reports for certification, like DO-178B for the aviation industry.« I guess in both industries the value of more testing cannot be understated.
- marcosdumay 12y ago
  I guess you meant overstated... And yes, it can. Are you trying to state their value compared to what? A good schema for task division, with encapsulation and whatchdogs? Proofs of correctness? Proofs of halting? Testing is much less valuable than any of those.
alkonaut 12y ago
I have yet to see a compelling argument that 100% test coverage would be "better" than 50% without knowing what the quality of the tests are In those cases. I'm always suspicious of code bases that claim 100% test coverage because they usually have it because someone has decided it must have, which invariably means it has tests that exist only to execute some corner case lines of code. To have good test coverage you should both test all possible inputs and have proper asserts for those inputs. Testing is hard, covering lines of code isn't. To put it another way: in a test the hard bit is the assert, not the call into the tested code. And coverage only reports on the former.
dack 12y ago
One thing I like to say when people bring up test coverage is that code being "covered" only says it was run, not that it was correct, so it's only a weak statement of quality. However, code that is not covered is completely unknown, so there is a bigger chance of bugs there. Obviously though, if that code is trivial, it may not be worth the maintenance overhead of another test for just that. So yeah, when a team claims 100% code coverage, usually that is just a signal that they care about testing and the quality of the code, therefore it tends to be less buggy. Not necessarily because 100% coverage itself made it so. Really I only use a code coverage tool to check for important places that aren't covered at all, AFTER I have tried to think of the proper behavior/spec of the code from an outside perspective. It's like a secondary check after you think you are already done. That keeps you focused on what correct input and output are, and then patching up the little areas that you missed with a tool.

gamegoblin 12y ago

This is why I like using a randomized framework such as QuickCheck (or its derivatives in non-Haskell languages, such as Java) along with typical unit testing. It's often easier to write than tedious unit tests, and it can catch a lot of funny corner cases you don't even think to test. Of course, this still isn't a guarantee. But it does make me feel better about my code.

sargun 12y ago
I love QuickCheck. It might be a little hard to do for Tetris.
- gamegoblin 12y ago
  Definitely true, but with enough abstraction to small, composable, pure functions, you can get pretty far.

lucaspiller 12y ago

One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This ends up causing trouble if you ever want to change the implementation as you end up having to throw away half the tests, which means a lot of effort you spent to get 100% code coverage is now gone.

canadev 12y ago
I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.
- aikah 12y ago
  You cant have it both ways in my opinion. High test coverage == testing every possible paths == looking at implementation details. If you are testing an algorithm(and games are full of these)you want it to be a 100% accurate therefore you dont have much choice.
  - canadev 12y ago
    This is sort of the conclusion I was coming to. I am glad to hear someone else express it :)
  - alkonaut 12y ago
    Tests should be based on the specification. If I want to change some internal implementation detail I should only have to verify that the current tests pass. If a e.g game contains a sorting in some place in the renderer, I can replace the quicksort with a mergesort as long as the renderer interface is still testing ok. The new sort algorithm may have new special case paths (even number of items vs odd for example) but it's not a concern of the renderer public interface. I may however have introduced a bug with an odd number of items here and the old code was 100% covered and now it isn't. So there is a potential problem and the 99% has actually helped spot it. If the sorting is a private implementation detail of the renderer then there is no other place to test it than to add a new test to the renderer component only because the sorting algo requires it for a code path. This is BAD. The proper action here is NOT to add tests to the renderer component to test the sorting code path, but instead to make the sorting visible and testable in isolation via its own public interface. So one of the positive things about requiring coverage is that if you do it right, it will lead to smaller and more decoupled modules of code. The bad thing is that if you do it wrong you will have your God classes and a bunch of tests coupled tightly to them.

rrrrob 12y ago

For example, 100% coverage won't help with the classic Windows Tetris bug where the score overflows at 32768.

xxs 12y ago
Yes, overflows and data races are bane for the coverage tests - just a false sense of security.

ygra 12y ago

»Our customers tend to be makers of aircraft or car parts. Both businesses have strict safety standards which involve coverage testing, and our tools help you produce the relevant reports for certification, like DO-178B for the aviation industry.« I guess in both industries the value of more testing cannot be understated.

marcosdumay 12y ago
I guess you meant overstated... And yes, it can. Are you trying to state their value compared to what? A good schema for task division, with encapsulation and whatchdogs? Proofs of correctness? Proofs of halting? Testing is much less valuable than any of those.

alkonaut 12y ago

I have yet to see a compelling argument that 100% test coverage would be "better" than 50% without knowing what the quality of the tests are In those cases. I'm always suspicious of code bases that claim 100% test coverage because they usually have it because someone has decided it must have, which invariably means it has tests that exist only to execute some corner case lines of code. To have good test coverage you should both test all possible inputs and have proper asserts for those inputs. Testing is hard, covering lines of code isn't. To put it another way: in a test the hard bit is the assert, not the call into the tested code. And coverage only reports on the former.

dack 12y ago

One thing I like to say when people bring up test coverage is that code being "covered" only says it was run, not that it was correct, so it's only a weak statement of quality. However, code that is not covered is completely unknown, so there is a bigger chance of bugs there. Obviously though, if that code is trivial, it may not be worth the maintenance overhead of another test for just that. So yeah, when a team claims 100% code coverage, usually that is just a signal that they care about testing and the quality of the code, therefore it tends to be less buggy. Not necessarily because 100% coverage itself made it so. Really I only use a code coverage tool to check for important places that aren't covered at all, AFTER I have tried to think of the proper behavior/spec of the code from an outside perspective. It's like a secondary check after you think you are already done. That keeps you focused on what correct input and output are, and then patching up the little areas that you missed with a tool.