4 ms·

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutatio

by g14i 6y ago

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.

whymarrh 6y ago
I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org
- g14i 6y ago
  whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well. For Python we use mutmut [1]. [1] - https://pypi.org/project/mutmut/ https://pypi.org/project/mutmut/
  - boxed 6y ago
    Mutmut author here. Great to hear you're using mutmut! I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.
- TeaVMFan 6y ago
  Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTesting.html https://frequal.com/java/HighQualityTestsWithPitMutationTest...
ay 6y ago
test coverage is calculated as % of the locations seen by profiler during the test runs, right ? This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program. Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.
- dalke 6y ago
  There are many types of test coverage measurement. I believe path coverage ("Has every possible route through a given part of the code been executed?") is your latter example. Quoting https://en.wikipedia.org/wiki/Code_coverage https://en.wikipedia.org/wiki/Code_coverage which also says: > Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."

by g14i 6y ago

whymarrh 6y ago

I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org

g14i 6y ago
whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well. For Python we use mutmut [1]. [1] - https://pypi.org/project/mutmut/ https://pypi.org/project/mutmut/
- boxed 6y ago
  Mutmut author here. Great to hear you're using mutmut! I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.
TeaVMFan 6y ago
Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTesting.html https://frequal.com/java/HighQualityTestsWithPitMutationTest...

ay 6y ago

test coverage is calculated as % of the locations seen by profiler during the test runs, right ? This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program. Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.

dalke 6y ago
There are many types of test coverage measurement. I believe path coverage ("Has every possible route through a given part of the code been executed?") is your latter example. Quoting https://en.wikipedia.org/wiki/Code_coverage https://en.wikipedia.org/wiki/Code_coverage which also says: > Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."