4 ms·

For a great explanation of why you would want to use mutation testing, see https://pitest.org/ https://pitest.org/ (a similar tool but for Java). In short - it

For a great explanation of why you would want to use mutation testing, see https://pitest.org/ https://pitest.org/ (a similar tool but for Java). In short - it is to gauge the quality of your test suite, because ideally any mutation of your app should cause a test failure.

amelius 7y ago
Well, not any mutation. Deleting an assertion should not cause any problems. Replacing bubblesort by quicksort shouldn't either. Etc.
- sixstringtheory 7y ago
  Hmm, I’m thinking that removing an assertion would potentially allow your program to continue into an undefined state that the assertion was guarding. You’d also want to watch out for the particular implementation of sorting algos... if you swap out a stable quicksort for a nonstable bubblesort, your benchmark tests might still pass, but you could wind up with different orderings of “equivalent” items in your list before/after that could cause subtle issues. If you have a test that truly still passes regardless of any mutation, I’d have to wonder if it should exist at all.
- AlexDenisov 7y ago
  This is a valid concern, for sure. In fact, these are called Equivalent Mutants. We observed it several times, here are some examples. You flip the if/then branches and the code essentially does the same, switching from a fast to slow implementation of the same algorithm. Another example is mutating the code in a C++ destructor, which is not easy to test. With that being said, even despite the equivalent mutants, you can get a pretty good insight. Here is an example of finding from a proprietory real-time OS used in the space industry: https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d96183ed2 https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d... (this is a part of an email which I could safely post online).

4 ms·

For a great explanation of why you would want to use mutation testing, see https://pitest.org/ https://pitest.org/ (a similar tool but for Java). In short - it

by gkfasdfasdf 7y ago

amelius 7y ago
Well, not any mutation. Deleting an assertion should not cause any problems. Replacing bubblesort by quicksort shouldn't either. Etc.
- sixstringtheory 7y ago
  Hmm, I’m thinking that removing an assertion would potentially allow your program to continue into an undefined state that the assertion was guarding. You’d also want to watch out for the particular implementation of sorting algos... if you swap out a stable quicksort for a nonstable bubblesort, your benchmark tests might still pass, but you could wind up with different orderings of “equivalent” items in your list before/after that could cause subtle issues. If you have a test that truly still passes regardless of any mutation, I’d have to wonder if it should exist at all.
- AlexDenisov 7y ago
  This is a valid concern, for sure. In fact, these are called Equivalent Mutants. We observed it several times, here are some examples. You flip the if/then branches and the code essentially does the same, switching from a fast to slow implementation of the same algorithm. Another example is mutating the code in a C++ destructor, which is not easy to test. With that being said, even despite the equivalent mutants, you can get a pretty good insight. Here is an example of finding from a proprietory real-time OS used in the space industry: https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d96183ed2 https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d... (this is a part of an email which I could safely post online).