3 ms·

A problem with this is that sometimes nondeterminism is OK. For instance, changing the constant to another prime in the "classic" hashcode implementation (repe

A problem with this is that sometimes nondeterminism is OK. For instance, changing the constant to another prime in the "classic" hashcode implementation (repeatedly multiply by a prime and add the next field) will (probably) not trigger any (well-written) tests, and indeed generally won't be detrimental at all, but will be flagged by this sort of test.

the_af 11y ago
I'm not an expert on this topic, but I don't think those are the kinds of changes generated by mutation testing (or at least, not the only ones). A more interesting example of a mutation is changing a "<" to a ">" in a condition. If your tests don't detect this, you're in trouble...
- TheLoneWolfling 11y ago
  There are plenty of places where inverting a condition won't matter. Anything that just requires a consistent ordering, for one.
  - the_af 11y ago
    Your tests should be written in a way that they test there is a consistent ordering, if that's a relevant property for you. If they don't, you have a problem. If they do, I guess the mutation framework will merely point out your tests don't detect some changes; it doesn't necessarily mean they are bad tests, it's merely a "smell" that deserves further investigation. (By the way, I think you're overstating your case when you claim "there are plenty of places" where it doesn't matter. I think in the vast majority of cases, if you change a "<" to a ">", you introduce a bug!)
    - TheLoneWolfling 11y ago
      I agree that in the majority of cases it will cause bugs. But what I'm saying is that there are cases where it can legitimately not cause a bug. The problem I have is this: this testing framework explicitly states the following: > The quality of your tests can be gauged from the percentage of mutations killed So, you write your tests to test that there is a consistent ordering, it changes the "<" to a ">", finds that all tests still pass (because, as you said, you're only testing if there is a consistent ordering, because that's the only thing that matters in this case), and considers your tests lower-quality because of that. This is a flaw most mutation testing - and, for that matter, test driven design - suffers. For another flaw: there are many things that end up along the lines of a bunch of shortcut "fast paths" at the start, with a slow path at the end that does slow but exhaustive checking. For example, if you're checking that two objects are equal you may insert a check at the start to see if they are the same pointer, and if so return `true`. If you're doing division in software you may check if you're dividing by 1 instead of doing the computation. That sort of thing. Any mutation that introduces false negatives into the shortcuts (but no false positives) won't be flagged by mutation testing.
0hjc 11y ago
This is what it known an "equivalent mutation". Along with high computational cost, it's one of the things identified in academic research as preventing the widespread use of mutation testing. There is no general method for distinguishing an equivalent mutation from a normal surviving one except for getting a human to have a look at it. Having built pitest to address the concerns around computational cost, it was a pleasant surprise to find out that in practice equivalent mutations are not much of a problem. This isn't entirely by accident - the default set of mutation operations are carefully designed to make equivalent mutations unlikely (they don't/can't guarantee not to create them, but they make them as unlikely as possible). There's a trade off here. Pitest has a smaller set of operators than a lot of research focussed systems. A larger set of operators would catch more issues, but would also create a larger proportion of equivalent mutants (and also take longer to run). There are more operators you can enable you wish to change this balance - an operator that changes constants as you describe is one of them. I rarely encounter equivalent mutants using the default operators and I know of some rollouts of pitest where they break the build on anything less than 100% mutation coverage. I have no figures to back this up, but I strongly suspect the % of equivalent mutants will be highly dependent on coding style and the domain in which the code operates.
- TheLoneWolfling 11y ago
  > I have no figures to back this up, but I strongly suspect the % of equivalent mutants will be highly dependent on coding style and the domain in which the code operates. Agreed. In everything I've written, at least, I can identify multiple places where such equivalent mutants do exist, even with just the default operators. (In particular, hashcode methods - there are very few mutators that break the contract of a hashcode method, though removing entropy in most cases) But I can easily see that not being the case with other coding styles. Are the mutations done documented anywhere? I had to look through the source to see what's done.
  - 0hjc 11y ago
    The mutators are documented at http://pitest.org/quickstart/mutators/ http://pitest.org/quickstart/mutators/

the_af 11y ago

I'm not an expert on this topic, but I don't think those are the kinds of changes generated by mutation testing (or at least, not the only ones). A more interesting example of a mutation is changing a "<" to a ">" in a condition. If your tests don't detect this, you're in trouble...

TheLoneWolfling 11y ago
There are plenty of places where inverting a condition won't matter. Anything that just requires a consistent ordering, for one.
- the_af 11y ago
  Your tests should be written in a way that they test there is a consistent ordering, if that's a relevant property for you. If they don't, you have a problem. If they do, I guess the mutation framework will merely point out your tests don't detect some changes; it doesn't necessarily mean they are bad tests, it's merely a "smell" that deserves further investigation. (By the way, I think you're overstating your case when you claim "there are plenty of places" where it doesn't matter. I think in the vast majority of cases, if you change a "<" to a ">", you introduce a bug!)
  - TheLoneWolfling 11y ago
    I agree that in the majority of cases it will cause bugs. But what I'm saying is that there are cases where it can legitimately not cause a bug. The problem I have is this: this testing framework explicitly states the following: > The quality of your tests can be gauged from the percentage of mutations killed So, you write your tests to test that there is a consistent ordering, it changes the "<" to a ">", finds that all tests still pass (because, as you said, you're only testing if there is a consistent ordering, because that's the only thing that matters in this case), and considers your tests lower-quality because of that. This is a flaw most mutation testing - and, for that matter, test driven design - suffers. For another flaw: there are many things that end up along the lines of a bunch of shortcut "fast paths" at the start, with a slow path at the end that does slow but exhaustive checking. For example, if you're checking that two objects are equal you may insert a check at the start to see if they are the same pointer, and if so return `true`. If you're doing division in software you may check if you're dividing by 1 instead of doing the computation. That sort of thing. Any mutation that introduces false negatives into the shortcuts (but no false positives) won't be flagged by mutation testing.

0hjc 11y ago

This is what it known an "equivalent mutation". Along with high computational cost, it's one of the things identified in academic research as preventing the widespread use of mutation testing. There is no general method for distinguishing an equivalent mutation from a normal surviving one except for getting a human to have a look at it. Having built pitest to address the concerns around computational cost, it was a pleasant surprise to find out that in practice equivalent mutations are not much of a problem. This isn't entirely by accident - the default set of mutation operations are carefully designed to make equivalent mutations unlikely (they don't/can't guarantee not to create them, but they make them as unlikely as possible). There's a trade off here. Pitest has a smaller set of operators than a lot of research focussed systems. A larger set of operators would catch more issues, but would also create a larger proportion of equivalent mutants (and also take longer to run). There are more operators you can enable you wish to change this balance - an operator that changes constants as you describe is one of them. I rarely encounter equivalent mutants using the default operators and I know of some rollouts of pitest where they break the build on anything less than 100% mutation coverage. I have no figures to back this up, but I strongly suspect the % of equivalent mutants will be highly dependent on coding style and the domain in which the code operates.

TheLoneWolfling 11y ago
> I have no figures to back this up, but I strongly suspect the % of equivalent mutants will be highly dependent on coding style and the domain in which the code operates. Agreed. In everything I've written, at least, I can identify multiple places where such equivalent mutants do exist, even with just the default operators. (In particular, hashcode methods - there are very few mutators that break the contract of a hashcode method, though removing entropy in most cases) But I can easily see that not being the case with other coding styles. Are the mutations done documented anywhere? I had to look through the source to see what's done.
- 0hjc 11y ago
  The mutators are documented at http://pitest.org/quickstart/mutators/ http://pitest.org/quickstart/mutators/