4 ms·

I'm so sad that so few test frameworks/libraries offer mutation testing. (And its profiling cousin Causal Profiling.) Admittedly the some languages might make

I'm so sad that so few test frameworks/libraries offer mutation testing. (And its profiling cousin Causal Profiling.) Admittedly the some languages might make them very hard to do, but the payoff would be so, so sweet.

taeric 4y ago
My team at work took it out of our product. :( I couldn't make strong arguments that it was providing value, as it had turned into the slow test that always passed, by the time new folks joined. I do think it has a lot of merit, though.
- pfdietz 4y ago
  To be clear... mutation testing is a way to assess the power of a test suite, not a way to test software itself (in this it is analogous to using statement/branch coverage to assess a test suite). Do you mean the tests added to achieve high mutation coverage were not worth the time it took to run them? The mutation process itself is not something that would be part of CI. Doing all that mutation and generating the tests to kill non-equivalent mutants is very time consuming, I will admit.
  - taeric 4y ago
    On the code base in question, the full but with mutation tests was only 8 minutes. Not snappy, but hardly a disaster.
    - pfdietz 4y ago
      I'm guessing these tests were run constantly, in CI, not periodically (say, daily, or even weekly, or before release). SQLite has tests that take many hours that must pass before release. There are testing modes that are worth doing, and I think should be part of a development process, but shouldn't be run at every check-in. Long soaks of random input property-based testing is like this, for example.
      - taeric 4y ago
        We run a lot of tests constantly. So, yeah. That said, as soon as it is in the release pipeline, I really encourage the team to not directly watch it. If we don't trust our tests to allow a change to go out, we need to fix the tests. Not train members to watch the deployment.
        pfdietz 4y ago
        It's not a matter of watching a deployment, it's a matter of having separate testing modes used in appropriate ways. Rapid tests should catch most bugs in CI, but more laborious tests could catch other bugs that would escape CI. Personally, there's a free software effort where my contribution is running tens of millions of randomized property-based tests each day and registering bugs when failures occur. This supplements and complements the fast tests that all must pass before changes are merged to master.
        taeric 4y ago
        Apologies, I meant my point to be tangential. I agree with your general point. Our deployment pipelines go from build to prod though. With a general cadence on my current team of less than an hour. 6ish minutes to that is not a concern. Even when I was on teams where the general cadence of getting changes out all the way to prod was on the order of a week, we still had changes getting to our beta environments in less than an hour. And that was on a team that wanted mutation testing, so we had it. Edit: So my point was team mates that were worried about the 6 minutes added by mutation testing there, were the ones that would commit and then constantly refresh the page so that they could "approve" a change as it goes out. I pushed for all approvals to be automated, so there was nothing for them to watch.

taeric 4y ago

My team at work took it out of our product. :( I couldn't make strong arguments that it was providing value, as it had turned into the slow test that always passed, by the time new folks joined. I do think it has a lot of merit, though.

pfdietz 4y ago
To be clear... mutation testing is a way to assess the power of a test suite, not a way to test software itself (in this it is analogous to using statement/branch coverage to assess a test suite). Do you mean the tests added to achieve high mutation coverage were not worth the time it took to run them? The mutation process itself is not something that would be part of CI. Doing all that mutation and generating the tests to kill non-equivalent mutants is very time consuming, I will admit.
- taeric 4y ago
  On the code base in question, the full but with mutation tests was only 8 minutes. Not snappy, but hardly a disaster.
  - pfdietz 4y ago
    I'm guessing these tests were run constantly, in CI, not periodically (say, daily, or even weekly, or before release). SQLite has tests that take many hours that must pass before release. There are testing modes that are worth doing, and I think should be part of a development process, but shouldn't be run at every check-in. Long soaks of random input property-based testing is like this, for example.
    - taeric 4y ago
      We run a lot of tests constantly. So, yeah. That said, as soon as it is in the release pipeline, I really encourage the team to not directly watch it. If we don't trust our tests to allow a change to go out, we need to fix the tests. Not train members to watch the deployment.
      - pfdietz 4y ago
        It's not a matter of watching a deployment, it's a matter of having separate testing modes used in appropriate ways. Rapid tests should catch most bugs in CI, but more laborious tests could catch other bugs that would escape CI. Personally, there's a free software effort where my contribution is running tens of millions of randomized property-based tests each day and registering bugs when failures occur. This supplements and complements the fast tests that all must pass before changes are merged to master.
        taeric 4y ago
        Apologies, I meant my point to be tangential. I agree with your general point. Our deployment pipelines go from build to prod though. With a general cadence on my current team of less than an hour. 6ish minutes to that is not a concern. Even when I was on teams where the general cadence of getting changes out all the way to prod was on the order of a week, we still had changes getting to our beta environments in less than an hour. And that was on a team that wanted mutation testing, so we had it. Edit: So my point was team mates that were worried about the 6 minutes added by mutation testing there, were the ones that would commit and then constantly refresh the page so that they could "approve" a change as it goes out. I pushed for all approvals to be automated, so there was nothing for them to watch.