This is fun. Seems like you did some good analysis. I couldn't help but notice you run a "most common" test suite, implying that you have more tests that don't

This is fun. Seems like you did some good analysis. I couldn't help but notice you run a "most common" test suite, implying that you have more tests that don't get run for whatever reason (the change doesn't affect that code, it's too slow, whatever). We end up having to do something similar to a small degree, and it bugs me. What I would really like to see (and this might become a personal project at some point) is a way to optimize the tests at the step level. I believe this would accomplish several goals: - It would reduce redundancy, leading to faster execution time and easier maintenance - It would make the test scenarios easier to see and evaluate from a high level Inevitably when you write tests you end up covering scenario X, then months or years later when working to cover scenario Y you accidentally create redundant coverage for scenario X. This waste continues to build over time. I think this could be improved if I could break the tests up into chunks (steps) that could be composed to cover multiple scenarios. Then a tool could analyze those chunks to remove redundancy or even make suggestions for optimization. And it could outline and communicate coverage more clearly. If formatted properly it might serve not only as a regression test suite definition, but also as documentation of current behavior. (Think Cucumber, but in reverse.)