6 ms·
I remember reading the original article that inspired it and thinking "huh, that's a cool idea." In practice I wonder if they'll ever really sort this out: >
by wdewind 11y ago
I remember reading the original article that inspired it and thinking "huh, that's a cool idea." In practice I wonder if they'll ever really sort this out:
> Test selection algorithm is not perfect yet (it may produce false-positives and false-negatives)
The problem is you're going to need to run the entire test suite before deploying anyway unless the algorithm is perfected. If you have to do this anyway, while it's helpful to have more confidence going into an hour long test build that it's going to pass, it's probably more helpful to just build an infrastructure on which the entire test suite runs more quickly. I've worked on sites with extremely large test suites and it was never an issue to run the tests in < 10 minutes because they could be run on an infrastructure that allowed tests to be run in parallel. I think that's the solution rather than a complex and flakey algorithm.
- __alexs 11y agoI implemented much the same thing for Python a few years ago. Accuracy on real programs is actually quite good but ultimately Python and Ruby don't really make it easy to get accurate results. This is mostly caused by (perhaps unsurprisingly) issues around global and otherwise shared state between tests that is often difficult to handle efficiently. In my prototype this caused effectively any change outside of a function body to end up disabling the optimisation entirely :(
- wdewind 11y agoWhy would the language being untyped have anything to do with whether or not there is global state? I think I'm missing something.
- stupidcar 11y agoIt seems like there could be more value than just speeding up test runs though. Imagine a system that gave feedback, just prior to committing, about the number of tests that your changes will trigger. If the number of tests is larger than you intuitively expect, it could be an indication that the change has side-effects you haven't anticipated.
- mannykannot 11y ago> Test selection algorithm is not perfect yet (it may produce false-positives and false-negatives) I haven't read the code, but the associated document explaining the method makes no mention of accounting for line renumbering as a result of changes. The accuracy with which changes are identified depends on the diff algorithm. I imagine it is more likely to generate false positives than false negatives (which is preferable), but I am sure there are cases where a line is unchanged textually but is different semantically (e.g. by the insertion or removal of an 'else' above it), and the diff program will consider it unchanged.
- wdewind 11y agoIt's not only that, once you know whether or not something has changed you then need to know what tests need to be run.
- hinkley 11y agoFlakey always seems tolerable when you have 400 tests and you get one or two false reports a week. What people don't get is the the chance of failure is multiplicative, not additive, not fixed. So when you use that function 10 times as often it fails 12 or more times as often. Tests are there to fight our natural tendency toward wishful thinking. Tests that fail for no reason enable people to keep thinking there was nothing wrong with their code. Meanwhile other people are twiddling waiting for them to fix something they don't think they broke.
- wdewind 11y agoSorry I didn't mean flakey in the sense of the tests themselves, but the algorithm that selects which tests need to be run.