3 ms·
As an aside, it would be fairly simple to decrease the time taken for a unit test suite to run by switching to incremental testing--that is, by not running test
by codex 16y ago
As an aside, it would be fairly simple to decrease the time taken for a unit test suite to run by switching to incremental testing--that is, by not running tests for code that hasn't changed.
While I've never implemented something like this, I imagine it would be straightforward for compiled languages if you have a good linker:
1) compile each unit test as a statically linked executable
2) make sure that each unit test outputs to an individual file; e.g. ./test_a >> test_results_a.xml
3) jigger your build process such that the linker removes dead and unused code from each test executable. Now each test executable contains code that is directly used by the test.
4) ensure that your build process only touches a statically linked test if a dependency has actually changed. This comes for free most of the time, but you may still have the linker touching files that it doesn't have to. You can remove these cases by staging the test files using a binary comparitor tool like rsync. If you have a crappy linker, or there's something in your test executables which is always changing (like a build stamp) you might need to compare using more advanced tools (like binutils) or something like Google's Courgette.
5) Run your tests only when the test is newer than the results--that is, when test_a is newer than test_a_result.xml. This is straightforward to implement using most build systems, like make, or using a test runner script.
Bam--now a test only runs if some dependent code has changed. If your test also uses data, you should list it as a dependency in your build system--either manually or by detecting open() calls at runtime via a shim over libc, or via strace. Your test run takes much less time, and, best of all, developers get a lot less spam to read through. This is perhaps a much greater benefit than increased speed.
Dynamic languages are a much harder nut to crack, as are integration tests that are loosely bound to the code.
- elliottkember 16y agoI use autotest for this - it notices files being altered and runs relevant parts of your suite. If something goes wrong it'll keep trying that test everytime you save a file, until it works. I think it's fantastic.
- icefox 16y agoDo you have a link to autotest? I run something similar. My ruby projects are all in git. I have a git pre-commit hook that when file X is modified before allowing a commit will run the test for X and only allow me to commit if the tests for X passes. This happens with zero work on my part forcing me to never forget to run the tests. While it doesn't run all of the tests just running the associated test catches a very larger number of accidental test failures for very little cost. Lastly because the tests are being run all of the time if one suddenly takes a long time to run I quickly profile it and speed it up. And --no-verify is always there to bypass the hook if need be.
- Kaya 16y agoHow do you handle the case where file X uses file Y, and you commit file Y? You could run the test for X when Y is committed in case Y broke X.
- icefox 16y agoFor most of my projects I don't. Some of them have some rules listing test x depends upon y, but for the vast majority of my projects modify file X and only test X is run. This catches a huge non-insignificant amount of regressions. Really it comes down to effort. I can either A) try to remember to always run tests before committing or B) Have a basic hook that takes minutes to write/install that will always run at least one test and catch near all regressions. I tried to do A for years. Most of the time you run the tests, but not always. And heaven help you if you are on a team. There will be someone who never runs tests. And you will end up having to schedule a chunk of time for regression fixing before every release. So to answer you question who cares about when file X uses file Y.
- steveklabnik 16y agohttp://rubygems.org/gems/autotest http://rubygems.org/gems/autotest