3 ms·

Does anyone else here dislike automated unit testing? Where I work, we have unit tests that are all hand-coded. Even with that, it takes half an hour to run t

Does anyone else here dislike automated unit testing? Where I work, we have unit tests that are all hand-coded. Even with that, it takes half an hour to run through the full suite - too long to do on any minor change, but still manageable when preparing a sandbox for checkin. If we added in automated tests, we would have many times the number of tests, with a corresponding increase in the time taken to make them run. If you can't run unit tests quickly, their value is reduced. I personally feel that a well-designed test framework should test all of the common use cases, some obvious edge cases, and should be easily adaptable so that when bugs are found, you can readily create a unit test that reproduces the bug, preventing regression. I'm curious as to how others here decide which unit tests to write.

gojomo 19y ago
Sounds like you've got great test coverage, but the length of time it takes to run them is diluting their value, making you reluctant to run them as often and concerned about adding more. Maybe you could offload/parallelize the test runs -- eg to a on-demand virtual hosting -- so as to minimize test run time as an issue?
- demallien 18y ago
  This is probably the direction that we need to be taking. Another requirement that we have is that the code compiles without warnings for about 20 different target platforms. This takes hours, and we are currently writing a tool to parallelise those builds - we're hoping for a 20x speed increase. Maybe a similar approach could be achieved for running tests...
mleonhard 18y ago
How about splitting your tests into a fast smoke test and a long test suite? You could run the smoke test before checking in. You could set up a continuous build system that would build with your latest changes, launch the long-running test suite, and email you the results.
- demallien 18y ago
  In practice we run a subset of tests during development, but the current rule is that we should never checkin code that hasn't passed ALL of the unit tests. The idea being that each person is responsable for their own modifications passing all tests. If you check-in, and then get shifted onto another high priority task, and a later run of the unit tests discovers that a funky side effect from your changes breaks the unit tests, you are no longer available to fix the problem. Someone else has to do it, allowing you to escape responsability for your error. At least, that's the theory management here ascribes to...

gojomo 19y ago

Sounds like you've got great test coverage, but the length of time it takes to run them is diluting their value, making you reluctant to run them as often and concerned about adding more. Maybe you could offload/parallelize the test runs -- eg to a on-demand virtual hosting -- so as to minimize test run time as an issue?

demallien 18y ago
This is probably the direction that we need to be taking. Another requirement that we have is that the code compiles without warnings for about 20 different target platforms. This takes hours, and we are currently writing a tool to parallelise those builds - we're hoping for a 20x speed increase. Maybe a similar approach could be achieved for running tests...

mleonhard 18y ago

How about splitting your tests into a fast smoke test and a long test suite? You could run the smoke test before checking in. You could set up a continuous build system that would build with your latest changes, launch the long-running test suite, and email you the results.

demallien 18y ago
In practice we run a subset of tests during development, but the current rule is that we should never checkin code that hasn't passed ALL of the unit tests. The idea being that each person is responsable for their own modifications passing all tests. If you check-in, and then get shifted onto another high priority task, and a later run of the unit tests discovers that a funky side effect from your changes breaks the unit tests, you are no longer available to fix the problem. Someone else has to do it, allowing you to escape responsability for your error. At least, that's the theory management here ascribes to...