4 ms·

While bringing down test runtime 1000x is certainly a nice feat, I don't think this is setting a right precedent. I, for one, would be very afraid. Tests are no

While bringing down test runtime 1000x is certainly a nice feat, I don't think this is setting a right precedent. I, for one, would be very afraid. Tests are not supposed to be written with speed in mind. Metrics seem to dominate the world of tests, where the spirit is supposed to be catching real world bugs. For example, the following can be done without destroying the spirit of testing: * Stub the sign in (big savings) * Keep a tab on fixtures (big savings) * Parallellize (big savings) * Buy a better machine (big savings) * Truncate with transactions/container-snapshots/etc (medium savings) * Tune database for speed than safety (medium savings) * Tune GC (medium savings), with enough RAM (16GB) disable it entirely (big savings) * Disable logging and instrumentation, yes including timing measurements, unless intentionally needed (small to medium savings) * Mount entire codebase, database and resources in tmpfs (small savings) The below, while they might improve performance, go against the spirit: * Tautological tests (http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-test-driven-development-anti-pattern/ http://fabiopereira.me/blog/2010/05/27/ttdd-tautological-tes...) * Too many mocks/metaprogramming * Using a different database driver I'm planning to write a tool, which given a Class, will pump out lines and lines of tautological tests, line for line (a.a should == a.a), method for method (a.should receive save, a.save), branch for branch (with some metaprogramming). Coverage will report a cool 100%, tests will be blazing fast, and that should calm the lead devs down.

maccard 11y ago
> Tests are not supposed to be written with speed in mind This is why the tests took 100 seconds in the first place. While I agree that speed isn't the number one priority for everything, Reducing the length of time your entire test suite takes to run to be doable on every single run of your framework is absolutely worthwhile.
josteink 11y ago
> Tests are not supposed to be written with speed in mind. There's always extremes. If people write tests with no regard for speed (as I've seen) you can have "unit-tests" run for 45 minutes before completing. That might be OK to guard against regressions on a nightly-build, but it will in no way be a test-suite I'm willing to running prior to every commit I ship. If the test-suite runs in less than a minute, I'm much more willing to do so before submitting code, and in that way avoiding regressions which needs to be fixed later on. If the tests runs in less than 10 seconds, I'm probably going to run them for every major change I do, even if I'm not planning on submitting anything yet, just for peace of mind. There's IDEs where tests can be run automatically in the background as you make changes and let you know the second you broke something. For that to work efficiently, your tests has to be sufficiently fast. So maybe the aim shouldn't be that tests must be fast above everything else. But if the test-suite can be run quickly and casually, it's value will increase tremendously.

maccard 11y ago

> Tests are not supposed to be written with speed in mind This is why the tests took 100 seconds in the first place. While I agree that speed isn't the number one priority for everything, Reducing the length of time your entire test suite takes to run to be doable on every single run of your framework is absolutely worthwhile.

josteink 11y ago

> Tests are not supposed to be written with speed in mind. There's always extremes. If people write tests with no regard for speed (as I've seen) you can have "unit-tests" run for 45 minutes before completing. That might be OK to guard against regressions on a nightly-build, but it will in no way be a test-suite I'm willing to running prior to every commit I ship. If the test-suite runs in less than a minute, I'm much more willing to do so before submitting code, and in that way avoiding regressions which needs to be fixed later on. If the tests runs in less than 10 seconds, I'm probably going to run them for every major change I do, even if I'm not planning on submitting anything yet, just for peace of mind. There's IDEs where tests can be run automatically in the background as you make changes and let you know the second you broke something. For that to work efficiently, your tests has to be sufficiently fast. So maybe the aim shouldn't be that tests must be fast above everything else. But if the test-suite can be run quickly and casually, it's value will increase tremendously.