4 ms·

At work, we're definitely turning away from traditional unit tests, which is not to say we're turning away from testing. Instead, for complex or algorithmic co

by lukev 14y ago

At work, we're definitely turning away from traditional unit tests, which is not to say we're turning away from testing. Instead, for complex or algorithmic code we're using generative testing, barraging the code with random inputs instead of a few cherry-picked test cases. Then, we're structuring our systems and operations in such a way that we can spin up test instances of the entire system and run full, end-to-end integration tests at various levels of load. We haven't figured it all out yet, but the results are promising compared to unit tests, especially tests which contain a lot of mocks and stubs.

DanWaterworth 14y ago
This is the approach that I take with my personal projects; I firmly believe quickcheck style testing is a huge improvement over manually writing out individual test cases. Do you tend to find, as I do, that when you write you code to be testable using randomized testing that it tends to look more functional? (I'm referring to the paradigm rather than the opposite of dysfunctional)
- lukev 14y ago
  Well, considering the code I'm talking about is mostly in Clojure, yeah, definitely functional. It's really cool to see how all the pieces fit together once you buy into the functional paradigm.
- kernel_sanders 14y ago
  I find that the same quickcheck style testing is the fastest way to get out quality code for a project that I am the sole devleoper of. I usually code in a terminal running screen with many shells going, one of which is running ipython so I can interactively import and test snippets, classes, or small functions that I'm writing in a larger module. It works great. However - those tests disappear! Once a recent project of mine was handed off to a team for maintenance and more features, I found that, in particular places, where the code was subtle, it was subject to breaking by others who were just trying to fix something unrelated. This was when I, personally, realized the value of unit testing. Yes, its slower and takes more time up front, but if your code may someday be worked on by others, it protects the sections of your code that have test coverage against breaking - which over a longer time scale can save time.
  - lukev 14y ago
    I believe the parent post was referring to QuickCheck (http://en.wikipedia.org/wiki/QuickCheck http://en.wikipedia.org/wiki/QuickCheck) as opposed to "quickly checking" code in the REPL. Tests definitely need to be repeatable.
dmlorenzetti 14y ago
For my algorithmic code, the unit tests do the random input selection. Actually, it's often a sort of guided random testing. For example, if the algorithm has a straightforward linear regime with an exact solution, and a nonlinear regime that has to be solved iteratively, the unit test will target, say, 2% of inputs at the linear regime, 1% at the breakpoint, and the rest at the nonlinear regime. Mockups and stubs are another matter. My algorithmic code is numerical, so stubs are rarely an issue. If an algorithm needs, say, a matrix solver, you pretty much just have to use a real (unit-tested) matrix solver in there.
scott_w 14y ago
The issue with using random test data is the it doesn't expose your edge cases. There was a post on HN a short while back that showed why - in essence your edge cases sit at the edge of the normal distribution ergo they will rarely be hit by the unit tests. This leads to you doing more work, as you need to log what data caused the test to fail. It also gives you a false sense of security, since the tests aren't really doing anything 99% of the time.

by lukev 14y ago

DanWaterworth 14y ago

This is the approach that I take with my personal projects; I firmly believe quickcheck style testing is a huge improvement over manually writing out individual test cases. Do you tend to find, as I do, that when you write you code to be testable using randomized testing that it tends to look more functional? (I'm referring to the paradigm rather than the opposite of dysfunctional)

lukev 14y ago
Well, considering the code I'm talking about is mostly in Clojure, yeah, definitely functional. It's really cool to see how all the pieces fit together once you buy into the functional paradigm.
kernel_sanders 14y ago
I find that the same quickcheck style testing is the fastest way to get out quality code for a project that I am the sole devleoper of. I usually code in a terminal running screen with many shells going, one of which is running ipython so I can interactively import and test snippets, classes, or small functions that I'm writing in a larger module. It works great. However - those tests disappear! Once a recent project of mine was handed off to a team for maintenance and more features, I found that, in particular places, where the code was subtle, it was subject to breaking by others who were just trying to fix something unrelated. This was when I, personally, realized the value of unit testing. Yes, its slower and takes more time up front, but if your code may someday be worked on by others, it protects the sections of your code that have test coverage against breaking - which over a longer time scale can save time.
- lukev 14y ago
  I believe the parent post was referring to QuickCheck (http://en.wikipedia.org/wiki/QuickCheck http://en.wikipedia.org/wiki/QuickCheck) as opposed to "quickly checking" code in the REPL. Tests definitely need to be repeatable.

dmlorenzetti 14y ago

For my algorithmic code, the unit tests do the random input selection. Actually, it's often a sort of guided random testing. For example, if the algorithm has a straightforward linear regime with an exact solution, and a nonlinear regime that has to be solved iteratively, the unit test will target, say, 2% of inputs at the linear regime, 1% at the breakpoint, and the rest at the nonlinear regime. Mockups and stubs are another matter. My algorithmic code is numerical, so stubs are rarely an issue. If an algorithm needs, say, a matrix solver, you pretty much just have to use a real (unit-tested) matrix solver in there.

scott_w 14y ago

The issue with using random test data is the it doesn't expose your edge cases. There was a post on HN a short while back that showed why - in essence your edge cases sit at the edge of the normal distribution ergo they will rarely be hit by the unit tests. This leads to you doing more work, as you need to log what data caused the test to fail. It also gives you a false sense of security, since the tests aren't really doing anything 99% of the time.