3 ms·

>Meta-programming to generate tests has only lead to pain in my experience. Likewise, although I view it as a deficiency of testing frameworks that require met

>Meta-programming to generate tests has only lead to pain in my experience. Likewise, although I view it as a deficiency of testing frameworks that require metaprogramming for parameterized testing. They shouldn't.

barrkel 10y ago
And on the other hand, I'm always suspicious of code that does something programming-like that is specific to the test framework. If you need a loop, ideally you write the loop using the language, rather than using a test framework's special loop construct. For parameterization, ideally you'd use a function, rather than a special test framework concept for parameterization. Having duals of every abstraction feature of a language in the test framework increases the cognitive load and ramp up time to become effective in a codebase, and there's always the risk of incomplete knowledge, leading to mixing and matching different abstractions - and test framework abstractions are usually more leaky and less well thought out than language abstractions.
- aisofteng 10y ago
  Furthermore, they decrease readability and clarity of tests themselves. I am of the strong opinion that if you need to test 20 inputs, you should write 20 asserts, not a loop - when the test fails, you should be able to immediately tell _which_ input(s) failed, not just that a failure has occurred. I came to this opinion by way of desiring to be clever in my tests and then finding myself having to insert `printf(input)` inside a loop in order to quickly figure out which input failed. Code should be written so that it doesn't need to be modified. Feelings of aesthetics of desires to be clever should be avoided and are, ultimately, I think, often just vanity.
  - barrkel 10y ago
    I'm thinking of rspec in particular, as it's the framework I write most tests in. It's trivial to include the loop variable in the string interpolation that describes the "context" block for the loop inner test setup. But usually I wouldn't be using a plain loop, I'd be iterating over entries from a hash map. The same thing in something like junit is less workable, and copy and paste becomes more viable simply because the language is so clumsy and inexpressive. I've seen various extensions and annotations that let you parameterize junit tests but I don't think they make things much clearer. A key benefit of driving the test cases off a table of inputs to outputs is that missing cases in cross products are much easier to spot. Visualising the state space and its coverage is easier when you can see a map of the covered space in one block of code.

barrkel 10y ago

And on the other hand, I'm always suspicious of code that does something programming-like that is specific to the test framework. If you need a loop, ideally you write the loop using the language, rather than using a test framework's special loop construct. For parameterization, ideally you'd use a function, rather than a special test framework concept for parameterization. Having duals of every abstraction feature of a language in the test framework increases the cognitive load and ramp up time to become effective in a codebase, and there's always the risk of incomplete knowledge, leading to mixing and matching different abstractions - and test framework abstractions are usually more leaky and less well thought out than language abstractions.

aisofteng 10y ago
Furthermore, they decrease readability and clarity of tests themselves. I am of the strong opinion that if you need to test 20 inputs, you should write 20 asserts, not a loop - when the test fails, you should be able to immediately tell _which_ input(s) failed, not just that a failure has occurred. I came to this opinion by way of desiring to be clever in my tests and then finding myself having to insert `printf(input)` inside a loop in order to quickly figure out which input failed. Code should be written so that it doesn't need to be modified. Feelings of aesthetics of desires to be clever should be avoided and are, ultimately, I think, often just vanity.
- barrkel 10y ago
  I'm thinking of rspec in particular, as it's the framework I write most tests in. It's trivial to include the loop variable in the string interpolation that describes the "context" block for the loop inner test setup. But usually I wouldn't be using a plain loop, I'd be iterating over entries from a hash map. The same thing in something like junit is less workable, and copy and paste becomes more viable simply because the language is so clumsy and inexpressive. I've seen various extensions and annotations that let you parameterize junit tests but I don't think they make things much clearer. A key benefit of driving the test cases off a table of inputs to outputs is that missing cases in cross products are much easier to spot. Visualising the state space and its coverage is easier when you can see a map of the covered space in one block of code.