4 ms·

A new approach to property based testing

35 points by luu 11y ago

exDM69 11y ago
Interesting read and a very clever idea to minimize test cases. I've been doing similar property based testing on some math/physics heavy code, with a somewhat similar idea but without minimization. I have test cases that take a number of floating point arguments in range 0..1, which are generated from a 64 bit integer "seed" value picked from a special sequence that is ordered to produce failures as early as possible (failures are usually at 0, 1 or 1/2). Any failure can be reproduced by re-running the test case with the same seed value. In my case, minimization doesn't make sense since each test case takes a certain number of floats as input. I use about a minute of CPU time per test case (running with 8M different seeds). Example test case (it's about orbital mechanics): https://github.com/rikusalminen/twobody/blob/master/test/twobody/eccentric_anomaly_test.c https://github.com/rikusalminen/twobody/blob/master/test/two...
_pmf_ 11y ago
By all means, please deliver; this sounds really nice! If there's one area in software development, it's more approachable and extensive testing rigs that work end-to-end (i.e. that don't rely on mocks, but work with actual data input and output).
StavrosK 11y ago
I love Hypothesis. I can't find many practical uses for it (it's more useful in more "algorithmic" code, rather than for interactive apps), but it's fantastic in what it does.
- DRMacIver 11y ago
  Have you seen the talk I did about using it with Django? http://www.drmaciver.com/2015/06/hypothesis-for-django/ http://www.drmaciver.com/2015/06/hypothesis-for-django/ We're all still figuring out the details, but this sort of testing definitely works for testing your applications as well as your algorithms. ETA: Oh, and thanks for the kind words. :-)
  - StavrosK 11y ago
    I didn't, but I will and get back to you. There's certainly a need for randomness in unit tests, since I can't be trusted to write tests for the code I wrote (if I knew everything that would break before writing the tests, I'd fix it in the first place).
- davidrusu 11y ago
  I've been using hypothesis for testing a decent sized flask project at work and it's been great.
- chriswarbo 11y ago
  Not sure if it's directly applicable to Hypothesis, but one way to use QuickCheck for interactive code is to define a type representing actions to take, eg. data Action = DoFoo | DoBar | DoSequence [Action] | DoParallel [Action] We can then write an "run" function which executes these action, and use it in our properties.
  - DRMacIver 11y ago
    It's pretty easy to do this sort of thing in Hypothesis. You can also use [the support for stateful testing directly](http://hypothesis.readthedocs.org/en/latest/stateful.html http://hypothesis.readthedocs.org/en/latest/stateful.html). I find there's still a place for more granular tests that look more like your classic property based tests with a bit more imperative state floating around though.
jasonwatkinspdx 11y ago
This is quite cool, and opened up a different way of looking at property based testing more like Metropolis or other random walk optimizers.

35 points by luu 11y ago

Interesting read and a very clever idea to minimize test cases. I've been doing similar property based testing on some math/physics heavy code, with a somewhat similar idea but without minimization. I have test cases that take a number of floating point arguments in range 0..1, which are generated from a 64 bit integer "seed" value picked from a special sequence that is ordered to produce failures as early as possible (failures are usually at 0, 1 or 1/2). Any failure can be reproduced by re-running the test case with the same seed value. In my case, minimization doesn't make sense since each test case takes a certain number of floats as input. I use about a minute of CPU time per test case (running with 8M different seeds). Example test case (it's about orbital mechanics): https://github.com/rikusalminen/twobody/blob/master/test/twobody/eccentric_anomaly_test.c https://github.com/rikusalminen/twobody/blob/master/test/two...

_pmf_ 11y ago

By all means, please deliver; this sounds really nice! If there's one area in software development, it's more approachable and extensive testing rigs that work end-to-end (i.e. that don't rely on mocks, but work with actual data input and output).

StavrosK 11y ago

I love Hypothesis. I can't find many practical uses for it (it's more useful in more "algorithmic" code, rather than for interactive apps), but it's fantastic in what it does.

DRMacIver 11y ago
Have you seen the talk I did about using it with Django? http://www.drmaciver.com/2015/06/hypothesis-for-django/ http://www.drmaciver.com/2015/06/hypothesis-for-django/ We're all still figuring out the details, but this sort of testing definitely works for testing your applications as well as your algorithms. ETA: Oh, and thanks for the kind words. :-)
- StavrosK 11y ago
  I didn't, but I will and get back to you. There's certainly a need for randomness in unit tests, since I can't be trusted to write tests for the code I wrote (if I knew everything that would break before writing the tests, I'd fix it in the first place).
davidrusu 11y ago
I've been using hypothesis for testing a decent sized flask project at work and it's been great.
chriswarbo 11y ago
Not sure if it's directly applicable to Hypothesis, but one way to use QuickCheck for interactive code is to define a type representing actions to take, eg. data Action = DoFoo | DoBar | DoSequence [Action] | DoParallel [Action] We can then write an "run" function which executes these action, and use it in our properties.
- DRMacIver 11y ago
  It's pretty easy to do this sort of thing in Hypothesis. You can also use [the support for stateful testing directly](http://hypothesis.readthedocs.org/en/latest/stateful.html http://hypothesis.readthedocs.org/en/latest/stateful.html). I find there's still a place for more granular tests that look more like your classic property based tests with a bit more imperative state floating around though.

jasonwatkinspdx 11y ago

This is quite cool, and opened up a different way of looking at property based testing more like Metropolis or other random walk optimizers.