3 ms·
> There are generally two main categories of tests. There are unit tests [and] Integration tests Later on there is a mention of fuzzing. However, there is some
by rdtsc 5y ago
> There are generally two main categories of tests. There are unit tests [and] Integration tests
Later on there is a mention of fuzzing. However, there is something better called property tests. Where the on failure there is automatic "shrinking" of input complexity to find a minimally failing test scenario.
For Erlang and Elixir I recommend https://propertesting.com/ https://propertesting.com/ book and the PropEr test framework https://github.com/proper-testing/proper https://github.com/proper-testing/proper.
As an example, here is the use of an advanced feature called targeted property based testing: https://proper-testing.github.io/tutorials/PropEr_testing_with_search_strategies.html https://proper-testing.github.io/tutorials/PropEr_testing_wi... used to test a labyrinth.
EDIT: as spockz mentioned below, the original idea was developed by QuviQ as QuickCheck http://www.quviq.com/products/erlang-quickcheck/ http://www.quviq.com/products/erlang-quickcheck/. It's a commercial offering but there is a QuickCheck mini offered as a free download as well.
- spockz 5y agoThe original is quickcheck for erlang which also has an Haskell flavour. It has inspired scalacheck for scala. I test pure logic with property based testing and simple integration testing to ensure all the steps are aligned properly.
- rdtsc 5y agoGood point. I edited my comment to mention QuviQ and their QuickCheck product.
- mananaysiempre 5y agoMy understanding was that property tests and fuzzing are orthogonal in roughly the same way unit and integration tests are: you do property test to ensure well-delimited pieces of your code satisfy well-defined constraints, and you fuzz to ensure the whole thing just doesn’t go completely nuts when faced with something unusual. But I could be wrong because I never could figure out how to set up fuzzing for some sort of moderately-involved realistic thing (a CSV parser? something like that).
- rdtsc 5y agoI think property based testing could be used for integration tests as well. There are essentially two major steps to use it: 1) Define generators to produce input. These could start as random with some constraints. Say, "positive integers between 10 and 100" and then those can be combined in various ways with other generators like "positive integers between 10 and 100 in a tuple with a utf8 encoded binary between lengths 0 and 256" and so on. 2) Define properties which model your system. Given the inputs from generators, calling your application APIs should keep certain properties true. It should crash in only certain ways, or return only HTTP 200 responses, etc. These could be applied at any level from the simplest pure functions up to an HTTP API. These steps don't have to be taking in the same order. I end up starting with 2 going back to 1, then 2 again etc. The difference from basic fuzzing is that once you find a failure, it has a consistent and automatic way to backtrack and try to simplify the input down to a minimum which still causes the failure. So, if a giant 1Gb worth of JSON fails the fuzz test, it can be hard to see why it failed. With property testing, however, it might be simplified down a simple case with one utf8 encoded string of length 10 which has some specific emoji or something like that.
- icsa 5y ago+1
- yccs27 5y agoThose are separate classifications, right? A unit test can be property-based and an integration test can be accomplished theough fuzzing, and versa.
- rdtsc 5y agoIt's almost an extra dimension, yeah. I was just highlighting that there is something that is a strict improvement over fuzzing and it's not just academic but available for practical use today.