4 ms·

This technique is also known as characterisation testing, golden-master testing, snapshot testing, and probably other names too. I recommend looking into https:

This technique is also known as characterisation testing, golden-master testing, snapshot testing, and probably other names too. I recommend looking into https://approvaltests.com/ https://approvaltests.com/ (already mentioned by another commenter). I use characterisation testing all the time, in a perhaps unusual application: Checking the behaviour of "Page Objects" (classes used to model the application-under-test in GUI-level automated testing) when the application-under-test has changed. That's right: Automated [characterisation] tests for my automated [GUI] tests! It makes GUI testing oh so much easier to maintain. I wrote it up here: https://david.rothlis.net/pageobjects/characterisation-tests https://david.rothlis.net/pageobjects/characterisation-tests

drothlis 6y ago
Another thought related to characterisation testing, this time from Jeremias Roessler's talk at QAFest Ukraine 2017 (https://www.youtube.com/watch?v=f4PT_u8hjhU https://www.youtube.com/watch?v=f4PT_u8hjhU): Traditional automated tests (with asserts) are "blacklisting" the changes that aren't allowed, whereas a characterisation test will catch any change in behaviour, and instead you have to "whitelist" the changes that are allowed (by providing regexes or some kind of pattern to match the dynamic output that is allowed to change).
pgt 6y ago
What I like about this idea of Approval Testing is that I don't have to write too many tests if I know my system is working, as long as I am notified when the behaviour of my system has changed (usually in some unexpected way). This would require auto-mocking of all subsystems to prevent side-effecting functions from sending email & SMSes.
noctune 6y ago
I recently found it useful for an approximation algorithm. Since it was an approximation, it didn't really make any hard guarantees on the output for a given input, but snapshot testing at least made it very clear when a change affected the output and to what.
abathur 6y ago
Thanks for the term "characterisation testing". I started doing this a few years ago while working on a Sphinx extension that defines new directives but also customizes a few output targets (including two ~plaintext output targets). It's not only useful to avoid accidental output changes, but it's essential for iterating with confidence that tiny changes at the code level affected the output as-expected over a mostly-representative fraction of the documentation set. I didn't have a term for it at the time. I guess about a year later I bumped into snapshot testing, but that's always felt like more of a metaphor.
chrismorgan 6y ago
I’m really glad to have these terms of art; now I know what to search for to find similar things. Thanks!

drothlis 6y ago

Another thought related to characterisation testing, this time from Jeremias Roessler's talk at QAFest Ukraine 2017 (https://www.youtube.com/watch?v=f4PT_u8hjhU https://www.youtube.com/watch?v=f4PT_u8hjhU): Traditional automated tests (with asserts) are "blacklisting" the changes that aren't allowed, whereas a characterisation test will catch any change in behaviour, and instead you have to "whitelist" the changes that are allowed (by providing regexes or some kind of pattern to match the dynamic output that is allowed to change).

pgt 6y ago

What I like about this idea of Approval Testing is that I don't have to write too many tests if I know my system is working, as long as I am notified when the behaviour of my system has changed (usually in some unexpected way). This would require auto-mocking of all subsystems to prevent side-effecting functions from sending email & SMSes.

noctune 6y ago

I recently found it useful for an approximation algorithm. Since it was an approximation, it didn't really make any hard guarantees on the output for a given input, but snapshot testing at least made it very clear when a change affected the output and to what.

abathur 6y ago

Thanks for the term "characterisation testing". I started doing this a few years ago while working on a Sphinx extension that defines new directives but also customizes a few output targets (including two ~plaintext output targets). It's not only useful to avoid accidental output changes, but it's essential for iterating with confidence that tiny changes at the code level affected the output as-expected over a mostly-representative fraction of the documentation set. I didn't have a term for it at the time. I guess about a year later I bumped into snapshot testing, but that's always felt like more of a metaphor.

chrismorgan 6y ago

I’m really glad to have these terms of art; now I know what to search for to find similar things. Thanks!