3 ms·
I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems
by dkarl 10mo ago
I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead.
"Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and can be combined to create generators of other types. (The terminology varies from one library to another. Different libraries use the terms "generators," "arbitraries," and "strategies" in slightly different and overlapping ways.)
For example, if you have a generator for strings and a generator for non-negative integers, it's trivial to create a generator for a type Person(name, age).
Generators can also be filtered. For example, if you have a generator for Account instances, and you need active Account instances in your test, you can apply a filter to the base generator to select only the instances where _.isActive is true.
Once you have a base generator for each type you need in your tests, the individual tests become clear and succinct. There is a learning curve for working with generators, but as a rule, the test code is very easy to read, even if it's tricky to write at first.
- radanskoric 10mo agoAuthor here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages). The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite considerably. If you use them to generate purely in memory objects, this problem goes away and then I also prefer to use factories (or generators, as you describe them).
- dkarl 10mo agoOh, that's a very different set of requirements than I was thinking, and I missed that context even though you did mention database testing at one point. You're right, property-based testing is less helpful in that situation, because your database may contain legacy data that your current application code must be able to read but also shouldn't be able to write.
- tclancy 10mo agoNo, property-based testing is something more like https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/ -- it's like fuzz testing with some smarts and it is lovely where it fits.
- radanskoric 10mo agoAh, ok, now I understand. Ok, I wasn't talking about that. From what I understand about property based testing it's sort of half way between regular example based testing and formal proofs: It tries to prove a statement but instead of a symbolic proof it does it stohastically via a set of examples? Unfortunately, I'm not aware of a good property based testing library in Ruby, although it would be useful to have one. Even so I'm guessing that property based testing in practice would be too resource intensive to test the entire application with it? You'd probably only test critical domain logic components and use regular example tests for the rest.
- strehldev 10mo agoThis is basically how I solved this in a past codebase. I called them "builders" and for complex scenarios requiring multiple different entities I called them "scenario builders" that created multiple entities. My rule was to randomize every property by default. The test needs to specify which property needs to have a certain value. E.g. set the address if you're testing something about the address. So it was immediately obvious which properties a test relied on.
- japhyr 10mo agoHow did you deal with reproducibility when your tests use randomized data? Do you run with a random seed or something, so you can reproduce failures when they come up?
- dkarl 10mo agoScalaCheck includes the random seed in its failure messages, so it's easy to pull the seed out of CI/CD logs and reproduce the failure deterministically.
- FuckButtons 10mo agoForgive me if I’m just reading this incorrectly, but that doesn’t sound exactly like property testing as I’ve done it. the libraries implement an algorithm for narrowing down to the simplest reproducer for a given failure mode, so all of the inputs to a test that are randomized are provided by the library.
- dkarl 10mo agoYou should see if your language has a property-based testing library; it'll have a ton of useful functionality to help with what you're already doing! A clarification on terminology, the "property" in "property-based testing" refers to properties that code under test is supposed to obey. For example, in the author's Example 2, the collection being sorted is the property that the test is checking.