3 ms·
> Manual changes in our staging environment corrupted test data fixtures there's a lot here. Manual changes in your staging environment shouldn't affect your
by zemo 5y ago
> Manual changes in our staging environment corrupted test data fixtures
there's a lot here.
Manual changes in your staging environment shouldn't affect your tests, because your tests should be isolated from other environments.
Also fixtures are generally bad. Given some fixture representing an initial state S, a test utilizing this fixture along with some acceptance criteria is essentially testing that given the state S, running the tests executes some transformation T such that the state of the system is now S2; acceptance criteria evaluate S2 against some known-good value to confirm that T is the desired transformation. This is meaningless if the initial state S is not actually reachable by the system. The fixture itself does not prove that S is reachable: that S is reachable is taken as an act of faith.
So how do you determine that the initial state S is reachable by the system? Well, you have some other test, that starts with an initial state of nothing, that performs some transition (generating and inserting random data instead of using a fixture, for example) and gets nothing into the state S. By doing this, you've both created the state S _and_ verified that S is a valid, reachable state. Now you run your second test after the first test in sequence. To run N tests off of initial state S, you replay the initial test that produced state S N times, once for each dependent test. Sure that's a lot of work, but each sequence of testing events can be run in isolation from the others, so they can be run in parallel.
- aidenn0 5y agoSometimes you have "state S that we got from a coredump from a customer that happened once every 30 computer-years in their deployment so we know it is reachable, but haven't ever seen it happen in-house"