3 ms·
> the tests never try to mix PSYNC2 with replication of Lua scripts This sort of testing gap is so hard to avoid. Any complex system has an inexhaustible numbe
by snewman 9y ago
> the tests never try to mix PSYNC2 with replication of Lua scripts
This sort of testing gap is so hard to avoid. Any complex system has an inexhaustible number of potential feature interactions. It's very difficult at best to guess which combinations need to be tested, and often difficult to even write a test which exercises a particular combination.
What techniques do people use to ensure test coverage of complex cross-feature interactions like this? Two things I emphasize:
* Emphasize integration tests, and try to use multiple features in each test. This can help drive out interaction problems; the downside is that these tests are harder to write and (especially) maintain. It can also be difficult to exercise specific code paths.
* Write randomized tests, that exercise as many features as possible. The challenge here is identifying failures. "The program didn't assert or crash" is often a good start. A gold standard is to run a simple reference implementation alongside your real implementation, and compare the output... but that's not always possible.
- sillysaurus3 9y agoWhat techniques do people use to ensure test coverage of complex cross-feature interactions like this? The Feynman Algorithm: http://wiki.c2.com/?FeynmanAlgorithm http://wiki.c2.com/?FeynmanAlgorithm It's the uncomfortable truth we like to dance around. Just accept you can't plan for everything. Think carefully before making design decisions. Build in contingency plans in case things go wrong.
- antirez 9y agoThis. Fuzzing and integration tests is what usually is able to signal a regression.
- cortesoft 9y agoIf you mean randomized tests as in adding random choices to your tests, I think that is a very bad idea (that I have been bitten by before). You NEED tests to behave consistently. If you run the tests twice against the same code, it needs to return the same result. Otherwise, how will you know you have fixed the issue the tests caught? Speaking from experience, soon after you introduce randomness to tests, you will start to get intermittent test failures. When that happens, people will start to ignore failing tests because, 'hey, this is probably just a random failure'
- anonacct37 9y agoSeeded RNG.
- snewman 9y agoExactly, you use a fixed RNG seed so that the behavior is reproducible. For complex multithreaded code, it can become difficult to achieve consistent execution, but a fixed seed is at least a good start.
- cortesoft 9y agoSo not really random, then? What is the point of adding an RNG if it is going to output the same values every time?
- anonacct37 9y agoIf you generate 10k test cases that find edge cases you wouldn't have normally found, that's value. If the framework is deterministic and always runs the same 10k test cases, that's generally valuable as well. I'm not aware of anyone currently looking to ensure their test cases our generated using cryptographically random data.
- jeanmichelx 9y agoBecause then the testing stimulus is not hard coded anymore, but generated. It's then trivial to run longer sequences, or tweak the test. In hardware design, where reliability is key (making a chip costs multiple millions and 6 months time), there is only random testing (with seed for reproducibility), not "that bunch of case that the person testing thought of that day"
- zackelan 9y agoIt's possible to have both randomness and reproduciblility - generate a random seed, log it, then explicitly seed the RNG. If you encounter a failure you suspect is caused by a particular random seed, temporarily hard-code that seed while you debug it. The "that's an intermittent failure, just re-run it and it usually passes" attitude is in my experience more likely due to just plain old poorly-written tests, usually with a time.sleep() or something similar that makes them unreliable.