3 ms·
I just had this discussion yesterday on Slack. I think that your core test suite should be completely, 100% deterministic. Property-based testing/random input
by pmarreck 5y ago
I just had this discussion yesterday on Slack.
I think that your core test suite should be completely, 100% deterministic.
Property-based testing/random input testing/chaosmonkey/profiling/etc. are all a different type of testing to me. Still useful, but separate. Still should be run, but not every time.
You do not want non-deterministic tests. (And if you use random testing, ALWAYS record the seed so you can reproduce it!)
If the core functionality of your code is fundamentally random (I once tested a maze generator this way https://github.com/pmarreck/elixir-snippets/blob/master/maze_generator.exs https://github.com/pmarreck/elixir-snippets/blob/master/maze...), use seeded random data.
- softwarebeware 5y agoCame here to say this same thing. Create a deterministic suite of tests that runs on every commit. Save the exploratory testing for another time / place. Keep them separate.
- hinkley 5y agoI’ve spent more time than I wanted to fixing tests that used a factory for test data and the factory didn’t understand things like, “all the results have to be unique”,and so would fail once in a while. When you repeat “once in a while” 100 times, now they are failing a couple times a week. The whole point of tests is 1) to have confidence in the code you’re shipping, and 2) to only demand attention when something is wrong. The latter is why tests have to be fast, and deterministic. Random failures and thumb twiddling both cost you things you are paying a huge chunk of theoretical developer output not to have to deal with. It’s often been the people who don’t like tests who write the least effective ones, and sometimes it’s hard to discern lack of experience from wish fulfillment. If these were acceptable tests, I’d hate unit testing too.