4 ms·
> As a person who worked on systems with thousands of tests. I cannot stress enough that there should never be any randomness added to the automatic tests. To
by someotherperson 5y ago
> As a person who worked on systems with thousands of tests. I cannot stress enough that there should never be any randomness added to the automatic tests.
To take a counter position, I'm a person who has also worked on many systems with many thousands of tests. Every single test suite I've seen pulling static fixtures as test data was brittle and near worthless. Introducing randomness to the data you test against is the only way to actually test your product with near-realistic inputs. To this end, packages like Faker[0] are worth their weight in gold.
[0] https://github.com/joke2k/faker https://github.com/joke2k/faker
- AtNightWeCode 5y agoWith random in this case, I mean data that changes between test runs. I often use generated data sets. If you can find errors by feeding random data into your tests, the tests does not cover the code in the first place.
- michaelt 5y agoWell, the question is what you do when a test starts failing non-deterministically. If your team respond to a random test failure by finding the root cause and fixing it, uncertain test inputs could be right for you. On the other hand, if your team responds by hitting the 'rerun tests' button until it passes? In that case random failures are just slowing you down.
- goto11 5y agoThe correct response is to "freeze" the parameters of the failing test into a regular test which fails consistently.
- someotherperson 5y agoI'm not entirely following here. What is the point of even having tests if there are teams out there who disregard their outputs? Freezing a set of parameters that always pass, in case you don't run into the failing edge case, defeats the entire point of testing in the first place. If it's one test that is failing because the test itself is defective, and if that test is not for a crucial part of the product, then skip the test in the interim and come back to it later. But giving it a happy path for success sounds really strange to me.
- pmarreck 5y agoI just had this discussion yesterday on Slack. I think that your core test suite should be completely, 100% deterministic. Property-based testing/random input testing/chaosmonkey/profiling/etc. are all a different type of testing to me. Still useful, but separate. Still should be run, but not every time. You do not want non-deterministic tests. (And if you use random testing, ALWAYS record the seed so you can reproduce it!) If the core functionality of your code is fundamentally random (I once tested a maze generator this way https://github.com/pmarreck/elixir-snippets/blob/master/maze_generator.exs https://github.com/pmarreck/elixir-snippets/blob/master/maze...), use seeded random data.
- softwarebeware 5y agoCame here to say this same thing. Create a deterministic suite of tests that runs on every commit. Save the exploratory testing for another time / place. Keep them separate.
- hinkley 5y agoI’ve spent more time than I wanted to fixing tests that used a factory for test data and the factory didn’t understand things like, “all the results have to be unique”,and so would fail once in a while. When you repeat “once in a while” 100 times, now they are failing a couple times a week. The whole point of tests is 1) to have confidence in the code you’re shipping, and 2) to only demand attention when something is wrong. The latter is why tests have to be fast, and deterministic. Random failures and thumb twiddling both cost you things you are paying a huge chunk of theoretical developer output not to have to deal with. It’s often been the people who don’t like tests who write the least effective ones, and sometimes it’s hard to discern lack of experience from wish fulfillment. If these were acceptable tests, I’d hate unit testing too.
- tpoacher 5y agoYou're both right. You're just talking about different things.
- kazinator 5y agoAutomatic tests should be brittle. Their job is not to robustly complete their apparent task, regardless of how the software has broken.