2 ms·
I think I get what you are saying, but in the absence of good tests I still vastly prefer slow, inexact tests to no tests at all. Our integration test suite is
by Aeolun 4y ago
I think I get what you are saying, but in the absence of good tests I still vastly prefer slow, inexact tests to no tests at all.
Our integration test suite is absolutely horrid. Like, the thing takes literally 25 hours of real time to run, so it’s split into 50 pieces running in parallel, ultimately taking just under 30 minutes. It has like 3000 unique scenarios and pretty nearly 95% of every scenario is exactly the same. It’s a _huge_ waste.
If it breaks, I pretty much only know that it’s broken (often because of some flaky behavior, or timing related thing, not an actual issue). Sometimes pinpointing the issue goes meta and is a matter of looking at which tests are broken, instead of how they fail.
But when all those tests succeed, I know that my system is just as stable as it was before. All the major flows are covered, and if it runs successfully, I can be fairly certain my system works.
Now imagine dealing with the codebase that leads to such a test suite without it…
- marginalia_nu 4y agoThis is a false dichotomy. The alternatives aren't having no tests having a half an hour test suite. As you say, 95% of every scenario is exactly the same, which I think is relatively common in this type of code base. The ideal test time is thus 90 seconds instead of 30 minutes. Maybe that unattainable, but I wager there exists a test suite that has the same coverage but with significantly less redundancy. Test engineering is actually hard. If testing is just some afterthought primarily aimed at coverage with not much consideration put toward performance, what you typically get is redundant tests with abysmal run-times.