2 ms·
Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on. Example: a function makes two RPC cal
by theamk 2y ago
Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on.
Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests.
Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive.
Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives.
I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.