4 ms·
That's part of what I was talking about, but the original statement in the article was that rigorous testing guarantees software quality, and it also says "ther
by bstpierre 6y ago
That's part of what I was talking about, but the original statement in the article was that rigorous testing guarantees software quality, and it also says "there is no other solution", and that is demonstrably false.
One example: A telecom product that processed phone calls. I stepped into a role on the team building the call progression handling system, which was based on an ad hoc state machine and some inter-task communication that included a combination of mutexes and message queues. It was a mess. Our testers succeeded in churning out a lot of bug reports. Not a single one of those bug reports did anything to improve the quality of the system -- even if we're not being pedantic and saying that only the patches in response to the bug reports were what might have improved the quality. Because none of the patching we could do really made any net increase in quality. The only thing that improved system quality was stepping back and re-examining the design. We changed the existing undocumented design to get rid of deadlocks by just using one inter-task mechanism (just queues, no mutexes) and by formalizing the state machines so that none of them got stranded in an unexpected state. Once the design was formalized this way we found -- by inspection, not testing -- a number of fundamental bugs in the states and transitions. After fixing those, more system testing showed some more bugs, but we were able to assess these in the context of the design and fix them at the root cause rather than just hacking on patches. The design changes also enabled automated integration testing that wasn't possible with the old ball of mud. This helped save time in allowing the [mostly manual] system testers to focus on certain areas of the system.
I've worked in other situations where lots of testing just resulted in lots of bug reports, which ended up mostly getting deferred and thus did nothing whatsoever to improve product quality.
And I've worked on software teams that were in fact producing rather high quality software, but we were building the wrong thing. It's really hard to fix bad requirements through testing.
I don't want to sound like I'm anti-testing, but I think it should be understood that testing isn't the source of quality, it's just the last phase of a process that hopefully has (non-test) mechanisms to ensure a quality product in the upstream process phases.
- yawaramin 6y agoWhile I agree with you in general, it should be noted that OP was talking about unit tests, not manual or integration tests. Unit tests are always touted by dynamic typing people as their equivalent of the static typing fast feedback loop.