5 ms·

> What improves (and guarantees) software quality is rigorous testing. To deliver high quality software, there is no other solution. This is 100% false. You ca

> What improves (and guarantees) software quality is rigorous testing. To deliver high quality software, there is no other solution. This is 100% false. You can’t inject quality into a bad design via testing. You can’t guarantee quality or correctness through testing. Testing is the second worst place to find errors. You get a MUCH higher roi by producing thoughtful written designs and getting them peer reviewed. And again with code reviews. (Peer review is best but self review can also be very effective.) I’m not saying don’t test — testing is valuable, especially because it forces you to design for testability, and a suite of automated tests is valuable when making changes later.

tggreene 6y ago
I agree that testing is not a panacea, but I also don't think it is completely divorced from design. I see testing and design as cooperative processes, I agree that design is almost always the highest yielding process in any non-trivial program, but I think of testing almost as a design phase that tries to invert the design to look for weaknesses.
- chestervonwinch 6y ago
  Right, spending all your effort on the one true design gets you in architect astronaut territory. Similarly for existing systems, conjuring up the grand rewrite in the sky (as Robert Martin calls it, I think?) is equally poor. Instead, recognize that design is important but will constantly evolve. Tests give the ability to incrementally redesign quickly and confidently.
- Zelphyr 6y ago
  I suspect what he’s referring to is the fact that so many people think that, because they have 100% code coverage in their tests, they have quality software. I’ve seen so many times a team claim that with pride when it’s brutally obvious to anyone who looks that the product is broken. They just don’t seem to be able to reconcile that problem when asked about it. What you test matters every bit as much as how much you test. Tests require thought and design as well. Too many developers don’t seem to understand that.
  - bstpierre 6y ago
    That's part of what I was talking about, but the original statement in the article was that rigorous testing guarantees software quality, and it also says "there is no other solution", and that is demonstrably false. One example: A telecom product that processed phone calls. I stepped into a role on the team building the call progression handling system, which was based on an ad hoc state machine and some inter-task communication that included a combination of mutexes and message queues. It was a mess. Our testers succeeded in churning out a lot of bug reports. Not a single one of those bug reports did anything to improve the quality of the system -- even if we're not being pedantic and saying that only the patches in response to the bug reports were what might have improved the quality. Because none of the patching we could do really made any net increase in quality. The only thing that improved system quality was stepping back and re-examining the design. We changed the existing undocumented design to get rid of deadlocks by just using one inter-task mechanism (just queues, no mutexes) and by formalizing the state machines so that none of them got stranded in an unexpected state. Once the design was formalized this way we found -- by inspection, not testing -- a number of fundamental bugs in the states and transitions. After fixing those, more system testing showed some more bugs, but we were able to assess these in the context of the design and fix them at the root cause rather than just hacking on patches. The design changes also enabled automated integration testing that wasn't possible with the old ball of mud. This helped save time in allowing the [mostly manual] system testers to focus on certain areas of the system. I've worked in other situations where lots of testing just resulted in lots of bug reports, which ended up mostly getting deferred and thus did nothing whatsoever to improve product quality. And I've worked on software teams that were in fact producing rather high quality software, but we were building the wrong thing. It's really hard to fix bad requirements through testing. I don't want to sound like I'm anti-testing, but I think it should be understood that testing isn't the source of quality, it's just the last phase of a process that hopefully has (non-test) mechanisms to ensure a quality product in the upstream process phases.
    - yawaramin 6y ago
      While I agree with you in general, it should be noted that OP was talking about unit tests, not manual or integration tests. Unit tests are always touted by dynamic typing people as their equivalent of the static typing fast feedback loop.
noisy_boy 6y ago
Testing tends to validate design from the point of testability. If it feels like the code is resisting testing, that usually ends up being an indicator of bad design (though not always vice versa).
- plafl 6y ago
  Exactly. Testability requires modularity and reproducibility.
- karmakaze 6y ago
  This is where we are now. We know how to make testable designs, since effort has been put towards it. What we lost is being able to distinguish good vs bad designs in other key aspects. One indicator I use is the number of test cases that are needed for covering. A poorly separated design will need to test many combinations having not isolated the factors.
  - noisy_boy 6y ago
    I have noticed another indicator of good design is how easy it is to add specific fixes/features without having to think too much about how many places you'll need to make changes. Interesting point about no. of test cases. I tend to end up with several combinations because I prefer to focus on integration tests more for crud-type applications (a test configuration type/class comes handy in such scenarios).
    - karmakaze 6y ago
      As an example, if testing a 'send' function that takes a sender, content/media type, and recipients where recipients can be a single contact or a group, one shouldn't have to check that all combinations of content/media-type works with single or group recipients and other variable option variables. If decomposed in orthogonal ways K+L+N+M tests and perhaps a handful overall edge cases should do and not KxLxNxM if each variable dimension has K, L, N, M possibilities. If not cleanly separated in the implementation structure, the cross product of test cases are needed.
koonsolo 6y ago
I wonder how much experience you have in the real world. All of us developers had this thought at some point. I've seen terrible code with plenty of quick hacks on top of that, which was running very stable, because it was battle tested for years in production. What do you think will happen when you refactor such code to a better design? All juniors would think this is the best way to get a stable codebase. Reality is that you will introduce bugs by refactoring, not get rid of it. Writing code means writing bugs, no matter how pretty your code is. This reminds me of some quote that "Junior developers write 1 bug per 10 lines of code, experienced coders know they write 1 bug per 10 lines of code"
- brabel 6y ago
  You might be one of those experienced developers who has gone through their entire career without ever meeting a well designed, beautifully architectured application, and therefore concluded, sensibly, that all real-world applications look like shit if you look inside, and you seem to have come to believe that this is the only possible way. I have a different experience. Code that is well designed makes mistakes look obvious and the logic feel natural. It's very hard to write code like that, but I've seen it and I've been able to write it myself, so when I see a bunch of complicated code that only tests can tell whether or not it works, I know that the problem is really that the best design simply has not been found yet. If I find code that looks like shit and I realize there's a better design for it that will make the code clear and natural, I won't think twice to refactor it ruthlessly if such code has been a source of problems and time wasted (and fixing difficult to read code is huge costly to morale as well, which when taken into consideration, increases the cost of technical debt by a lot).
  - koonsolo 6y ago
    > a well designed, beautifully architectured application I've seen everything during my career, bad code with lots of bugs, bad code running stable. Nice code full of bugs, and nice stable code. The reason something was stable was never how the code looked, but how battle tested the product was. You will learn, don't worry. Plus, at a certain point, it's about tradoffs and compromises. I would love to see highly optimized code that is easy to read. > only tests can tell whether or not it works Tests are nice, but the real test will be when users start breaking your app. I'm curious, what kind of product are you writing?

tggreene 6y ago

I agree that testing is not a panacea, but I also don't think it is completely divorced from design. I see testing and design as cooperative processes, I agree that design is almost always the highest yielding process in any non-trivial program, but I think of testing almost as a design phase that tries to invert the design to look for weaknesses.

chestervonwinch 6y ago
Right, spending all your effort on the one true design gets you in architect astronaut territory. Similarly for existing systems, conjuring up the grand rewrite in the sky (as Robert Martin calls it, I think?) is equally poor. Instead, recognize that design is important but will constantly evolve. Tests give the ability to incrementally redesign quickly and confidently.
Zelphyr 6y ago
I suspect what he’s referring to is the fact that so many people think that, because they have 100% code coverage in their tests, they have quality software. I’ve seen so many times a team claim that with pride when it’s brutally obvious to anyone who looks that the product is broken. They just don’t seem to be able to reconcile that problem when asked about it. What you test matters every bit as much as how much you test. Tests require thought and design as well. Too many developers don’t seem to understand that.
- bstpierre 6y ago
  That's part of what I was talking about, but the original statement in the article was that rigorous testing guarantees software quality, and it also says "there is no other solution", and that is demonstrably false. One example: A telecom product that processed phone calls. I stepped into a role on the team building the call progression handling system, which was based on an ad hoc state machine and some inter-task communication that included a combination of mutexes and message queues. It was a mess. Our testers succeeded in churning out a lot of bug reports. Not a single one of those bug reports did anything to improve the quality of the system -- even if we're not being pedantic and saying that only the patches in response to the bug reports were what might have improved the quality. Because none of the patching we could do really made any net increase in quality. The only thing that improved system quality was stepping back and re-examining the design. We changed the existing undocumented design to get rid of deadlocks by just using one inter-task mechanism (just queues, no mutexes) and by formalizing the state machines so that none of them got stranded in an unexpected state. Once the design was formalized this way we found -- by inspection, not testing -- a number of fundamental bugs in the states and transitions. After fixing those, more system testing showed some more bugs, but we were able to assess these in the context of the design and fix them at the root cause rather than just hacking on patches. The design changes also enabled automated integration testing that wasn't possible with the old ball of mud. This helped save time in allowing the [mostly manual] system testers to focus on certain areas of the system. I've worked in other situations where lots of testing just resulted in lots of bug reports, which ended up mostly getting deferred and thus did nothing whatsoever to improve product quality. And I've worked on software teams that were in fact producing rather high quality software, but we were building the wrong thing. It's really hard to fix bad requirements through testing. I don't want to sound like I'm anti-testing, but I think it should be understood that testing isn't the source of quality, it's just the last phase of a process that hopefully has (non-test) mechanisms to ensure a quality product in the upstream process phases.
  - yawaramin 6y ago
    While I agree with you in general, it should be noted that OP was talking about unit tests, not manual or integration tests. Unit tests are always touted by dynamic typing people as their equivalent of the static typing fast feedback loop.

noisy_boy 6y ago

Testing tends to validate design from the point of testability. If it feels like the code is resisting testing, that usually ends up being an indicator of bad design (though not always vice versa).

plafl 6y ago
Exactly. Testability requires modularity and reproducibility.
karmakaze 6y ago
This is where we are now. We know how to make testable designs, since effort has been put towards it. What we lost is being able to distinguish good vs bad designs in other key aspects. One indicator I use is the number of test cases that are needed for covering. A poorly separated design will need to test many combinations having not isolated the factors.
- noisy_boy 6y ago
  I have noticed another indicator of good design is how easy it is to add specific fixes/features without having to think too much about how many places you'll need to make changes. Interesting point about no. of test cases. I tend to end up with several combinations because I prefer to focus on integration tests more for crud-type applications (a test configuration type/class comes handy in such scenarios).
  - karmakaze 6y ago
    As an example, if testing a 'send' function that takes a sender, content/media type, and recipients where recipients can be a single contact or a group, one shouldn't have to check that all combinations of content/media-type works with single or group recipients and other variable option variables. If decomposed in orthogonal ways K+L+N+M tests and perhaps a handful overall edge cases should do and not KxLxNxM if each variable dimension has K, L, N, M possibilities. If not cleanly separated in the implementation structure, the cross product of test cases are needed.

koonsolo 6y ago

I wonder how much experience you have in the real world. All of us developers had this thought at some point. I've seen terrible code with plenty of quick hacks on top of that, which was running very stable, because it was battle tested for years in production. What do you think will happen when you refactor such code to a better design? All juniors would think this is the best way to get a stable codebase. Reality is that you will introduce bugs by refactoring, not get rid of it. Writing code means writing bugs, no matter how pretty your code is. This reminds me of some quote that "Junior developers write 1 bug per 10 lines of code, experienced coders know they write 1 bug per 10 lines of code"

brabel 6y ago
You might be one of those experienced developers who has gone through their entire career without ever meeting a well designed, beautifully architectured application, and therefore concluded, sensibly, that all real-world applications look like shit if you look inside, and you seem to have come to believe that this is the only possible way. I have a different experience. Code that is well designed makes mistakes look obvious and the logic feel natural. It's very hard to write code like that, but I've seen it and I've been able to write it myself, so when I see a bunch of complicated code that only tests can tell whether or not it works, I know that the problem is really that the best design simply has not been found yet. If I find code that looks like shit and I realize there's a better design for it that will make the code clear and natural, I won't think twice to refactor it ruthlessly if such code has been a source of problems and time wasted (and fixing difficult to read code is huge costly to morale as well, which when taken into consideration, increases the cost of technical debt by a lot).
- koonsolo 6y ago
  > a well designed, beautifully architectured application I've seen everything during my career, bad code with lots of bugs, bad code running stable. Nice code full of bugs, and nice stable code. The reason something was stable was never how the code looked, but how battle tested the product was. You will learn, don't worry. Plus, at a certain point, it's about tradoffs and compromises. I would love to see highly optimized code that is easy to read. > only tests can tell whether or not it works Tests are nice, but the real test will be when users start breaking your app. I'm curious, what kind of product are you writing?