4 ms·

"But you need to control complexity as you develop." One way to control the complexity is by consistent development of automatic regression tests which cover >

"But you need to control complexity as you develop." One way to control the complexity is by consistent development of automatic regression tests which cover >90% of functionality. Then you are much less afraid if you remove something or make a big change in production grade software used by many customers.

iofj 11y ago
This is one of those seductive statements that is true, but it's also such incredibly bad advice that just this one piece of theory has the potential to completely destroy your company if you follow it. But it's seductive for a reason : it's true. Doing regression tests on this scale will protect you from quite a sizable class of bugs. BUT there is no way to write lots of tests without locking down the implementation of the program. If you limit yourself to black box testing (meaning either no unit tests, or breaking and replacing unit tests is encouraged) you can retain some flexibility for a while. You want to be in the situation that if I go into an application and replace one algorithm with an equivalent one at any point in the app, none of the tests fail. I have never once seen any testsuite do that correctly. In most cases tests complicate an already complex codebase. This leads to having to choose between 2 approaches : either you ignore tests results (which of course does result in bugs), or you always fix tests (which of course leads to the tests becoming tautological over time as different people "fix" them and results in both more effort AND bugs). I don't disagree that for some software locking the functionality down is appropriate. But not for line of business software, apps with large UIs, essentially anything consumer facing ... It is only appropriate for standardized infrastructure software, things that manage accounts, process payments, regulate a power plant, avionics, ...
- akkartik 11y ago
  I've been thinking about this a lot, and I think a way out might be to drop the phrase 'black box' in your comment. I've been exploring white-box testing by making assertions on the log emitted by a program. Not only does it allow greater flexibility in large-scale reorganizations without needing to mess with tests, it also allows us to write tests for things that we don't currently write tests for, like performance (make sure the number of swaps in this sort function don't exceed this threshold), fault tolerance, race conditions, and so on. More details: http://akkartik.name/about http://akkartik.name/about. An early write-up on white-box testing: http://akkartik.name/post/tracing-tests http://akkartik.name/post/tracing-tests

iofj 11y ago

This is one of those seductive statements that is true, but it's also such incredibly bad advice that just this one piece of theory has the potential to completely destroy your company if you follow it. But it's seductive for a reason : it's true. Doing regression tests on this scale will protect you from quite a sizable class of bugs. BUT there is no way to write lots of tests without locking down the implementation of the program. If you limit yourself to black box testing (meaning either no unit tests, or breaking and replacing unit tests is encouraged) you can retain some flexibility for a while. You want to be in the situation that if I go into an application and replace one algorithm with an equivalent one at any point in the app, none of the tests fail. I have never once seen any testsuite do that correctly. In most cases tests complicate an already complex codebase. This leads to having to choose between 2 approaches : either you ignore tests results (which of course does result in bugs), or you always fix tests (which of course leads to the tests becoming tautological over time as different people "fix" them and results in both more effort AND bugs). I don't disagree that for some software locking the functionality down is appropriate. But not for line of business software, apps with large UIs, essentially anything consumer facing ... It is only appropriate for standardized infrastructure software, things that manage accounts, process payments, regulate a power plant, avionics, ...

akkartik 11y ago
I've been thinking about this a lot, and I think a way out might be to drop the phrase 'black box' in your comment. I've been exploring white-box testing by making assertions on the log emitted by a program. Not only does it allow greater flexibility in large-scale reorganizations without needing to mess with tests, it also allows us to write tests for things that we don't currently write tests for, like performance (make sure the number of swaps in this sort function don't exceed this threshold), fault tolerance, race conditions, and so on. More details: http://akkartik.name/about http://akkartik.name/about. An early write-up on white-box testing: http://akkartik.name/post/tracing-tests http://akkartik.name/post/tracing-tests