4 ms·
>The benefit of testing is confidence that things are working. No it gives you confidence that the test are passing. I have worked on code bases that are well
by collyw 6y ago
>The benefit of testing is confidence that things are working.
No it gives you confidence that the test are passing.
I have worked on code bases that are well tested and code bases with non existing testing. The tested ones contain just as many bugs as the non-tested ones as far as the end user is concerned. In a year and a half working at the "well tested" company, regression tests caught an error one time. That was good but considering the testing team was bigger than the dev team, I question if it was actually efficient or not.
- chriswarbo 6y agoYou misunderstand me. I never said that testing is an efficient way to gain confidence, or that tests give much confidence, or that only testing can provide confidence. Rather, I said: > The benefit of testing is confidence that things are working. Some people treat testing as a gamified metric (e.g. code coverage). Some people treat tests as executable documentation. Some people treat tests as some essential requirement, without which the software cannot be shipped. I'm saying that those are not good reasons to write tests. Tests should be written to give us confidence that things are working. How much confidence will we gain? How much do we want? How much will it cost? Where should our efforts be directed? Those vary from project to project, and is exactly why I say thought should be given to the cost/benefit tradeoff. (Personally I think automated testing can be an efficient way to quickly gain a lot of confidence in a system; especially if we do property checking of high-level functionality, instead of unit testing of low-level details. However, that's a separate point; we can disagree on that, whilst agreeing on the need for weighing costs versus benefits.)
- Jtsummers 6y agoRegression tests don't usually identify bugs in new features, unless they interact with the older code or required changing the older code. In my experience, they're useful to have, but not useful for identifying novel issues. I recall reading someone's analysis of applying fuzzing and that using a particular fuzzer would eventually stop discovering issues, but that didn't mean the system under test was perfect. It meant that they had worked out most (if not all) the errors that fuzzer could identify for them and needed a new fuzzer. The same thing with regression testing. I wouldn't stop running regression tests just because they aren't identifying issues anymore, but I wouldn't rely on them to discover novel issues. That's what the new, updated test suite should be doing. A regression test suite will not have a test to cover a new feature. It will only make sure that the new feature doesn't break the old one. [0] https://blog.regehr.org/archives/1796 https://blog.regehr.org/archives/1796 - found it