3 ms·

The benefit of testing is confidence that things are working. There are side benefits, like acting as documentation and examples, but they should not be the foc

The benefit of testing is confidence that things are working. There are side benefits, like acting as documentation and examples, but they should not be the focus. Writing tests (or deleting them, refactoring them, etc.) should always involve a cost/benefit calculation, even if it's a rough mental estimate that's not written down. In particular, that requires answering "How much effort will this test take me to write/debug?", "How much extra confidence will having this test give me?", "What level of confidence do I feel comfortable with?" and "Could I achieve higher confidence spending this effort on something else?". This doesn't need much effort. For example, we might think "This array indexing took me a few attempts to get right; it could result in misleading reports, so I'd better throw a few edge cases at it to make sure I understand it correctly.". On the other hand we might think "This class doesn't have any unit tests, but there's already a bunch of validation on the results; it only affects the page layout, and it'll be obvious if something's wrong, so I'll leave it for now and put some extra checks on the gnarly payment system." This approach makes sense regardless of the situation. In your startup example, it might make a lot of sense to write some tests that, for example, spider our site for broken links; or look for certain strings on certain pages. Things which are quick to write (they could even be shell scripts), but give us prior warning that our demo will go wrong. It would make less sense to write a pile of unit tests for the internals of some boilerplate piece of the system. Code coverage, dogma, etc. should not be the deciding factors for what to work on.

collyw 6y ago
>The benefit of testing is confidence that things are working. No it gives you confidence that the test are passing. I have worked on code bases that are well tested and code bases with non existing testing. The tested ones contain just as many bugs as the non-tested ones as far as the end user is concerned. In a year and a half working at the "well tested" company, regression tests caught an error one time. That was good but considering the testing team was bigger than the dev team, I question if it was actually efficient or not.
- chriswarbo 6y ago
  You misunderstand me. I never said that testing is an efficient way to gain confidence, or that tests give much confidence, or that only testing can provide confidence. Rather, I said: > The benefit of testing is confidence that things are working. Some people treat testing as a gamified metric (e.g. code coverage). Some people treat tests as executable documentation. Some people treat tests as some essential requirement, without which the software cannot be shipped. I'm saying that those are not good reasons to write tests. Tests should be written to give us confidence that things are working. How much confidence will we gain? How much do we want? How much will it cost? Where should our efforts be directed? Those vary from project to project, and is exactly why I say thought should be given to the cost/benefit tradeoff. (Personally I think automated testing can be an efficient way to quickly gain a lot of confidence in a system; especially if we do property checking of high-level functionality, instead of unit testing of low-level details. However, that's a separate point; we can disagree on that, whilst agreeing on the need for weighing costs versus benefits.)
- Jtsummers 6y ago
  Regression tests don't usually identify bugs in new features, unless they interact with the older code or required changing the older code. In my experience, they're useful to have, but not useful for identifying novel issues. I recall reading someone's analysis of applying fuzzing and that using a particular fuzzer would eventually stop discovering issues, but that didn't mean the system under test was perfect. It meant that they had worked out most (if not all) the errors that fuzzer could identify for them and needed a new fuzzer. The same thing with regression testing. I wouldn't stop running regression tests just because they aren't identifying issues anymore, but I wouldn't rely on them to discover novel issues. That's what the new, updated test suite should be doing. A regression test suite will not have a test to cover a new feature. It will only make sure that the new feature doesn't break the old one. [0] https://blog.regehr.org/archives/1796 https://blog.regehr.org/archives/1796 - found it
  - collyw 6y ago
    > Regression tests don't usually identify bugs in new features Neither do unit tests in most cases.

collyw 6y ago

>The benefit of testing is confidence that things are working. No it gives you confidence that the test are passing. I have worked on code bases that are well tested and code bases with non existing testing. The tested ones contain just as many bugs as the non-tested ones as far as the end user is concerned. In a year and a half working at the "well tested" company, regression tests caught an error one time. That was good but considering the testing team was bigger than the dev team, I question if it was actually efficient or not.

chriswarbo 6y ago
You misunderstand me. I never said that testing is an efficient way to gain confidence, or that tests give much confidence, or that only testing can provide confidence. Rather, I said: > The benefit of testing is confidence that things are working. Some people treat testing as a gamified metric (e.g. code coverage). Some people treat tests as executable documentation. Some people treat tests as some essential requirement, without which the software cannot be shipped. I'm saying that those are not good reasons to write tests. Tests should be written to give us confidence that things are working. How much confidence will we gain? How much do we want? How much will it cost? Where should our efforts be directed? Those vary from project to project, and is exactly why I say thought should be given to the cost/benefit tradeoff. (Personally I think automated testing can be an efficient way to quickly gain a lot of confidence in a system; especially if we do property checking of high-level functionality, instead of unit testing of low-level details. However, that's a separate point; we can disagree on that, whilst agreeing on the need for weighing costs versus benefits.)
Jtsummers 6y ago
Regression tests don't usually identify bugs in new features, unless they interact with the older code or required changing the older code. In my experience, they're useful to have, but not useful for identifying novel issues. I recall reading someone's analysis of applying fuzzing and that using a particular fuzzer would eventually stop discovering issues, but that didn't mean the system under test was perfect. It meant that they had worked out most (if not all) the errors that fuzzer could identify for them and needed a new fuzzer. The same thing with regression testing. I wouldn't stop running regression tests just because they aren't identifying issues anymore, but I wouldn't rely on them to discover novel issues. That's what the new, updated test suite should be doing. A regression test suite will not have a test to cover a new feature. It will only make sure that the new feature doesn't break the old one. [0] https://blog.regehr.org/archives/1796 https://blog.regehr.org/archives/1796 - found it
- collyw 6y ago
  > Regression tests don't usually identify bugs in new features Neither do unit tests in most cases.