3 ms·

I generally agree with this post. What I have found to be ironic, personally, is that good automated testing often takes a long time to implement and it feels l

by rpedela 12y ago

I generally agree with this post. What I have found to be ironic, personally, is that good automated testing often takes a long time to implement and it feels like it takes forever sometimes. However it usually shortens the overall development time and increases quality because you can add new features, re-factor others, etc and then just run the tests to make sure you didn't break something. And if you did break something, you know immediately what it is wrong and can fix it.

sanderjd 12y ago
Lots of people seem to think that an automated test suite is a guarantee that you know when you've broken something. I have never found that to be true. Sometimes I have even found it to be a dangerous idea, leading to complacency in reasoning about the potential impacts of a change and actively verifying its behavior. It's great to have a good automated test suite, but it's much greater to also have people using and verifying software.
- rpedela 12y ago
  That is why I used the adjective "good". Like you said, just having an automated test suite is not necessarily useful. Let me define what I mean by "good". A good automated test suite has a very high (>90%) coverage of all cases that an actual user may encounter in the real world. It is extremely difficult to achieve this in the initial version of the test suite. The goal initially should be to build a test suite that can be quickly and easily updated so that >90% test coverage can be achieved over time. When people are "using and verifying software" or when there is a bug found in production, the bug fix should be accompanied by an update to the test suite. This way this particular bug will never happen again in production. In addition, you never let your guard down and continue this process for as long as the product exists. Over time this process will yield a test suite so good that you can assume that a particular change hasn't broken existing functionality even though there isn't a 100% guarantee. The cool thing now is that build tools are so good that you can codify some of this process into the build to ensure commits are tested before merged. And commits are reviewed to verify relevant tests were added or updated before merged.
  - sanderjd 12y ago
    I'd be curious how many people feel like they have automated test suites that achieve greater than 90% coverage of user actions (presumably this means the tests are at the integration level). Really good integration test suites are definitely great for catching regressions and being confident in making changes, but in my experience they are also difficult to build, slow to run, and expensive to maintain. Which is why, I think, people often focus more on getting high coverage of unit tests, which are much lower cost, but not nearly as useful for catching regressions.

by rpedela 12y ago

sanderjd 12y ago

Lots of people seem to think that an automated test suite is a guarantee that you know when you've broken something. I have never found that to be true. Sometimes I have even found it to be a dangerous idea, leading to complacency in reasoning about the potential impacts of a change and actively verifying its behavior. It's great to have a good automated test suite, but it's much greater to also have people using and verifying software.

rpedela 12y ago
That is why I used the adjective "good". Like you said, just having an automated test suite is not necessarily useful. Let me define what I mean by "good". A good automated test suite has a very high (>90%) coverage of all cases that an actual user may encounter in the real world. It is extremely difficult to achieve this in the initial version of the test suite. The goal initially should be to build a test suite that can be quickly and easily updated so that >90% test coverage can be achieved over time. When people are "using and verifying software" or when there is a bug found in production, the bug fix should be accompanied by an update to the test suite. This way this particular bug will never happen again in production. In addition, you never let your guard down and continue this process for as long as the product exists. Over time this process will yield a test suite so good that you can assume that a particular change hasn't broken existing functionality even though there isn't a 100% guarantee. The cool thing now is that build tools are so good that you can codify some of this process into the build to ensure commits are tested before merged. And commits are reviewed to verify relevant tests were added or updated before merged.
- sanderjd 12y ago
  I'd be curious how many people feel like they have automated test suites that achieve greater than 90% coverage of user actions (presumably this means the tests are at the integration level). Really good integration test suites are definitely great for catching regressions and being confident in making changes, but in my experience they are also difficult to build, slow to run, and expensive to maintain. Which is why, I think, people often focus more on getting high coverage of unit tests, which are much lower cost, but not nearly as useful for catching regressions.