3 ms·

What are you talking about? The whole point of automated testing is to confront the empirical fact that all software tends to have bugs, so we shouldn't have mu

What are you talking about? The whole point of automated testing is to confront the empirical fact that all software tends to have bugs, so we shouldn't have much confidence that we got it right. We can increase our confidence by trying a few examples, but only a little, since the scope for failure is so huge. Hammering a system with thousands of edge-cases, interleavings, previous regressions, etc. can give us a bit more confidence; but takes a huge amount of time for a person to do manually. It's pretty easy to automate though, and there are a load of libraries and frameworks that make it even easier. If we want even more confidence in a system, we can verify it against a machine-checkable specification. That's still too hard to automate in most cases. In any particular situation we need to find an appropriate balance between the cost of increasing our confidence in the system (i.e. how much effort we're willing to spend on prove ourselves wrong) versus the cost of failure. Automated tests (especially property checks) are so low-cost and high-yield that I find them always worth doing, unless the consequence of failure is near-zero.

chungleong 8y ago
What logically follows from the two statements below: 1. "Automated tests are software" 2. "all software tends to have bugs"
- chriswarbo 8y ago
  It follows that automated tests tend to have bugs. So what? That doesn't at all imply that we shouldn't use them. You're presumably using a whole heap of software to post your comments, all of which will tend to have bugs; yet you still find it useful. Boolean logic rarely captures the nuances of real-world situations, and the "rounding errors" accumulate at each step of such arguments, which can lead to some silly (but logically consistent) outcomes. I think you've fallen victim to this here. In a similar way, it's a false dichotomy to think of software as being either buggy xor perfect; of testing (or any other QA) as proving brokenness xor correctness; or to consider bugginess as implying uselessness. Notice that I framed things in terms of confidence (in a Bayesian sense) and costs; not absolutes.

chungleong 8y ago

What logically follows from the two statements below: 1. "Automated tests are software" 2. "all software tends to have bugs"

chriswarbo 8y ago
It follows that automated tests tend to have bugs. So what? That doesn't at all imply that we shouldn't use them. You're presumably using a whole heap of software to post your comments, all of which will tend to have bugs; yet you still find it useful. Boolean logic rarely captures the nuances of real-world situations, and the "rounding errors" accumulate at each step of such arguments, which can lead to some silly (but logically consistent) outcomes. I think you've fallen victim to this here. In a similar way, it's a false dichotomy to think of software as being either buggy xor perfect; of testing (or any other QA) as proving brokenness xor correctness; or to consider bugginess as implying uselessness. Notice that I framed things in terms of confidence (in a Bayesian sense) and costs; not absolutes.