5 ms·

> 1) Automated tests are software Which is assumed to be free of defects because people are suckers for circular reasoning.

chriswarbo 7y ago
What are you talking about? The whole point of automated testing is to confront the empirical fact that all software tends to have bugs, so we shouldn't have much confidence that we got it right. We can increase our confidence by trying a few examples, but only a little, since the scope for failure is so huge. Hammering a system with thousands of edge-cases, interleavings, previous regressions, etc. can give us a bit more confidence; but takes a huge amount of time for a person to do manually. It's pretty easy to automate though, and there are a load of libraries and frameworks that make it even easier. If we want even more confidence in a system, we can verify it against a machine-checkable specification. That's still too hard to automate in most cases. In any particular situation we need to find an appropriate balance between the cost of increasing our confidence in the system (i.e. how much effort we're willing to spend on prove ourselves wrong) versus the cost of failure. Automated tests (especially property checks) are so low-cost and high-yield that I find them always worth doing, unless the consequence of failure is near-zero.
- chungleong 7y ago
  What logically follows from the two statements below: 1. "Automated tests are software" 2. "all software tends to have bugs"
  - chriswarbo 7y ago
    It follows that automated tests tend to have bugs. So what? That doesn't at all imply that we shouldn't use them. You're presumably using a whole heap of software to post your comments, all of which will tend to have bugs; yet you still find it useful. Boolean logic rarely captures the nuances of real-world situations, and the "rounding errors" accumulate at each step of such arguments, which can lead to some silly (but logically consistent) outcomes. I think you've fallen victim to this here. In a similar way, it's a false dichotomy to think of software as being either buggy xor perfect; of testing (or any other QA) as proving brokenness xor correctness; or to consider bugginess as implying uselessness. Notice that I framed things in terms of confidence (in a Bayesian sense) and costs; not absolutes.
jamietanna 7y ago
If you want true confidence in your functional tests then yep, [you need to write tests for them](https://www.jvt.me/posts/2018/11/07/unit-test-functional-tests/ https://www.jvt.me/posts/2018/11/07/unit-test-functional-tes...) - may seem like tests all the way down, but huge benefits reaped in the team + projects I'm on

chriswarbo 7y ago

What are you talking about? The whole point of automated testing is to confront the empirical fact that all software tends to have bugs, so we shouldn't have much confidence that we got it right. We can increase our confidence by trying a few examples, but only a little, since the scope for failure is so huge. Hammering a system with thousands of edge-cases, interleavings, previous regressions, etc. can give us a bit more confidence; but takes a huge amount of time for a person to do manually. It's pretty easy to automate though, and there are a load of libraries and frameworks that make it even easier. If we want even more confidence in a system, we can verify it against a machine-checkable specification. That's still too hard to automate in most cases. In any particular situation we need to find an appropriate balance between the cost of increasing our confidence in the system (i.e. how much effort we're willing to spend on prove ourselves wrong) versus the cost of failure. Automated tests (especially property checks) are so low-cost and high-yield that I find them always worth doing, unless the consequence of failure is near-zero.

chungleong 7y ago
What logically follows from the two statements below: 1. "Automated tests are software" 2. "all software tends to have bugs"
- chriswarbo 7y ago
  It follows that automated tests tend to have bugs. So what? That doesn't at all imply that we shouldn't use them. You're presumably using a whole heap of software to post your comments, all of which will tend to have bugs; yet you still find it useful. Boolean logic rarely captures the nuances of real-world situations, and the "rounding errors" accumulate at each step of such arguments, which can lead to some silly (but logically consistent) outcomes. I think you've fallen victim to this here. In a similar way, it's a false dichotomy to think of software as being either buggy xor perfect; of testing (or any other QA) as proving brokenness xor correctness; or to consider bugginess as implying uselessness. Notice that I framed things in terms of confidence (in a Bayesian sense) and costs; not absolutes.

jamietanna 7y ago

If you want true confidence in your functional tests then yep, [you need to write tests for them](https://www.jvt.me/posts/2018/11/07/unit-test-functional-tests/ https://www.jvt.me/posts/2018/11/07/unit-test-functional-tes...) - may seem like tests all the way down, but huge benefits reaped in the team + projects I'm on