3 ms·

It sounds like you're only talking about unit testing, or perhaps automated FV. Performance, infrequent bugs, heisenbugs etc are all things we find through sys

by NLips 11y ago

It sounds like you're only talking about unit testing, or perhaps automated FV. Performance, infrequent bugs, heisenbugs etc are all things we find through system test. On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! You could even say unexecuted code can be considered this way - if it doesn't ever matter that a program requires more RAM than it should, who cares? Of course at that point you are clearly increasing the risk of a future programmer mucking something up because of the unused code, so I don't think I'd go that far in real life. Clearly some things are easier to find in code review / code coverage etc, but saying you can't test for bugs that occur rarely is untrue.

falcolas 11y ago
> I'd say any bug is discoverable through testing ... given sufficient time. If I give you a routine which takes one second to execute, has four parameters, which each have a maximum of 100 options, you simply can't test it to the point where you can declare it as being bug free anywhere in the next three decades. Of course, most of our routines have more than 100 possible inputs - many have an effectively infinite number of inputs.
- NLips 11y ago
  Absolutely agree - testing is all about risk management, not finding all the bugs! And the right move is often to give up trying to reproduce/find a difficult bug. WRT to inputs: QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv. (possible Bill Sempf)
the_af 11y ago
> On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! Agreed, but this is like saying "for any (relevant) bug B, I can write a test T that finds it". It doesn't mean you will actually write said test or even be aware you should test for B, even though the end user will later experience it and be harmed by it. This is what Dijkstra seems to be saying: that if your tests don't find bugs it doesn't mean your tests are complete. Important bugs almost surely will happen regardless of your tests. That they pass is a good thing, but it shouldn't instill you with disproportionate confidence that your system works as needed.
- NLips 11y ago
  I agree much with the article; it was the parent comment I was replying to. I disagree with the idea that a tester shouldn't consider hard-to-find bugs, or performance bugs. I disagree somewhat with the idea that a programmer error or coverage issue is an entirely independent issue from other bugs - if there's no possible manifestation, it's not a bug, and if there is a manifestation, then it's possible to find under another 'class' of bug. I fully agree that we won't execute tests to find all possible bugs. But similarly, code review doesn't find all code errors.

by NLips 11y ago

falcolas 11y ago

> I'd say any bug is discoverable through testing ... given sufficient time. If I give you a routine which takes one second to execute, has four parameters, which each have a maximum of 100 options, you simply can't test it to the point where you can declare it as being bug free anywhere in the next three decades. Of course, most of our routines have more than 100 possible inputs - many have an effectively infinite number of inputs.

NLips 11y ago
Absolutely agree - testing is all about risk management, not finding all the bugs! And the right move is often to give up trying to reproduce/find a difficult bug. WRT to inputs: QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv. (possible Bill Sempf)

the_af 11y ago

> On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! Agreed, but this is like saying "for any (relevant) bug B, I can write a test T that finds it". It doesn't mean you will actually write said test or even be aware you should test for B, even though the end user will later experience it and be harmed by it. This is what Dijkstra seems to be saying: that if your tests don't find bugs it doesn't mean your tests are complete. Important bugs almost surely will happen regardless of your tests. That they pass is a good thing, but it shouldn't instill you with disproportionate confidence that your system works as needed.

NLips 11y ago
I agree much with the article; it was the parent comment I was replying to. I disagree with the idea that a tester shouldn't consider hard-to-find bugs, or performance bugs. I disagree somewhat with the idea that a programmer error or coverage issue is an entirely independent issue from other bugs - if there's no possible manifestation, it's not a bug, and if there is a manifestation, then it's possible to find under another 'class' of bug. I fully agree that we won't execute tests to find all possible bugs. But similarly, code review doesn't find all code errors.