3 ms·

What classes of bugs do you have in mind that testing cannot work for?

by pekk 11y ago

jacquesm 11y ago
Testing works extremely well for anything that you either find in the wild or anticipate beforehand. Hard to find using testing: performance related issues (for which you'll need a profiler), bugs that occur rarely (for instance, one bug I recently uncovered in 'hugo' was so rare it took 1000+ runs of the test software before it turned up on the machine of the lead dev, initially he did not believe my report, see https://github.com/spf13/hugo/issues/1293 https://github.com/spf13/hugo/issues/1293 ), programmer errors (this is where code review comes in), coverage issues such as routines that are simply never exercised (for that we use coverage tools), errors in the tests themselves, heisenbugs and so on. Bugs come in many shapes and sizes and testing is a very powerful strategy but it is not the only strategy.
- NLips 11y ago
  It sounds like you're only talking about unit testing, or perhaps automated FV. Performance, infrequent bugs, heisenbugs etc are all things we find through system test. On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! You could even say unexecuted code can be considered this way - if it doesn't ever matter that a program requires more RAM than it should, who cares? Of course at that point you are clearly increasing the risk of a future programmer mucking something up because of the unused code, so I don't think I'd go that far in real life. Clearly some things are easier to find in code review / code coverage etc, but saying you can't test for bugs that occur rarely is untrue.
  - falcolas 11y ago
    > I'd say any bug is discoverable through testing ... given sufficient time. If I give you a routine which takes one second to execute, has four parameters, which each have a maximum of 100 options, you simply can't test it to the point where you can declare it as being bug free anywhere in the next three decades. Of course, most of our routines have more than 100 possible inputs - many have an effectively infinite number of inputs.
    - NLips 11y ago
      Absolutely agree - testing is all about risk management, not finding all the bugs! And the right move is often to give up trying to reproduce/find a difficult bug. WRT to inputs: QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv. (possible Bill Sempf)
  - the_af 11y ago
    > On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! Agreed, but this is like saying "for any (relevant) bug B, I can write a test T that finds it". It doesn't mean you will actually write said test or even be aware you should test for B, even though the end user will later experience it and be harmed by it. This is what Dijkstra seems to be saying: that if your tests don't find bugs it doesn't mean your tests are complete. Important bugs almost surely will happen regardless of your tests. That they pass is a good thing, but it shouldn't instill you with disproportionate confidence that your system works as needed.
    - NLips 11y ago
      I agree much with the article; it was the parent comment I was replying to. I disagree with the idea that a tester shouldn't consider hard-to-find bugs, or performance bugs. I disagree somewhat with the idea that a programmer error or coverage issue is an entirely independent issue from other bugs - if there's no possible manifestation, it's not a bug, and if there is a manifestation, then it's possible to find under another 'class' of bug. I fully agree that we won't execute tests to find all possible bugs. But similarly, code review doesn't find all code errors.
gspetr 11y ago
One of these classes requires manual interaction from a user at the GUI level: http://blog.8thlight.com/uncle-bob/2014/04/30/When-tdd-does-not-work.html http://blog.8thlight.com/uncle-bob/2014/04/30/When-tdd-does-...

What classes of bugs do you have in mind that testing cannot work for?

by pekk 11y ago

What classes of bugs do you have in mind that testing cannot work for?

Testing works extremely well for anything that you either find in the wild or anticipate beforehand. Hard to find using testing: performance related issues (for which you'll need a profiler), bugs that occur rarely (for instance, one bug I recently uncovered in 'hugo' was so rare it took 1000+ runs of the test software before it turned up on the machine of the lead dev, initially he did not believe my report, see https://github.com/spf13/hugo/issues/1293 https://github.com/spf13/hugo/issues/1293 ), programmer errors (this is where code review comes in), coverage issues such as routines that are simply never exercised (for that we use coverage tools), errors in the tests themselves, heisenbugs and so on. Bugs come in many shapes and sizes and testing is a very powerful strategy but it is not the only strategy.

NLips 11y ago
It sounds like you're only talking about unit testing, or perhaps automated FV. Performance, infrequent bugs, heisenbugs etc are all things we find through system test. On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! You could even say unexecuted code can be considered this way - if it doesn't ever matter that a program requires more RAM than it should, who cares? Of course at that point you are clearly increasing the risk of a future programmer mucking something up because of the unused code, so I don't think I'd go that far in real life. Clearly some things are easier to find in code review / code coverage etc, but saying you can't test for bugs that occur rarely is untrue.
- falcolas 11y ago
  > I'd say any bug is discoverable through testing ... given sufficient time. If I give you a routine which takes one second to execute, has four parameters, which each have a maximum of 100 options, you simply can't test it to the point where you can declare it as being bug free anywhere in the next three decades. Of course, most of our routines have more than 100 possible inputs - many have an effectively infinite number of inputs.
  - NLips 11y ago
    Absolutely agree - testing is all about risk management, not finding all the bugs! And the right move is often to give up trying to reproduce/find a difficult bug. WRT to inputs: QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv. (possible Bill Sempf)
- the_af 11y ago
  > On some level, I'd say any bug is discoverable through testing - if a user can't experience a bug, then it doesn't exist! Agreed, but this is like saying "for any (relevant) bug B, I can write a test T that finds it". It doesn't mean you will actually write said test or even be aware you should test for B, even though the end user will later experience it and be harmed by it. This is what Dijkstra seems to be saying: that if your tests don't find bugs it doesn't mean your tests are complete. Important bugs almost surely will happen regardless of your tests. That they pass is a good thing, but it shouldn't instill you with disproportionate confidence that your system works as needed.
  - NLips 11y ago
    I agree much with the article; it was the parent comment I was replying to. I disagree with the idea that a tester shouldn't consider hard-to-find bugs, or performance bugs. I disagree somewhat with the idea that a programmer error or coverage issue is an entirely independent issue from other bugs - if there's no possible manifestation, it's not a bug, and if there is a manifestation, then it's possible to find under another 'class' of bug. I fully agree that we won't execute tests to find all possible bugs. But similarly, code review doesn't find all code errors.

gspetr 11y ago

One of these classes requires manual interaction from a user at the GUI level: http://blog.8thlight.com/uncle-bob/2014/04/30/When-tdd-does-not-work.html http://blog.8thlight.com/uncle-bob/2014/04/30/When-tdd-does-...