4 ms·

I don't agree with Anti-Pattern 10 - Not converting production bugs to tests I used to think this was a good idea too, until I saw the real statistics on a pro

I don't agree with Anti-Pattern 10 - Not converting production bugs to tests I used to think this was a good idea too, until I saw the real statistics on a project on this. This project (50+ developer team) tracked all bugs, and also if they were regressions or not. Almost 0 regression occurred of bugs that were fixed before. All testing needs to consider return on investment. The reality, at least for that project, was that testing time was best spent elsewhere.

convolvatron 9y ago
what seems to be lacking from these discussions is the notino that forethought and consideration can win the day. if there is some brain fart, and its tracked down and fixed, the value of encoding that in a unit to be run until heat death. as you say, exactly zero. but if there is a problem area that keeps throwing bugs like a metastatic tumor - say race conditions or memory footprint, or unicode handling. in that case the return on writing tests to try to shake out those areas is pretty high. I think the real failure here is to try to substitute informed judgement calls and meaningful discussion with simple checklists of dos and donts.
- koonsolo 9y ago
  > In that case the return on writing tests to try to shake out those areas is pretty high. I agree, but in that case you shouldn't test for that one specific case that failed, but try to make a test that handles similar issues, or tries catch bugs of the same kind. Of course it depends how far you can push this.
hamstercat 9y ago
Doesn't that just mean that the tests were effective? The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine? I personally believe that this advice is one of the most important one actually for a very specific reason: it leaves a very bad impression to client when a bug keep happening again and again after being fixed. Unfortunately I've lived this experience when I was just starting out in my career on a similar project (around 45-50 developers) with basically no tests at all. It wasn't fun explaining to the client, even if they were internal to the company, that the bug we fixed last month had to be fixed again.
- coldtea 9y ago
  >Doesn't that just mean that the tests were effective? Only in the sense that: - What's this? - A device that keeps tigers away. - But there are no tigers in Los Angeles! - See how effective it is? >The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine? No, he means that they had code to test for the presence of fixed bugs, but nobody reintroduced said bugs and triggered that bug catching test code ever. So even if they didn't have the code, the end result would have been the same.
  - closeparen 9y ago
    How would you know? If code is failing tests, it doesn’t leave my laptop.
    - coldtea 9y ago
      Because parent said they tracked the test suites. Perhaps the tests don't run on the devs laptop, but on an integration system (we have such a setup). (And of course you can make test suites report failures centrally, whether they tests run on a laptop or not).
      - closeparen 9y ago
        They said they tracked bugs. A test failure during development doesn’t generally go in the bug tracker.
    - koonsolo 9y ago
      We tracked all bugs in a custom made bug tracker: - Unit/Integration tests - Failures reported by QA - Bugs in production Integration tests were really heavy, and ran multiprocess on a server. So locally we would run the unit tests, and relevant integration tests. The rest was tested on the integration server, which automatically created bugreports when something failed.
  - ithkuil 9y ago
jbreiding 9y ago
While I can agree on the stated reason being not perfect. Production bugs converted to tests do help validate the bug is fixed for ci/cd and understanding for the necessary rca.
Joeri 9y ago
Bugs seek each other’s company. Bad code has many bugs. Therefore the best place to add tests is the code where bugs are found. It’s not quite the same though as adding a test to verify a bugfix.
- jeremyjh 9y ago
  Exactly and often times when I write tests for a bug fix, I'm exposed to thoughts of other cases that were also missing, and write tests for those too. Like everything to do with software testing, for it to be effective you have to use your brain and not blindly follow some prescription you read on the internet.
jpz 9y ago
The dog that didn't bark. Perhaps that's because people aren't committing code which regresses these cases, due to the presence of pre-checkin unit tests?
- koonsolo 9y ago
  There were very little unit tests on this project, and only written for new code, never for bugfixes.
corpMaverick 9y ago
In addition to other's comments. Writing a specific unit test helps you test that the bug exists and after it was fixed it helps you test that the bug was fixed. Something that you have to do anyway.
- koonsolo 9y ago
  Most things in life are a trade-off. If there is no trade-off, everyone would pick the winner. Manually testing your bugs is still cheaper in time and effort than writing unit tests. If it was not, everyone would write unit tests. Unit tests need to be able to fit into your codebase, sometimes contain bugs too, need to be maintained, slows down your release cycle, etc. I'm not claiming they are bad, I'm claiming it's a trade-off that you make in favor of quality, stability, etc. Unit tests also need to be produced by expensive, probably scarce programmers. Sometimes it's better to hire 'cheap' testers that can script tests into your framework. That's why I like to talk about return on investment. If you put in the extra effort of writing a test, you expect this to pay off in the long run. Nothing is cheaper than not testing your bug after you fixed it. But the chance that a lot of time will be wasted afterwards because it wasn't actually fixed is huge. So basically you have to find the most optimal way to spend your effort. And like I said previously, our statistics showed that writing unit tests on found bugs wasn't the optimal way to spend our programmers effort.

convolvatron 9y ago

what seems to be lacking from these discussions is the notino that forethought and consideration can win the day. if there is some brain fart, and its tracked down and fixed, the value of encoding that in a unit to be run until heat death. as you say, exactly zero. but if there is a problem area that keeps throwing bugs like a metastatic tumor - say race conditions or memory footprint, or unicode handling. in that case the return on writing tests to try to shake out those areas is pretty high. I think the real failure here is to try to substitute informed judgement calls and meaningful discussion with simple checklists of dos and donts.

koonsolo 9y ago
> In that case the return on writing tests to try to shake out those areas is pretty high. I agree, but in that case you shouldn't test for that one specific case that failed, but try to make a test that handles similar issues, or tries catch bugs of the same kind. Of course it depends how far you can push this.

hamstercat 9y ago

Doesn't that just mean that the tests were effective? The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine? I personally believe that this advice is one of the most important one actually for a very specific reason: it leaves a very bad impression to client when a bug keep happening again and again after being fixed. Unfortunately I've lived this experience when I was just starting out in my career on a similar project (around 45-50 developers) with basically no tests at all. It wasn't fun explaining to the client, even if they were internal to the company, that the bug we fixed last month had to be fixed again.

coldtea 9y ago
>Doesn't that just mean that the tests were effective? Only in the sense that: - What's this? - A device that keeps tigers away. - But there are no tigers in Los Angeles! - See how effective it is? >The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine? No, he means that they had code to test for the presence of fixed bugs, but nobody reintroduced said bugs and triggered that bug catching test code ever. So even if they didn't have the code, the end result would have been the same.
- closeparen 9y ago
  How would you know? If code is failing tests, it doesn’t leave my laptop.
  - coldtea 9y ago
    Because parent said they tracked the test suites. Perhaps the tests don't run on the devs laptop, but on an integration system (we have such a setup). (And of course you can make test suites report failures centrally, whether they tests run on a laptop or not).
    - closeparen 9y ago
      They said they tracked bugs. A test failure during development doesn’t generally go in the bug tracker.
  - koonsolo 9y ago
    We tracked all bugs in a custom made bug tracker: - Unit/Integration tests - Failures reported by QA - Bugs in production Integration tests were really heavy, and ran multiprocess on a server. So locally we would run the unit tests, and relevant integration tests. The rest was tested on the integration server, which automatically created bugreports when something failed.
- ithkuil 9y ago

jbreiding 9y ago

While I can agree on the stated reason being not perfect. Production bugs converted to tests do help validate the bug is fixed for ci/cd and understanding for the necessary rca.

Joeri 9y ago

Bugs seek each other’s company. Bad code has many bugs. Therefore the best place to add tests is the code where bugs are found. It’s not quite the same though as adding a test to verify a bugfix.

jeremyjh 9y ago
Exactly and often times when I write tests for a bug fix, I'm exposed to thoughts of other cases that were also missing, and write tests for those too. Like everything to do with software testing, for it to be effective you have to use your brain and not blindly follow some prescription you read on the internet.

jpz 9y ago

The dog that didn't bark. Perhaps that's because people aren't committing code which regresses these cases, due to the presence of pre-checkin unit tests?

koonsolo 9y ago
There were very little unit tests on this project, and only written for new code, never for bugfixes.

corpMaverick 9y ago

In addition to other's comments. Writing a specific unit test helps you test that the bug exists and after it was fixed it helps you test that the bug was fixed. Something that you have to do anyway.

koonsolo 9y ago
Most things in life are a trade-off. If there is no trade-off, everyone would pick the winner. Manually testing your bugs is still cheaper in time and effort than writing unit tests. If it was not, everyone would write unit tests. Unit tests need to be able to fit into your codebase, sometimes contain bugs too, need to be maintained, slows down your release cycle, etc. I'm not claiming they are bad, I'm claiming it's a trade-off that you make in favor of quality, stability, etc. Unit tests also need to be produced by expensive, probably scarce programmers. Sometimes it's better to hire 'cheap' testers that can script tests into your framework. That's why I like to talk about return on investment. If you put in the extra effort of writing a test, you expect this to pay off in the long run. Nothing is cheaper than not testing your bug after you fixed it. But the chance that a lot of time will be wasted afterwards because it wasn't actually fixed is huge. So basically you have to find the most optimal way to spend your effort. And like I said previously, our statistics showed that writing unit tests on found bugs wasn't the optimal way to spend our programmers effort.