3 ms·

I'd venture to guess nearly all developers using unit tests would say they are helpful, otherwise they wouldn't be using them. This isn't meant to be snide. I'm

by jm4 18y ago

I'd venture to guess nearly all developers using unit tests would say they are helpful, otherwise they wouldn't be using them. This isn't meant to be snide. I'm just saying that if you already suspect koolaid then any feedback is basically worthless because the only people responding are the ones who have seen measurable results and idiots who drink koolaid, and there's no way to tell the difference. The feedback from non-testers might be a little more useful, but it's still a little suspect considering it's from people who are confident more testing would improve their software yet still don't do it. Furthermore, unit tests are only as good as the test cases themselves and a survey can't account for quality of tests. I guess my point is that while surveys can be interesting, at the end of the day they're just a popularity contest and not very good for determining what works and what doesn't.

bhb 18y ago
Exactly. I'd still love to see real study on whether unit testing is worth it (which our survey doesn't even begin to address, since it's very hard to measure). I think some good studies on that would be invaluable, but it's nearly impossible to control for all the variables, so I'm not holding my breath. Failing that, most of the "evidence" one way or the other is based on asking people who test or don't test - and probably most testers would say that testing is great while non-testers would say testing wouldn't catch the kinds of bugs that matter. On the flip side, you have some testers saying it's not valuable (because their test quality is bad) and some non-testers saying they are sure tests would help, but only because they imagine tests to be a silver bullet. In any case, our survey was just supposed to get a rough feel for how many people were doing testing (and what tools they used). What surprised me most was that even small teams (1 and 2 people) often had a test suite. This flew in the face of the anecdotal evidence we had previously gotten from other startups (almost none of ones we have talked to use tests).
- KevinMS 18y ago
  On example of the question would be, "Do your tests ever find bugs, or do they just sit there always working." No koolaid, how about the post from this moron http://kurt.karmalab.org/articles/2008/02/07/rails-app-without-tests-guaranteed-fail http://kurt.karmalab.org/articles/2008/02/07/rails-app-witho... If you look at page 470 of the latest Code Complete, it shows that unit tests have a very low defect detection rate, Cleanroom software engineering has the best record of quality, but there's no buzz for that now in the ruby community, maybe because it doesnt sell books :-P

by jm4 18y ago

bhb 18y ago

Exactly. I'd still love to see real study on whether unit testing is worth it (which our survey doesn't even begin to address, since it's very hard to measure). I think some good studies on that would be invaluable, but it's nearly impossible to control for all the variables, so I'm not holding my breath. Failing that, most of the "evidence" one way or the other is based on asking people who test or don't test - and probably most testers would say that testing is great while non-testers would say testing wouldn't catch the kinds of bugs that matter. On the flip side, you have some testers saying it's not valuable (because their test quality is bad) and some non-testers saying they are sure tests would help, but only because they imagine tests to be a silver bullet. In any case, our survey was just supposed to get a rough feel for how many people were doing testing (and what tools they used). What surprised me most was that even small teams (1 and 2 people) often had a test suite. This flew in the face of the anecdotal evidence we had previously gotten from other startups (almost none of ones we have talked to use tests).

KevinMS 18y ago
On example of the question would be, "Do your tests ever find bugs, or do they just sit there always working." No koolaid, how about the post from this moron http://kurt.karmalab.org/articles/2008/02/07/rails-app-without-tests-guaranteed-fail http://kurt.karmalab.org/articles/2008/02/07/rails-app-witho... If you look at page 470 of the latest Code Complete, it shows that unit tests have a very low defect detection rate, Cleanroom software engineering has the best record of quality, but there's no buzz for that now in the ruby community, maybe because it doesnt sell books :-P