3 ms·

> The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. That's where you're wrong... ev

by beambot 10y ago

> The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. That's where you're wrong... even with "careful statistics", the point of sample-based studies is to disprove the null hypothesis with some confidence level. There is no requirement that it be 100% confident. In fact, with p<0.05, you might expect some small fraction of repeat experiments to show no significant effect. (See, it's easy to get this wrong!)

adwn 10y ago
Although I believe that you're technically correct, your objection does not refute jdlshore's central point.
- beambot 10y ago
  Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Results are based on statistical relevance subject to the sampling (and their biases). This study is meaningful in that it provides some limited evidence. It's fine to question biases and confounding factors... but that doesn't change the relevance of their results, merely the scope. In this case, what the researchers actually found: "At <X> confidence interval, TDD doesn't work for white, male graduate students at <Y> University working on <Z> problem. Generalize at your own peril." But that's a shitty headline.
  - adwn 10y ago
    > Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Neither jdlshore nor I were talking about "proof" or "truth" in the mathematical sense. > Results are based on statistical relevance subject to the sampling (and their biases). The point was that this study doesn't have sufficient statistical relevance to give any evidence whether TDD is effective in general. It doesn't matter if this study gives any evidence whether TDD is effective when used by graduate students working on toy problems, because that was not the intention of the study (besides, nobody cares about this highly specific case).
    - beambot 10y ago
      > [...] whether TDD is effective in general. If you read the actual article (even just the abstract), you would realize that evaluating TDD in a general professional settings was neither the goal nor the conclusion of the authors. You are arguing a straw man and hoping to make inferences that are unsupported by the paper's claims. The authors do a good job of scientific communication about their methods, results, and limitations. This is good practice for scientific communications. Don't think it's sufficiently generalizable? Fine, feel free to expand upon their work. That's how science functions. ("truth", "prove", and "rigor" were verbatim, primary components of jdlshore's comment. The two former words have very specific scientific meaning.)

by beambot 10y ago

adwn 10y ago

Although I believe that you're technically correct, your objection does not refute jdlshore's central point.

beambot 10y ago
Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Results are based on statistical relevance subject to the sampling (and their biases). This study is meaningful in that it provides some limited evidence. It's fine to question biases and confounding factors... but that doesn't change the relevance of their results, merely the scope. In this case, what the researchers actually found: "At <X> confidence interval, TDD doesn't work for white, male graduate students at <Y> University working on <Z> problem. Generalize at your own peril." But that's a shitty headline.
- adwn 10y ago
  > Sure it does. In the social sciences (which includes questions about productivity a la TDD), there is no "proof" or "truth" in the mathematical sense. Neither jdlshore nor I were talking about "proof" or "truth" in the mathematical sense. > Results are based on statistical relevance subject to the sampling (and their biases). The point was that this study doesn't have sufficient statistical relevance to give any evidence whether TDD is effective in general. It doesn't matter if this study gives any evidence whether TDD is effective when used by graduate students working on toy problems, because that was not the intention of the study (besides, nobody cares about this highly specific case).
  - beambot 10y ago
    > [...] whether TDD is effective in general. If you read the actual article (even just the abstract), you would realize that evaluating TDD in a general professional settings was neither the goal nor the conclusion of the authors. You are arguing a straw man and hoping to make inferences that are unsupported by the paper's claims. The authors do a good job of scientific communication about their methods, results, and limitations. This is good practice for scientific communications. Don't think it's sufficiently generalizable? Fine, feel free to expand upon their work. That's how science functions. ("truth", "prove", and "rigor" were verbatim, primary components of jdlshore's comment. The two former words have very specific scientific meaning.)