5 ms·

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flaw

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work; humans are flawed. The real challenge and benefit is to find the value in their work - find what has lasting value, learn from it, and carry it forward.

Ar-Curunir 10y ago
Exactly this; I find too many HN comments to be critical in a non-constructive manner.
- smallnamespace 10y ago
  The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.
  - vacri 10y ago
    How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done. So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group? Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.
    - smallnamespace 10y ago
      This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.
      - vacri 10y ago
        Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
        smallnamespace 10y ago
        I completely agree with you. Pointing out a study's flaws is indeed part of the process of 'caveat[ing] the knowledge gained from that study'.
    - flukus 10y ago
      Maybe we have to accept that it's untestable. Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.
    - icebraining 10y ago
      How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? Hackatons seem to manage. Why not set one up?
      - vacri 10y ago
        Hackathons wouldn't satisfy the OP's requirements for complexity, nor demographics - you'd be looking at a self-selecting group of highly motivated people, skewing young, who would come together hackathon style. There aren't going to be many thirty- or forty-something coders with young families spending a weekend (to work on the same set problems as everyone else) at the hackathon, yet there are plenty of those in industry.
unclebucknasty 10y ago
Indeed. It is predictable and--as a discussion point--stifling. It's far more interesting to discuss the actual merits than to dismiss the subject out of hand.
ericdykstra 10y ago
The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc. This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise." I'm guessing the only reason the story was upvoted at all in the first place is because some people who agree with the title clicked the up arrow without looking at the article.
- tbrownaw 10y ago
  the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely useless. It actually feels quite similar to the claims I've seen from practitioners that reikei, faith healing, etc aren't complete bunk.
  - bjelkeman-again 10y ago
    > provide no additional benefit over anything else that gets the tests written. Isn't this enough though? The tests gets written, which probably was the main point in the first place?
    - zorked 10y ago
      You can also add a coverage tool to your CI and get the same result (tests get written) without any of the ideology (TDD fairies sprinkle unicorn dust everywhere).
      - _ikke_ 10y ago
        Coverage tools are not sufficient for getting good test coverage. One can easily make code 'covered' without having proper tests for them. By writing tests early, you make sure the code you are testing is testable and your knowledge about the code is fresh.
tikhonj 10y ago
I completely disagree. The reason we care about these studies is to make decisions. If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in every one of those blog posts too, but it's by no means useful or valuable. It's a waste of time if not outright counterproductive. If all the studies are flawed, maybe the field is flawed. Maybe the subject is just not susceptible to (cheap) experimental studies. It's hard to trust peer review when there might be problems with the whole field. We may very well be better off relying on experience and opinions because realistic experiments are so far off the mark. It can be even worse: generalizing the results of a bad experiment might even be dangerously wrong. Psychology results from experiments on young, Western college students are a great example—we don't want to make laws or base diagnoses purely on experiments like that because that could actively harm groups that are fundamentally unlike young, Western college students. And all that is pretty much exactly where I see experimental software engineering: the results just don't generalize. And sometimes, I suspect, results generalize in ways that are counterproductive to experienced programmers working on large projects—exactly the people I actually care about. And yet empirical studies (even bad ones) still inherently carry a lot of unearned cachet. The real challenge at the end is overcoming this cachet, not finding value where there just might not be all that much.
- tlarkworthy 10y ago
  It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing every study as irrelevant due to lack of rigor is wasteful.
  - r00fus 10y ago
    Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.
divanvisagie 10y ago
If it is unfair to say in here then you should have waited for a better time to write this comment instead of just venting yourself on the first comment you see like it. This study has the potential to be misused by people to get their senseless argument across and could cause a lot of headache for people in the industry. It's hard enough as it is for a consultant to get people to pay for testable software. That line about your professor is cute, but the issue is the media gobbles up studies and reports on them no matter what the quality leaving large chunks of the population misinformed. So publishing a study that is inaccurate is frankly irresponsible.
flukus 10y ago
We live in a world with too much information. There are probably more words written everyday than we can read in a lifetime. The sooner you can remove worthless information the better.
Strilanc 10y ago
> the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. That's easily the best part of HN and reddit. A great help with the Murray Gell-Mann Amnesia effect. I don't think it's that HN is negative (though HN can certainly be negative), I think most articles really are just kinda crummy. Could be oversimplified, probably overhyped, maybe some wildly skewed sense of "providing both sides", or one of a million other possible problems. The comments will tell you about these problems, and that makes them a better barometer of "is this worth it?" than the title or even the content.
EdiX 10y ago
> In a graduate-level engineering class, the students were making similar statements about all the studies we read Yeah, that's because they are all terrible.
NumberCruncher 10y ago
Scientific methods were invented to minimise the flaws in scientific work. If you ignore these methods and base your study on biased data you better forget calling your results scientific. Fortunately there are folks on HN reminding you on this.
EugeneOZ 10y ago
And words of that professor don't fix anything, really. I absolutely, 101% agree with comment you are replying.
jacquesm 10y ago
Well, he did find value in the work, just not as much as the hyped title would have you believe and I'm perfectly ok with that. The fact that it matches my experience (that TDD has benefits but only for projects of a certain size with programmers of a certain minimum experience level) helps with that, but it is always good to be reminded of the importance of sample size and other priors related to a study that makes a very bold claim.

Ar-Curunir 10y ago

Exactly this; I find too many HN comments to be critical in a non-constructive manner.

smallnamespace 10y ago
The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.
- vacri 10y ago
  How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done. So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group? Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.
  - smallnamespace 10y ago
    This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.
    - vacri 10y ago
      Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
      - smallnamespace 10y ago
        I completely agree with you. Pointing out a study's flaws is indeed part of the process of 'caveat[ing] the knowledge gained from that study'.
  - flukus 10y ago
    Maybe we have to accept that it's untestable. Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.
  - icebraining 10y ago
    How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? Hackatons seem to manage. Why not set one up?
    - vacri 10y ago
      Hackathons wouldn't satisfy the OP's requirements for complexity, nor demographics - you'd be looking at a self-selecting group of highly motivated people, skewing young, who would come together hackathon style. There aren't going to be many thirty- or forty-something coders with young families spending a weekend (to work on the same set problems as everyone else) at the hackathon, yet there are plenty of those in industry.

unclebucknasty 10y ago

Indeed. It is predictable and--as a discussion point--stifling. It's far more interesting to discuss the actual merits than to dismiss the subject out of hand.

ericdykstra 10y ago

The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc. This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise." I'm guessing the only reason the story was upvoted at all in the first place is because some people who agree with the title clicked the up arrow without looking at the article.

tbrownaw 10y ago
the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely useless. It actually feels quite similar to the claims I've seen from practitioners that reikei, faith healing, etc aren't complete bunk.
- bjelkeman-again 10y ago
  > provide no additional benefit over anything else that gets the tests written. Isn't this enough though? The tests gets written, which probably was the main point in the first place?
  - zorked 10y ago
    You can also add a coverage tool to your CI and get the same result (tests get written) without any of the ideology (TDD fairies sprinkle unicorn dust everywhere).
    - _ikke_ 10y ago
      Coverage tools are not sufficient for getting good test coverage. One can easily make code 'covered' without having proper tests for them. By writing tests early, you make sure the code you are testing is testable and your knowledge about the code is fresh.

tikhonj 10y ago

I completely disagree. The reason we care about these studies is to make decisions. If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in every one of those blog posts too, but it's by no means useful or valuable. It's a waste of time if not outright counterproductive. If all the studies are flawed, maybe the field is flawed. Maybe the subject is just not susceptible to (cheap) experimental studies. It's hard to trust peer review when there might be problems with the whole field. We may very well be better off relying on experience and opinions because realistic experiments are so far off the mark. It can be even worse: generalizing the results of a bad experiment might even be dangerously wrong. Psychology results from experiments on young, Western college students are a great example—we don't want to make laws or base diagnoses purely on experiments like that because that could actively harm groups that are fundamentally unlike young, Western college students. And all that is pretty much exactly where I see experimental software engineering: the results just don't generalize. And sometimes, I suspect, results generalize in ways that are counterproductive to experienced programmers working on large projects—exactly the people I actually care about. And yet empirical studies (even bad ones) still inherently carry a lot of unearned cachet. The real challenge at the end is overcoming this cachet, not finding value where there just might not be all that much.

tlarkworthy 10y ago
It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing every study as irrelevant due to lack of rigor is wasteful.
- r00fus 10y ago
  Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.

divanvisagie 10y ago

If it is unfair to say in here then you should have waited for a better time to write this comment instead of just venting yourself on the first comment you see like it. This study has the potential to be misused by people to get their senseless argument across and could cause a lot of headache for people in the industry. It's hard enough as it is for a consultant to get people to pay for testable software. That line about your professor is cute, but the issue is the media gobbles up studies and reports on them no matter what the quality leaving large chunks of the population misinformed. So publishing a study that is inaccurate is frankly irresponsible.

flukus 10y ago

We live in a world with too much information. There are probably more words written everyday than we can read in a lifetime. The sooner you can remove worthless information the better.

Strilanc 10y ago

> the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. That's easily the best part of HN and reddit. A great help with the Murray Gell-Mann Amnesia effect. I don't think it's that HN is negative (though HN can certainly be negative), I think most articles really are just kinda crummy. Could be oversimplified, probably overhyped, maybe some wildly skewed sense of "providing both sides", or one of a million other possible problems. The comments will tell you about these problems, and that makes them a better barometer of "is this worth it?" than the title or even the content.

EdiX 10y ago

> In a graduate-level engineering class, the students were making similar statements about all the studies we read Yeah, that's because they are all terrible.

NumberCruncher 10y ago

Scientific methods were invented to minimise the flaws in scientific work. If you ignore these methods and base your study on biased data you better forget calling your results scientific. Fortunately there are folks on HN reminding you on this.

EugeneOZ 10y ago

And words of that professor don't fix anything, really. I absolutely, 101% agree with comment you are replying.

jacquesm 10y ago

Well, he did find value in the work, just not as much as the hyped title would have you believe and I'm perfectly ok with that. The fact that it matches my experience (that TDD has benefits but only for projects of a certain size with programmers of a certain minimum experience level) helps with that, but it is always good to be reminded of the importance of sample size and other priors related to a study that makes a very bold claim.