13 ms·

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was t

This study, like most software development studies I've seen, is seriously flawed. It doesn't justify the sensational title here on HN. * The sample size was tiny. (20 students) * The participants were selected by convenience. (They were students in the researcher's class.) * The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two years' experience.) * The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.) Maybe, maybe you could use this to draw conclusions about how TDD affects novices working on simple algorithmic problems. Given the tiny sample size and sampling by convenience, I'm not sure you can even draw that much of a conclusion. But it won't tell you anything about whether or not TDD impacts development time or code quality in real-world development.

wagonman 10y ago
Thank you for looking into the details.
hackuser 10y ago
It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work; humans are flawed. The real challenge and benefit is to find the value in their work - find what has lasting value, learn from it, and carry it forward.
- Ar-Curunir 10y ago
  Exactly this; I find too many HN comments to be critical in a non-constructive manner.
  - smallnamespace 10y ago
    The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.
    - vacri 10y ago
      How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done. So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group? Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.
      - smallnamespace 10y ago
        This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.
        vacri 10y ago
        Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
        smallnamespace 10y ago
- unclebucknasty 10y ago
  Indeed. It is predictable and--as a discussion point--stifling. It's far more interesting to discuss the actual merits than to dismiss the subject out of hand.
- ericdykstra 10y ago
  The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc. This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise." I'm guessing the only reason the story was upvoted at all in the first place is because some people who agree with the title clicked the up arrow without looking at the article.
  - tbrownaw 10y ago
    the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely useless. It actually feels quite similar to the claims I've seen from practitioners that reikei, faith healing, etc aren't complete bunk.
    - bjelkeman-again 10y ago
      > provide no additional benefit over anything else that gets the tests written. Isn't this enough though? The tests gets written, which probably was the main point in the first place?
      - zorked 10y ago
        You can also add a coverage tool to your CI and get the same result (tests get written) without any of the ideology (TDD fairies sprinkle unicorn dust everywhere).
        _ikke_ 10y ago
        Coverage tools are not sufficient for getting good test coverage. One can easily make code 'covered' without having proper tests for them. By writing tests early, you make sure the code you are testing is testable and your knowledge about the code is fresh.
- tikhonj 10y ago
  I completely disagree. The reason we care about these studies is to make decisions. If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in every one of those blog posts too, but it's by no means useful or valuable. It's a waste of time if not outright counterproductive. If all the studies are flawed, maybe the field is flawed. Maybe the subject is just not susceptible to (cheap) experimental studies. It's hard to trust peer review when there might be problems with the whole field. We may very well be better off relying on experience and opinions because realistic experiments are so far off the mark. It can be even worse: generalizing the results of a bad experiment might even be dangerously wrong. Psychology results from experiments on young, Western college students are a great example—we don't want to make laws or base diagnoses purely on experiments like that because that could actively harm groups that are fundamentally unlike young, Western college students. And all that is pretty much exactly where I see experimental software engineering: the results just don't generalize. And sometimes, I suspect, results generalize in ways that are counterproductive to experienced programmers working on large projects—exactly the people I actually care about. And yet empirical studies (even bad ones) still inherently carry a lot of unearned cachet. The real challenge at the end is overcoming this cachet, not finding value where there just might not be all that much.
  - tlarkworthy 10y ago
    It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing every study as irrelevant due to lack of rigor is wasteful.
    - r00fus 10y ago
      Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.
- divanvisagie 10y ago
  If it is unfair to say in here then you should have waited for a better time to write this comment instead of just venting yourself on the first comment you see like it. This study has the potential to be misused by people to get their senseless argument across and could cause a lot of headache for people in the industry. It's hard enough as it is for a consultant to get people to pay for testable software. That line about your professor is cute, but the issue is the media gobbles up studies and reports on them no matter what the quality leaving large chunks of the population misinformed. So publishing a study that is inaccurate is frankly irresponsible.
- flukus 10y ago
  We live in a world with too much information. There are probably more words written everyday than we can read in a lifetime. The sooner you can remove worthless information the better.
- Strilanc 10y ago
  > the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. That's easily the best part of HN and reddit. A great help with the Murray Gell-Mann Amnesia effect. I don't think it's that HN is negative (though HN can certainly be negative), I think most articles really are just kinda crummy. Could be oversimplified, probably overhyped, maybe some wildly skewed sense of "providing both sides", or one of a million other possible problems. The comments will tell you about these problems, and that makes them a better barometer of "is this worth it?" than the title or even the content.
- EdiX 10y ago
  > In a graduate-level engineering class, the students were making similar statements about all the studies we read Yeah, that's because they are all terrible.
- NumberCruncher 10y ago
  Scientific methods were invented to minimise the flaws in scientific work. If you ignore these methods and base your study on biased data you better forget calling your results scientific. Fortunately there are folks on HN reminding you on this.
- EugeneOZ 10y ago
  And words of that professor don't fix anything, really. I absolutely, 101% agree with comment you are replying.
- jacquesm 10y ago
  Well, he did find value in the work, just not as much as the hyped title would have you believe and I'm perfectly ok with that. The fact that it matches my experience (that TDD has benefits but only for projects of a certain size with programmers of a certain minimum experience level) helps with that, but it is always good to be reminded of the importance of sample size and other priors related to a study that makes a very bold claim.
mseebach 10y ago
From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.
- ivan_gammel 10y ago
  >or any style of automated tests, really, you don't really have to be driven by them Looks like you misunderstood what TDD is. TDD is not a method of testing, it's the method of design of system architecture, in which you formalize the contracts and design APIs by writing examples of their use in tests. Because of that TDD is applicable on every scale (e.g. even when you design complicated algorithm that can fit in just couple of screens) and because of that TDD must be compared not to other test methods, but to other methods of design. Of course, that comparison will not show the supremacy of TDD for many reasons, like, for example, insufficient expressiveness of testing framework or domain specifics. TDD is useful as one of the many tools we have today, but it shall never be the only tool.
  - spatulon 10y ago
    This benefit of test-driven design is not obvious (at least it wasn't for me) until you really experience it. Several years back I wasn't aware of the concept of dependency injection, but independently discovered it while trying to unit test some error logging functionality. Previously I'd always used a global/singleton logger object. That always felt wrong, but I always assumed it was necessary until writing unit tests forced me to avoid that design. Also note that TDD is not the only way of getting this benefit. IMHO Paul Graham's essay Programming Bottom Up[1] and Casey Muratori's concept of 'compression-oriented programming'[2] both espouse this same idea of improving API design by "writing examples of their use." [1] http://www.paulgraham.com/progbot.html http://www.paulgraham.com/progbot.html [2] https://mollyrocket.com/casey/stream_0019.html https://mollyrocket.com/casey/stream_0019.html
    - ivan_gammel 10y ago
willtim 10y ago
Processes like TDD are most often applied/enforced with inexperienced developers in mind.
- icebraining 10y ago
  I've never seen Kent Beck make that claim, have you?
  - willtim 10y ago
    Has Kent Beck been pushing for TDD at Facebook, or are there too many experienced devs? Experienced devs do what works best for them, which may be TDD but probably in most cases not.
nickpsecurity 10y ago
Whereas the same conditions you listed existed in the link I just found on Cleanroom method from 80's with clear evidence the method worked to produce low defect software. The weaknesses plus resulting quality just added further support. http://infohost.nmt.edu/~al/cseet-paper.html http://infohost.nmt.edu/~al/cseet-paper.html EDIT: This is now it's own submission below if anyone wants to discuss it there. https://news.ycombinator.com/item?id=12741237 https://news.ycombinator.com/item?id=12741237
apatters 10y ago
The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?
- sanderjd 10y ago
  Perhaps oddly, it turns out that presenting no data is better than presenting data with poor foundation. Comments like this get upvoted because many of us appreciate someone pointing out when there's no "there" there before we've spent our own time reading.
- algesten 10y ago
  uhm. lol? are you saying a study can't be criticised by anything less than a counter study?
- hueving 10y ago
  Have you heard the phrase "garbage in, garbage out?"
  - Udik 10y ago
    Referred to the study or to the commenters?
- parfe 10y ago
  >Why do comments like this get upvoted so much? They're dressed up versions of "correlation != causation". Low thought comments appealing to people with not-even stats 101 knowledge. There is no such thing as a perfect study, and a generic version of the OPs comment could be copy and pasted on any study ever. Unless you study the entire population of the planet, you're going to miss subgroups. Unless you study the entire population of the planet, you're going to need some sort of selection criteria. Unless you have infinite funding and time, you're going to need to make trade offs and sacrifices in your experiment design. A study will disclose these shortcomings for readers to balance the significance of results against. Take this complaint from OP: >* The sample size was tiny. (20 students) What sample size would satisfy him? Why is 20 too small? 40? 80? 1037? Is he basing his opinion of a proper sample size on his gut? 20 just doesn't feel right?
simula67 10y ago
One the one hand, your points are valid. But on the other hand, it is frustrating to see that TDD proponents resort solely to rhetoric to sway people.
- icebraining 10y ago
  Nope, there are also studies (done by Microsoft and IBM) indicating that TDD helps: https://www.infoq.com/news/2009/03/TDD-Improves-Quality https://www.infoq.com/news/2009/03/TDD-Improves-Quality
  - thedufer 10y ago
    It isn't terribly clear from the abstract and I'm not going to pay for the paper, but it sounds like this is really a comparison of tests vs no tests. That has nothing to say about the practice of TDD. This is why this newer study is interesting; it compares to TLD, which allows you to say something about TDD rather than just about testing.
  - Silhouette 10y ago
    Careful: sometimes even widely cited studies like Nagappan don't actually show what they are misreported as showing either. For example, if you look at the development processes being compared in the cases examined, often they aren't quite TDD as widely described and advocated.
- Semiapies 10y ago
  Since when is pointing out the serious flaws of a study "rhetoric"? I'm ambivalent on TDD, but the arguments defending this study here have been ridiculous and anti-intellectual.
staticelf 10y ago
Yes it's flawed, but so is development methodologies in general. The truth is that it depends on the team writing the code not how you write it. While consistency and a method may help to organize the group it doesn't really say anything about any quality what so ever.
zorked 10y ago
Criticism to the study goes both ways. It COULD be that with a larger sample size, more meaningful project and more experienced developers the result would be "TDD actively hampers productivity by 70% and leads to more bugs compared to test-last". Being mindful of the conditions of the study is important. Shrugging it off isn't helpful in the least.
kriro 10y ago
Sample size is 21 and they used Kruskal–Wallis (which is parameter free). Might not be the best tests (iirc. it's usually recommended for the exact combination of ordinal+nominal but I'd have to check) but it's suitable enough. Also I don't think convenience sampling is bad in this case (and they clearly indicate they did it etc.) as it allows you to control for TDD experience and the like. I'd even go further and say the typical "lol students as participants" isn't an issue either. "How effective is test-Driven Development. Making Software: What Really Works, and Why We Believe It" by Turhan et al. is a good meta analysis if you want more studies. As an aside I like the fact that this is a replication study and that there have been a couple of these. I mean worst case you can uses theses studies to guestimate better priors for your own Bayesian analysis :)
blazespin 10y ago
The study is huge huge HUGELY flawed. TDD only comes into value when you start having to refactor extremely large projects that you don't understand. You need the tests in order to refactor with confidence.
- rubber_duck 10y ago
  You can have tests without TDD ? TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests. Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the insight gained. Then I spec out the behavior with tests. TDD assumes you have the design/spec right from the start and all you need to do is write the implementation - very little of my work falls in to that category - perhaps it's different for others. One segment where I found TDD useful is writing story level E2E tests that spec out the requirements before the code is written - this is the least ambiguous way to spec the problem I've found - the downside being that the person doing the spec needs to know how to write E2E tests.
  - crdoconnor 10y ago
    >You can have tests without TDD ? I've worked on several projects without this and the tests done without TDD tend to be of higher quality. I noticed a common anti-pattern of "write the code, run the code, copy the output of the code, paste it into a test and write an assert to check that the output was precisely what came out". This was brittle, it killed the self-documenting aspect of the tests and it often concealed various bugs. As soon as the same team members started writing the test first, it stopped happening. I also personally felt better about doing this since it made it easier to correct API design mistakes before implementing the code and baking them in.
    - rubber_duck 10y ago
      >I also personally felt better about doing this since it made it easier to correct API design mistakes before implementing the code and baking them in. How do you know you made API mistakes from writing unit tests ? API mistakes become apparent when you integrate stuff and use the API in conjunction with other things (using API in isolated scenarios like unit tests is not really insightful you can figure out those flaws just by looking at the API). This is my primary criticism of TDD - you write tests around the initial bad API and you end up keeping that bad design because you already spent so much time testing that. Implementation errors caught by unit tests are cheaper to fix afterwards then design errors. My approach is hack together a POC -> refactor that and write tests. If you discard the POC then TDD makes sense to me.
ckastner 10y ago
> The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.) The top-most comment on that page emphasizes this point: Here's my hypothesis, based on personal experience: the benefits of TDD begin to manifest when they are applied at scale. During design and development, if a single developer can plausibly understand an entire system in their head, the benefits of TDD (and, in fact, unit testing) are negligible. However, there's a non-linear benefit as systems become larger, particularly in the diagnosis of large and complex system failures.
- awjr 10y ago
  This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible. However, the moment it becomes more than just a one-off project this debt becomes a problem.
  - hirzel 10y ago
    Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process? In terms of technical debt, does it matter when the tests are written?
    - noxToken 10y ago
      Yes. Some people do not know how to write extensible software. They solve the issue at hand without any regard to big picture, so when something needs to be added (like test) the code needs to be heavily refactored to accommodate the change. As others have stated, this usually(!) isn't a big deal when the project is small with a single contributor. However, writing the tests last without any forethought to what the tests are or how they might be implemented will incur debt when the code needs to be refactored. Conversely, extensible code can have the tests written before or after without much change to the amount of technical debt. The code is written to adapt to change, so tacking on robust tests wouldn't require too much technical debt. Note that there is a balance. If you know your goals and have no reasonable expectation for them to change, then making everything extensible is unnecessary overhead. You can also have a mix of abstract functions that are extensible alongside immutable functions that handle something specific. It's all about what works best for your project.
zamalek 10y ago
* The code-base was new, simple, and subsequently thrown away. * There is no way to scientifically measure absolute code quality. There are numerous subjective indicators (tabs vs. spaces) and the objective indicators only really become apparent once you are dealing with more than one component/service.
ChoHag 10y ago
Promotion of schemes akin to TDD tend to build their foundations on even smaller sample sizes, usually of those close in some fashion to the new wheel's reinventor; the naïve solving text-book-example problems. But maybe I missed something.
- shawn-butler 10y ago
  You didn't. The top comment is a book writer on TDD who is shocked and downright appalled TDD provides absolutely no value over TLD.
sytelus 10y ago
Most importantly, their solutions didn't need to actually work in the real world. If that wasn't the case, one would find that there are often massive rewrites (not "refactors") when software is in embryonic form and/or users are giving it a try. A major advantage and disadvantage with TDD approach is that you pay the price upfront to avoid more charges letter. But if your features, business model and code is still haven't taken solid form then your upfront payment gets wasted.
varjag 10y ago
But you have same variation in professional environment as well. It well could be that TDD is not that much of a gain if it gets lost in the noise of confounding factors that easily.
StreamBright 10y ago
I agree with you on this. The study is irrelevant from the statistical point of view. However, there are other considerations with TDD, probably on a more theoretical level that: * There is no other major engineering branch that uses testing this way in production (would you bang a car against the guard rail to see if it works?) * There is one thing common for sure with all of the bugs you catch in production, they all passed the unit tests I think testing has it benefits but driving(!) your software development efforts is probably too much, at least for me.
- regularfry 10y ago
  To your first point, the usual response is that there's no other major engineering branch which demands quite such extensive changes be possible after you've broken ground. To extend your analogy a little, nobody takes a car production line and expects the people running it to be able to switch to helicopters without starting over.
  - StreamBright 10y ago
    I agree, and it does not happen to my projects either. We agree in advance on the set of features for a certain version and we deliver that. If you doing much car building when you actually need a helicopter than there is something wrong.
- Beltiras 10y ago
  If you could bang a car against a guard rail without incurring the cost of a car, you would.
  - drostie 10y ago
    I mean, you do incur a cost for programming with TDD though. The benefits may repay the cost in the end, but the cost is still there. And for that matter cars do undergo destructive testing, just not TDD--and the problem there is presumably that the car benefits from taking a holistic component-driven approach, "here is what I want the car overall to look like, and what components I want to fit into the space inside, now how can I reduce the rollover risk?" rather than "here are some basic expectations of a car, I expect it to be able to move forward and hold a passenger, let's make the smallest thing which causes that to happen. OK now let's add a single passenger, and let's test that bugs aren't flying into your face at highway speeds so that we remember to add a windshield eventually. (1000 steps later) Crap now we're testing the ability to shift into reverse and none of my homespun bare-minimum engine design is going to work there." TDD probably pays its best rewards with relatively simple well-defined projects, and needs to be deployed with a modularization strategy. Possibly the wishful-thinking strategy of top-down design would be a natural complement, since when you wishfully think of a function you can hopefully quickly write a test or two for it.
dang 10y ago
> It doesn't justify the sensational title here on HN. We've changed the title along with the url. Please see https://news.ycombinator.com/item?id=12742120 https://news.ycombinator.com/item?id=12742120. The submitted title was “TDD has little or no impact on development time or code quality” (including the quotes).
bnegreve 10y ago
To be fair, the abtract says "The results failed to support the claims." and the conclusion says "We recommend future studies to survey the tasks used in experiments evaluating TDD, and assess them with respect to the treatments.". This is not a claim that TDD is useless.
cheez 10y ago
Students are not even a good sample for the efficacy of TDD.
omouse 10y ago
At the same time we should still praise them for trying to replicate a study, which doesn't happen nearly often enough in the computer science field (aside from algorithms and data structures studies).

wagonman 10y ago

Thank you for looking into the details.

hackuser 10y ago

It may be unfair to say this in response to the parent comment, but the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. Occasionally it's true, but the noise drowns out the signal. In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work; humans are flawed. The real challenge and benefit is to find the value in their work - find what has lasting value, learn from it, and carry it forward.

Ar-Curunir 10y ago
Exactly this; I find too many HN comments to be critical in a non-constructive manner.
- smallnamespace 10y ago
  The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.
  - vacri 10y ago
    How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done. So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group? Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.
    - smallnamespace 10y ago
      This may sound harsh, but taking the researcher's difficulties into account is not our responsibility. The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.
      - vacri 10y ago
        Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
        smallnamespace 10y ago
unclebucknasty 10y ago
Indeed. It is predictable and--as a discussion point--stifling. It's far more interesting to discuss the actual merits than to dismiss the subject out of hand.
ericdykstra 10y ago
The conversation around TDD is tired, and the conclusion is always the same: "it depends." It depends on the person writing the code, the type of problem they're tackling, the language they're using, the needs of the business, etc. This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise." I'm guessing the only reason the story was upvoted at all in the first place is because some people who agree with the title clicked the up arrow without looking at the article.
- tbrownaw 10y ago
  the conclusion is always the same: "it depends." Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written. The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely useless. It actually feels quite similar to the claims I've seen from practitioners that reikei, faith healing, etc aren't complete bunk.
  - bjelkeman-again 10y ago
    > provide no additional benefit over anything else that gets the tests written. Isn't this enough though? The tests gets written, which probably was the main point in the first place?
    - zorked 10y ago
      You can also add a coverage tool to your CI and get the same result (tests get written) without any of the ideology (TDD fairies sprinkle unicorn dust everywhere).
      - _ikke_ 10y ago
        Coverage tools are not sufficient for getting good test coverage. One can easily make code 'covered' without having proper tests for them. By writing tests early, you make sure the code you are testing is testable and your knowledge about the code is fresh.
tikhonj 10y ago
I completely disagree. The reason we care about these studies is to make decisions. If a study is flawed... maybe it's flawed. Maybe it shouldn't be relied upon any more than some random opinion blog post. (Maybe even less!) But just being "a study" lends it a lot of weight compared to that same random blog post, so it absolutely should be held to strong scrutiny. It might be a "real challenge" to find value in every one of those blog posts too, but it's by no means useful or valuable. It's a waste of time if not outright counterproductive. If all the studies are flawed, maybe the field is flawed. Maybe the subject is just not susceptible to (cheap) experimental studies. It's hard to trust peer review when there might be problems with the whole field. We may very well be better off relying on experience and opinions because realistic experiments are so far off the mark. It can be even worse: generalizing the results of a bad experiment might even be dangerously wrong. Psychology results from experiments on young, Western college students are a great example—we don't want to make laws or base diagnoses purely on experiments like that because that could actively harm groups that are fundamentally unlike young, Western college students. And all that is pretty much exactly where I see experimental software engineering: the results just don't generalize. And sometimes, I suspect, results generalize in ways that are counterproductive to experienced programmers working on large projects—exactly the people I actually care about. And yet empirical studies (even bad ones) still inherently carry a lot of unearned cachet. The real challenge at the end is overcoming this cachet, not finding value where there just might not be all that much.
- tlarkworthy 10y ago
  It's a total waste of scientific resources to insist every study requires an army of test subjects, and the top statisticians of the times to apply the latest modeling, to work blind and replicated independently before publication. There are limited resources to do research. You can get insights into how the world works much cheaper if you use critical thinking and facts outside the controls. Casually dismissing every study as irrelevant due to lack of rigor is wasteful.
  - r00fus 10y ago
    Nobody needs an army of test subjects - just statistically valid numbers. But even with 20 test subjects the study fails at trying to make that 20 representative of any meaningful population.
divanvisagie 10y ago
If it is unfair to say in here then you should have waited for a better time to write this comment instead of just venting yourself on the first comment you see like it. This study has the potential to be misused by people to get their senseless argument across and could cause a lot of headache for people in the industry. It's hard enough as it is for a consultant to get people to pay for testable software. That line about your professor is cute, but the issue is the media gobbles up studies and reports on them no matter what the quality leaving large chunks of the population misinformed. So publishing a study that is inaccurate is frankly irresponsible.
flukus 10y ago
We live in a world with too much information. There are probably more words written everyday than we can read in a lifetime. The sooner you can remove worthless information the better.
Strilanc 10y ago
> the great majority of HN discussions start with a comment like this one: It's seriously flawed, etc. That's easily the best part of HN and reddit. A great help with the Murray Gell-Mann Amnesia effect. I don't think it's that HN is negative (though HN can certainly be negative), I think most articles really are just kinda crummy. Could be oversimplified, probably overhyped, maybe some wildly skewed sense of "providing both sides", or one of a million other possible problems. The comments will tell you about these problems, and that makes them a better barometer of "is this worth it?" than the title or even the content.
EdiX 10y ago
> In a graduate-level engineering class, the students were making similar statements about all the studies we read Yeah, that's because they are all terrible.
NumberCruncher 10y ago
Scientific methods were invented to minimise the flaws in scientific work. If you ignore these methods and base your study on biased data you better forget calling your results scientific. Fortunately there are folks on HN reminding you on this.
EugeneOZ 10y ago
And words of that professor don't fix anything, really. I absolutely, 101% agree with comment you are replying.
jacquesm 10y ago
Well, he did find value in the work, just not as much as the hyped title would have you believe and I'm perfectly ok with that. The fact that it matches my experience (that TDD has benefits but only for projects of a certain size with programmers of a certain minimum experience level) helps with that, but it is always good to be reminded of the importance of sample size and other priors related to a study that makes a very bold claim.

mseebach 10y ago

From experience, the value of TDD (or any style of automated tests, really, you don't really have to be driven by them) only really kicks in when the code is of a certain size, complexity and age. They are much closer to a tax in the early phases when you can reasonably keep the full scope readily in your mind.

ivan_gammel 10y ago
>or any style of automated tests, really, you don't really have to be driven by them Looks like you misunderstood what TDD is. TDD is not a method of testing, it's the method of design of system architecture, in which you formalize the contracts and design APIs by writing examples of their use in tests. Because of that TDD is applicable on every scale (e.g. even when you design complicated algorithm that can fit in just couple of screens) and because of that TDD must be compared not to other test methods, but to other methods of design. Of course, that comparison will not show the supremacy of TDD for many reasons, like, for example, insufficient expressiveness of testing framework or domain specifics. TDD is useful as one of the many tools we have today, but it shall never be the only tool.
- spatulon 10y ago
  This benefit of test-driven design is not obvious (at least it wasn't for me) until you really experience it. Several years back I wasn't aware of the concept of dependency injection, but independently discovered it while trying to unit test some error logging functionality. Previously I'd always used a global/singleton logger object. That always felt wrong, but I always assumed it was necessary until writing unit tests forced me to avoid that design. Also note that TDD is not the only way of getting this benefit. IMHO Paul Graham's essay Programming Bottom Up[1] and Casey Muratori's concept of 'compression-oriented programming'[2] both espouse this same idea of improving API design by "writing examples of their use." [1] http://www.paulgraham.com/progbot.html http://www.paulgraham.com/progbot.html [2] https://mollyrocket.com/casey/stream_0019.html https://mollyrocket.com/casey/stream_0019.html
  - ivan_gammel 10y ago

willtim 10y ago

Processes like TDD are most often applied/enforced with inexperienced developers in mind.

icebraining 10y ago
I've never seen Kent Beck make that claim, have you?
- willtim 10y ago
  Has Kent Beck been pushing for TDD at Facebook, or are there too many experienced devs? Experienced devs do what works best for them, which may be TDD but probably in most cases not.

nickpsecurity 10y ago

Whereas the same conditions you listed existed in the link I just found on Cleanroom method from 80's with clear evidence the method worked to produce low defect software. The weaknesses plus resulting quality just added further support. http://infohost.nmt.edu/~al/cseet-paper.html http://infohost.nmt.edu/~al/cseet-paper.html EDIT: This is now it's own submission below if anyone wants to discuss it there. https://news.ycombinator.com/item?id=12741237 https://news.ycombinator.com/item?id=12741237

apatters 10y ago

The study presents imperfect data. You have presented no data. Which is more credible? Why do comments like this get upvoted so much?

sanderjd 10y ago
Perhaps oddly, it turns out that presenting no data is better than presenting data with poor foundation. Comments like this get upvoted because many of us appreciate someone pointing out when there's no "there" there before we've spent our own time reading.
algesten 10y ago
uhm. lol? are you saying a study can't be criticised by anything less than a counter study?
hueving 10y ago
Have you heard the phrase "garbage in, garbage out?"
- Udik 10y ago
  Referred to the study or to the commenters?
parfe 10y ago
>Why do comments like this get upvoted so much? They're dressed up versions of "correlation != causation". Low thought comments appealing to people with not-even stats 101 knowledge. There is no such thing as a perfect study, and a generic version of the OPs comment could be copy and pasted on any study ever. Unless you study the entire population of the planet, you're going to miss subgroups. Unless you study the entire population of the planet, you're going to need some sort of selection criteria. Unless you have infinite funding and time, you're going to need to make trade offs and sacrifices in your experiment design. A study will disclose these shortcomings for readers to balance the significance of results against. Take this complaint from OP: >* The sample size was tiny. (20 students) What sample size would satisfy him? Why is 20 too small? 40? 80? 1037? Is he basing his opinion of a proper sample size on his gut? 20 just doesn't feel right?

simula67 10y ago

One the one hand, your points are valid. But on the other hand, it is frustrating to see that TDD proponents resort solely to rhetoric to sway people.

icebraining 10y ago
Nope, there are also studies (done by Microsoft and IBM) indicating that TDD helps: https://www.infoq.com/news/2009/03/TDD-Improves-Quality https://www.infoq.com/news/2009/03/TDD-Improves-Quality
- thedufer 10y ago
  It isn't terribly clear from the abstract and I'm not going to pay for the paper, but it sounds like this is really a comparison of tests vs no tests. That has nothing to say about the practice of TDD. This is why this newer study is interesting; it compares to TLD, which allows you to say something about TDD rather than just about testing.
- Silhouette 10y ago
  Careful: sometimes even widely cited studies like Nagappan don't actually show what they are misreported as showing either. For example, if you look at the development processes being compared in the cases examined, often they aren't quite TDD as widely described and advocated.
Semiapies 10y ago
Since when is pointing out the serious flaws of a study "rhetoric"? I'm ambivalent on TDD, but the arguments defending this study here have been ridiculous and anti-intellectual.

staticelf 10y ago

Yes it's flawed, but so is development methodologies in general. The truth is that it depends on the team writing the code not how you write it. While consistency and a method may help to organize the group it doesn't really say anything about any quality what so ever.

zorked 10y ago

Criticism to the study goes both ways. It COULD be that with a larger sample size, more meaningful project and more experienced developers the result would be "TDD actively hampers productivity by 70% and leads to more bugs compared to test-last". Being mindful of the conditions of the study is important. Shrugging it off isn't helpful in the least.

kriro 10y ago

Sample size is 21 and they used Kruskal–Wallis (which is parameter free). Might not be the best tests (iirc. it's usually recommended for the exact combination of ordinal+nominal but I'd have to check) but it's suitable enough. Also I don't think convenience sampling is bad in this case (and they clearly indicate they did it etc.) as it allows you to control for TDD experience and the like. I'd even go further and say the typical "lol students as participants" isn't an issue either. "How effective is test-Driven Development. Making Software: What Really Works, and Why We Believe It" by Turhan et al. is a good meta analysis if you want more studies. As an aside I like the fact that this is a replication study and that there have been a couple of these. I mean worst case you can uses theses studies to guestimate better priors for your own Bayesian analysis :)

blazespin 10y ago

The study is huge huge HUGELY flawed. TDD only comes into value when you start having to refactor extremely large projects that you don't understand. You need the tests in order to refactor with confidence.

rubber_duck 10y ago
You can have tests without TDD ? TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests. Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the insight gained. Then I spec out the behavior with tests. TDD assumes you have the design/spec right from the start and all you need to do is write the implementation - very little of my work falls in to that category - perhaps it's different for others. One segment where I found TDD useful is writing story level E2E tests that spec out the requirements before the code is written - this is the least ambiguous way to spec the problem I've found - the downside being that the person doing the spec needs to know how to write E2E tests.
- crdoconnor 10y ago
  >You can have tests without TDD ? I've worked on several projects without this and the tests done without TDD tend to be of higher quality. I noticed a common anti-pattern of "write the code, run the code, copy the output of the code, paste it into a test and write an assert to check that the output was precisely what came out". This was brittle, it killed the self-documenting aspect of the tests and it often concealed various bugs. As soon as the same team members started writing the test first, it stopped happening. I also personally felt better about doing this since it made it easier to correct API design mistakes before implementing the code and baking them in.
  - rubber_duck 10y ago
    >I also personally felt better about doing this since it made it easier to correct API design mistakes before implementing the code and baking them in. How do you know you made API mistakes from writing unit tests ? API mistakes become apparent when you integrate stuff and use the API in conjunction with other things (using API in isolated scenarios like unit tests is not really insightful you can figure out those flaws just by looking at the API). This is my primary criticism of TDD - you write tests around the initial bad API and you end up keeping that bad design because you already spent so much time testing that. Implementation errors caught by unit tests are cheaper to fix afterwards then design errors. My approach is hack together a POC -> refactor that and write tests. If you discard the POC then TDD makes sense to me.

ckastner 10y ago

> The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.) The top-most comment on that page emphasizes this point: Here's my hypothesis, based on personal experience: the benefits of TDD begin to manifest when they are applied at scale. During design and development, if a single developer can plausibly understand an entire system in their head, the benefits of TDD (and, in fact, unit testing) are negligible. However, there's a non-linear benefit as systems become larger, particularly in the diagnosis of large and complex system failures.

awjr 10y ago
This is absolutely key. TDD within small projects has little value. It's a technical debt people are happy to carry as the debt is negligible. However, the moment it becomes more than just a one-off project this debt becomes a problem.
- hirzel 10y ago
  Although, isn't this more true for writing tests at all versus writing tests first or last in the coding process? In terms of technical debt, does it matter when the tests are written?
  - noxToken 10y ago
    Yes. Some people do not know how to write extensible software. They solve the issue at hand without any regard to big picture, so when something needs to be added (like test) the code needs to be heavily refactored to accommodate the change. As others have stated, this usually(!) isn't a big deal when the project is small with a single contributor. However, writing the tests last without any forethought to what the tests are or how they might be implemented will incur debt when the code needs to be refactored. Conversely, extensible code can have the tests written before or after without much change to the amount of technical debt. The code is written to adapt to change, so tacking on robust tests wouldn't require too much technical debt. Note that there is a balance. If you know your goals and have no reasonable expectation for them to change, then making everything extensible is unnecessary overhead. You can also have a mix of abstract functions that are extensible alongside immutable functions that handle something specific. It's all about what works best for your project.

zamalek 10y ago

* The code-base was new, simple, and subsequently thrown away. * There is no way to scientifically measure absolute code quality. There are numerous subjective indicators (tabs vs. spaces) and the objective indicators only really become apparent once you are dealing with more than one component/service.

ChoHag 10y ago

Promotion of schemes akin to TDD tend to build their foundations on even smaller sample sizes, usually of those close in some fashion to the new wheel's reinventor; the naïve solving text-book-example problems. But maybe I missed something.

shawn-butler 10y ago
You didn't. The top comment is a book writer on TDD who is shocked and downright appalled TDD provides absolutely no value over TLD.

sytelus 10y ago

Most importantly, their solutions didn't need to actually work in the real world. If that wasn't the case, one would find that there are often massive rewrites (not "refactors") when software is in embryonic form and/or users are giving it a try. A major advantage and disadvantage with TDD approach is that you pay the price upfront to avoid more charges letter. But if your features, business model and code is still haven't taken solid form then your upfront payment gets wasted.

varjag 10y ago

But you have same variation in professional environment as well. It well could be that TDD is not that much of a gain if it gets lost in the noise of confounding factors that easily.

StreamBright 10y ago

I agree with you on this. The study is irrelevant from the statistical point of view. However, there are other considerations with TDD, probably on a more theoretical level that: * There is no other major engineering branch that uses testing this way in production (would you bang a car against the guard rail to see if it works?) * There is one thing common for sure with all of the bugs you catch in production, they all passed the unit tests I think testing has it benefits but driving(!) your software development efforts is probably too much, at least for me.

regularfry 10y ago
To your first point, the usual response is that there's no other major engineering branch which demands quite such extensive changes be possible after you've broken ground. To extend your analogy a little, nobody takes a car production line and expects the people running it to be able to switch to helicopters without starting over.
- StreamBright 10y ago
  I agree, and it does not happen to my projects either. We agree in advance on the set of features for a certain version and we deliver that. If you doing much car building when you actually need a helicopter than there is something wrong.
Beltiras 10y ago
If you could bang a car against a guard rail without incurring the cost of a car, you would.
- drostie 10y ago
  I mean, you do incur a cost for programming with TDD though. The benefits may repay the cost in the end, but the cost is still there. And for that matter cars do undergo destructive testing, just not TDD--and the problem there is presumably that the car benefits from taking a holistic component-driven approach, "here is what I want the car overall to look like, and what components I want to fit into the space inside, now how can I reduce the rollover risk?" rather than "here are some basic expectations of a car, I expect it to be able to move forward and hold a passenger, let's make the smallest thing which causes that to happen. OK now let's add a single passenger, and let's test that bugs aren't flying into your face at highway speeds so that we remember to add a windshield eventually. (1000 steps later) Crap now we're testing the ability to shift into reverse and none of my homespun bare-minimum engine design is going to work there." TDD probably pays its best rewards with relatively simple well-defined projects, and needs to be deployed with a modularization strategy. Possibly the wishful-thinking strategy of top-down design would be a natural complement, since when you wishfully think of a function you can hopefully quickly write a test or two for it.

dang 10y ago

> It doesn't justify the sensational title here on HN. We've changed the title along with the url. Please see https://news.ycombinator.com/item?id=12742120 https://news.ycombinator.com/item?id=12742120. The submitted title was “TDD has little or no impact on development time or code quality” (including the quotes).

bnegreve 10y ago

To be fair, the abtract says "The results failed to support the claims." and the conclusion says "We recommend future studies to survey the tasks used in experiments evaluating TDD, and assess them with respect to the treatments.". This is not a claim that TDD is useless.

cheez 10y ago

Students are not even a good sample for the efficacy of TDD.

omouse 10y ago

At the same time we should still praise them for trying to replicate a study, which doesn't happen nearly often enough in the computer science field (aside from algorithms and data structures studies).