7 ms·

TDD Derangement Syndrome

29 points by gthank 17y ago

DannoHung 17y ago
Anyone else sort of think that the "To TDD or not" argument eclipses the much more important argument of "To have high unit test coverage or not"?
- akeefer 17y ago
  The research I've seen on TDD seems to essentially point to two things: quality and productivity is positively correlated with the number of tests, and people who use TDD tend to write more tests. People who write the same number of tests but don't use TDD get similar benefits to people who do use TDD. So TDD is itself useful primarily in that it ensures that you actually do write the tests. (Unfortunately I don't have any citations handy to back that up; that's just my recollection of what I've read on the subject over the years). There's that ever-elusive "it improves design" argument for TDD, but that one's far harder to prove either way, and I've personally seen it cut both ways: sometimes things end up more nicely decomposed, and sometimes they end up wayyyyy too decomposed.
  - DannoHung 17y ago
    This squares with my experiences. As for the design argument, I think that the thing that TDD actually encourages in terms of design is testability (something that I do think is desirable, in general). If you have high test coverage without using TDD, you've probably either taken testability into consideration before you wrote your code or you made some changes after the fact to accommodate your tests.
    - akeefer 17y ago
      I'd definitely agree that TDD leads to testable code, which is usually a good thing. If you know how to write testable code, it's not as much of a win (in my opinion), since you tend to write with testing in mind. If you don't have experience with unit testing, though, doing strict TDD for a while can really help you learn what sort of constructs make for testable code. The places I've seen TDD lead to less-than-desirable design is when some larger problem is broken into such tiny pieces with so many interfaces that it becomes hard to read through the code and follow the flow of control. That's certainly testable, but often times I personally get as much mileage out of writing tests one level up, against some higher-level abstraction or API that those other things are all components of, and not being so religious about breaking up all the little parts. In my experience those slightly-higher-level tests (they're not even "integration" tests, since they're still strictly against one logical component) tend to provide the same amount of value in terms of preventing regressions while requiring fewer compromises to make the little parts testable and giving more flexibility as to future refactoring. Essentially, you're treating a whole bunch of classes/parts as internal implementation details.
      - DannoHung 17y ago
        I think we see eye to eye here. The sort of "one level up" thing is what initially attracted me to spec testing because that seemed to not only give you tools for performing your tests, but also helped direct you towards what would be useful to test: the behaviors of the code that you actually care about. So when I use spec testing tools, I tend not to think about making sure that the low level function traverses the directories in a particular way, but rather that all of the directories have been renamed (for example).
      - steveklabnik 17y ago
        > I personally get as much mileage out of writing tests one level up This is why I prefer the rebranded "BDD" as opposed to the original "TDD". I find that the perspective that BDD offers makes me write much more useful tests, since the ultimate reason we write objects in the first place is to use them, right? I don't care about how it works inside, I care about how it gets used. There's nothing technically different between BDD and TDD, it's a matter of philosophy, and I find BDD's take on things to be better.
silentbicycle 17y ago
Does it seem weird to anybody else that about half the list of prominent projects using TDD he listed are testing tools? That's like saying type systems are good because they help you write sound proofs about type systems. I know they can be really useful, but it's an incredibly circular argument.
- idlewords 17y ago
  It seems really weird. I've been trying to get an answer to the question "what popular apps have actually been built using TDD?" for a while, without success. All of the entries on his list are either libraries, programming languages, or testing tools and frameworks. Can anyone name actual apps that TDD has blessed us with?
  - silentbicycle 17y ago
    More importantly, that were done with TDD as opposed to unit testing, testing with a repl, etc.? A lot of claims about TDD's benefits seem to take credit for testing in general. (Automated testing is a clear win, IMHO.) Insisting on writing tests upfront does mean that you test consistently, but then it's hardly the primary cause. I'm sure it's a highly useful tool in some cases, but the people pushing it uncritically make it harder to tell when it's actually a good fit. In my experience, it's very useful for safely making changes in a legacy code, but I suspect it can encourage over-engineering surface details or getting caught in local maxima when used for initial design. (I usually just design with a REPL or notepad/whiteboard.)
  - chadaustin 17y ago
    Much of IMVU's client and website (tens of millions of customers) were written with TDD. I've long been a proponent of TDD once you pass the prototyping stage and you need to write a production-ready component with [most of] the edge cases figured out. The "red green refactor" loop is more like: 1) What's the next problem this software needs to solve? 2) Write a failing test that demonstrates the missing need. 3) Implement said behavior. 4) Refactor until you don't see anything else to refactor. I think step #1 is one of the most important and yet I don't see much discussion of it. Pick the most "key" tests to write first, whether that be your happy path or the functionality that has the most risk. On refactoring... I work with people who TDD all of their work and other who write unit tests afterwards. In my experience, those who test-drive are more comfortable refactoring otherwise. When I was test-driving imvu.task ( http://thespeedbump.livejournal.com/63798.html http://thespeedbump.livejournal.com/63798.html ), I spent a great deal of time just staring at the code after it was written... it's very continuation-heavy code... and suddenly I saw a refactoring that collapsed it by 1/3. We've hardly touched the code since. It solves the right problem with the right amount of code and I think it's only had a couple very obscure race conditions. In summary, TDD is one of my favorite tools for focusing my efforts so we build what we need, but only what we need.
deleted 17y ago
[deleted]
JustAGeek 17y ago
I also think that this list isn't very convincing. What I don't get, if Robert Martin uses TDD, why doesn't he simply list those "real" projects (real as in projects for clients) , he's worked on?
- zacharydanger 17y ago
  How many of his client projects do you expect anyone to have heard of?
  - nickelplate 17y ago
    Probably not many, but I don't think that is necessary. He just has to show us a convincing list of projects (e), and demonstrate how these projects improved with the introduction of TDD, WITHOUT blurring the line between TDD and unit testing. Not student assignments, but real world projects where real jobs, real money, real stakes are on the line. The problem here is that if TDD is as effective as the zealots and XP gurus claim it is, then the evidence for it should be overwhelming. Edit: (e) Listing projects that are: 1. real (not just student, testing or open source projects); 2. large enough that real stakes are on the line (I want to see an example of a 2 MLOC system before and after TDD, not the 70 KLOC of Fitnesse) would make me pay attention. Then I want to see how TDD improved the state of the project (and that picture has to include things like productivity and cost analysis of test first testing everything). Then I want to see why the same results could not have been achieved with "test last" or "test whenever" unit testing. Then I would want to see an explanation for the rate of success other people have had without ever practising TDD - because if TDD is as effective as the zealots claim, and if the zealots make a living making those claims, then that explanation HAS to be there.
raganwald 17y ago
What I see far too often in these debates is the following: Person A claims "Process P is good!" Person B retorts "Sez you, can't prove it to my satisfaction!" And the debate rolls on and on about whether we can prove P or not. I am not blaming Person B, but the question I want to put to them is this: Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?
- plinkplonk 17y ago
  "Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?" Good Question. Another question is "Do you go around calling legendary programmers who don't use your practices, yet ship product, "Stone Age Programmers"? Or the questioning of your favorite practice X as "X derangement syndrome"? If so why?
  - raganwald 17y ago
    My comment said absolutely nothing in support of or in refutation of statements along those lines, so I am disinclined to pursue this line of discussion. But if it amuses you, consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming. For that matter, consider the question of whether we are still in the stone age. I am inclined to say we are remarkably unenlightened. Some people ship without process, we don't really know why other than to make vague statement like "they're legends." Others have process but fail to ship and we again don't fully understand why. Although our tools are concrete, our application of them seems to be a mystery with very little hard research or true science behind it.
    - plinkplonk 17y ago
      "My comment said absolutely nothing in support of or in refutation of statements along those lines," Oh I never said or implied you did. That is why I phrased my comment as "Another question . I unreservedly apologise if anything i wrote here I gave that impression. ", consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming." but those are not the people being addressed as "stone Age programmers". The people interviewed in CaW (which is how this hole brouhaha started in the first place) and Joel Spolsky are hardly "Stone Age" programmers just because they don't "evolve design" using TDD. As per Bob Martin , people who don't use TDD are "Stone age". People who question the value of TDD are "deranged". This sounds like religious fanaticism to me. To extend your original comment. "Person A claims "Process P is good!" Person B retorts "Sez you, can't prove it to my satisfaction!"" Person A replies "That is because you are a deranged Stone Age Programmer".
Semiapies 17y ago
When people start throwing around terms like "derangement syndrome" to refer to those who don't buy their ideas, that's a warning indicator.
- kashif 17y ago
  this is such a smart and enlightening comment - why has it just got 8 votes? (sarcasm)
- ZeroGravitas 17y ago
  I get a similar feeling when the word "zealot" gets thrown around.
thunk 17y ago
It seems to me that TDD works against both bottom-up and exploratory programming. Bottom-up programming develops tiny functions that are each trivially correct. These tiny functions form a base on which the next layer of tiny functions is built, and so on -- each layer written in the layer below. Most of these tiny functions begin life as tests, which are then generalized and encapsulated in a function. So in that sense BUP is sort of test-driven. But if it's necessary to write further tests to determine the correctness of these tiny functions, You're Doing It Worng. In exploratory programming you're not sure where you're going yet. The development path during exploratory programming looks like a crooked tree with lots of stubby dead-end branches. If you first developed tests for each of those stubs, only to find that that's not where you're going, you may never stop writing tests, assuming you even know what to test for. Plus, testing during exploration would kill the flow. You can't be a pioneer and a settler at the same time. I'm not saying not to test. And I'm not saying not to use TDD. TDD is probably great for top-down projects where you already know where you're going. But if it preempts the process by which many of the great programs were written -- if it preempts hacking -- then it's not the final word.
- kristiandupont 17y ago
  My experience is that TDD helps you write code that consists of tiny functions that are trivially correct. For the simple reason that this is the easiest way to write a test for. As for the exploratory programming argument, a key element in TDD is to write one test and the make it succeed. You are not supposed to write a lot of tests and then the code that makes them succeed. So unless you are so exploratory that you don't even know what you want your next function to do, this would not hinder you either.
jlouis 17y ago
I read "An Initial Investigation of Test Driven Development in Industry" by Boby George and Laurie Williams, and "Realizing quality improvement through test driven development: results and experiences of four industrial teams" by Naggapan et al. and I am not too impressed. The first study has 6 TDD teams and 6 control teams. Our null hypothesis is that these teams produce exactly the same quality of code, measured in defect rate. Now, any sane study would take a look into (statistical) power, ie the probability the test will reject the null hypothesis. If we further look at the box-plots, we see that the overlap between the "treatment" and control group samples is pretty big. Hence, we will need to test for any statistic significance before coming to the conclusion. In fact, it may even be that we can reject the idea that writing in a TDD style takes 16% more time on average. The second study is much worse. It quotes studies secondhand, only has 4 studies (with 4 "controls" being completely different systems). It generally fails to the problem of lurking variables, or confounding: That some unmeasured external factor affects the study. For instance, it is put forth that the Microsoft teams in the study employ many methods for minimizing program defects including static analysis. The other grande problem here is that there are no assesment of the control-group systems at all. They will, invariably differ in more than one variable and the question is: Have we measured to benefits of TDD or have we measured the benefits of something else? We can't say for sure. Finally, we only test statically typed languages with a subtyping construct (Java, C#, C++). I think (and I stress it is a hypothesis) dynamically typed languages would benefit considerably more with a TDD system in place, and I hypothesize more expressively typed languages (ML, Haskell) would benefit less. But I have no good idea of how to design an experiment that can show this hypothesis true.
- joe_the_user 17y ago
  Actually, now that you point me toward the real data... It makes me even more skeptical of TDD. I would expect most projects using just about any new methodology to look notably better in tests through the programmer's equivalent of the placebo effect: a new methodology imparts energy and optimism compared to just doing the same old. This would lead me to deduce that once the "shine" wears off TDD, it would actually give no improvement in quality and result in considerably longer development time...

29 points by gthank 17y ago

Anyone else sort of think that the "To TDD or not" argument eclipses the much more important argument of "To have high unit test coverage or not"?

akeefer 17y ago
The research I've seen on TDD seems to essentially point to two things: quality and productivity is positively correlated with the number of tests, and people who use TDD tend to write more tests. People who write the same number of tests but don't use TDD get similar benefits to people who do use TDD. So TDD is itself useful primarily in that it ensures that you actually do write the tests. (Unfortunately I don't have any citations handy to back that up; that's just my recollection of what I've read on the subject over the years). There's that ever-elusive "it improves design" argument for TDD, but that one's far harder to prove either way, and I've personally seen it cut both ways: sometimes things end up more nicely decomposed, and sometimes they end up wayyyyy too decomposed.
- DannoHung 17y ago
  This squares with my experiences. As for the design argument, I think that the thing that TDD actually encourages in terms of design is testability (something that I do think is desirable, in general). If you have high test coverage without using TDD, you've probably either taken testability into consideration before you wrote your code or you made some changes after the fact to accommodate your tests.
  - akeefer 17y ago
    I'd definitely agree that TDD leads to testable code, which is usually a good thing. If you know how to write testable code, it's not as much of a win (in my opinion), since you tend to write with testing in mind. If you don't have experience with unit testing, though, doing strict TDD for a while can really help you learn what sort of constructs make for testable code. The places I've seen TDD lead to less-than-desirable design is when some larger problem is broken into such tiny pieces with so many interfaces that it becomes hard to read through the code and follow the flow of control. That's certainly testable, but often times I personally get as much mileage out of writing tests one level up, against some higher-level abstraction or API that those other things are all components of, and not being so religious about breaking up all the little parts. In my experience those slightly-higher-level tests (they're not even "integration" tests, since they're still strictly against one logical component) tend to provide the same amount of value in terms of preventing regressions while requiring fewer compromises to make the little parts testable and giving more flexibility as to future refactoring. Essentially, you're treating a whole bunch of classes/parts as internal implementation details.
    - DannoHung 17y ago
      I think we see eye to eye here. The sort of "one level up" thing is what initially attracted me to spec testing because that seemed to not only give you tools for performing your tests, but also helped direct you towards what would be useful to test: the behaviors of the code that you actually care about. So when I use spec testing tools, I tend not to think about making sure that the low level function traverses the directories in a particular way, but rather that all of the directories have been renamed (for example).
    - steveklabnik 17y ago
      > I personally get as much mileage out of writing tests one level up This is why I prefer the rebranded "BDD" as opposed to the original "TDD". I find that the perspective that BDD offers makes me write much more useful tests, since the ultimate reason we write objects in the first place is to use them, right? I don't care about how it works inside, I care about how it gets used. There's nothing technically different between BDD and TDD, it's a matter of philosophy, and I find BDD's take on things to be better.

silentbicycle 17y ago

Does it seem weird to anybody else that about half the list of prominent projects using TDD he listed are testing tools? That's like saying type systems are good because they help you write sound proofs about type systems. I know they can be really useful, but it's an incredibly circular argument.

idlewords 17y ago
It seems really weird. I've been trying to get an answer to the question "what popular apps have actually been built using TDD?" for a while, without success. All of the entries on his list are either libraries, programming languages, or testing tools and frameworks. Can anyone name actual apps that TDD has blessed us with?
- silentbicycle 17y ago
  More importantly, that were done with TDD as opposed to unit testing, testing with a repl, etc.? A lot of claims about TDD's benefits seem to take credit for testing in general. (Automated testing is a clear win, IMHO.) Insisting on writing tests upfront does mean that you test consistently, but then it's hardly the primary cause. I'm sure it's a highly useful tool in some cases, but the people pushing it uncritically make it harder to tell when it's actually a good fit. In my experience, it's very useful for safely making changes in a legacy code, but I suspect it can encourage over-engineering surface details or getting caught in local maxima when used for initial design. (I usually just design with a REPL or notepad/whiteboard.)
- chadaustin 17y ago
  Much of IMVU's client and website (tens of millions of customers) were written with TDD. I've long been a proponent of TDD once you pass the prototyping stage and you need to write a production-ready component with [most of] the edge cases figured out. The "red green refactor" loop is more like: 1) What's the next problem this software needs to solve? 2) Write a failing test that demonstrates the missing need. 3) Implement said behavior. 4) Refactor until you don't see anything else to refactor. I think step #1 is one of the most important and yet I don't see much discussion of it. Pick the most "key" tests to write first, whether that be your happy path or the functionality that has the most risk. On refactoring... I work with people who TDD all of their work and other who write unit tests afterwards. In my experience, those who test-drive are more comfortable refactoring otherwise. When I was test-driving imvu.task ( http://thespeedbump.livejournal.com/63798.html http://thespeedbump.livejournal.com/63798.html ), I spent a great deal of time just staring at the code after it was written... it's very continuation-heavy code... and suddenly I saw a refactoring that collapsed it by 1/3. We've hardly touched the code since. It solves the right problem with the right amount of code and I think it's only had a couple very obscure race conditions. In summary, TDD is one of my favorite tools for focusing my efforts so we build what we need, but only what we need.

deleted 17y ago

[deleted]

JustAGeek 17y ago

I also think that this list isn't very convincing. What I don't get, if Robert Martin uses TDD, why doesn't he simply list those "real" projects (real as in projects for clients) , he's worked on?

zacharydanger 17y ago
How many of his client projects do you expect anyone to have heard of?
- nickelplate 17y ago
  Probably not many, but I don't think that is necessary. He just has to show us a convincing list of projects (e), and demonstrate how these projects improved with the introduction of TDD, WITHOUT blurring the line between TDD and unit testing. Not student assignments, but real world projects where real jobs, real money, real stakes are on the line. The problem here is that if TDD is as effective as the zealots and XP gurus claim it is, then the evidence for it should be overwhelming. Edit: (e) Listing projects that are: 1. real (not just student, testing or open source projects); 2. large enough that real stakes are on the line (I want to see an example of a 2 MLOC system before and after TDD, not the 70 KLOC of Fitnesse) would make me pay attention. Then I want to see how TDD improved the state of the project (and that picture has to include things like productivity and cost analysis of test first testing everything). Then I want to see why the same results could not have been achieved with "test last" or "test whenever" unit testing. Then I would want to see an explanation for the rate of success other people have had without ever practising TDD - because if TDD is as effective as the zealots claim, and if the zealots make a living making those claims, then that explanation HAS to be there.

raganwald 17y ago

What I see far too often in these debates is the following: Person A claims "Process P is good!" Person B retorts "Sez you, can't prove it to my satisfaction!" And the debate rolls on and on about whether we can prove P or not. I am not blaming Person B, but the question I want to put to them is this: Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?

plinkplonk 17y ago
"Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?" Good Question. Another question is "Do you go around calling legendary programmers who don't use your practices, yet ship product, "Stone Age Programmers"? Or the questioning of your favorite practice X as "X derangement syndrome"? If so why?
- raganwald 17y ago
  My comment said absolutely nothing in support of or in refutation of statements along those lines, so I am disinclined to pursue this line of discussion. But if it amuses you, consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming. For that matter, consider the question of whether we are still in the stone age. I am inclined to say we are remarkably unenlightened. Some people ship without process, we don't really know why other than to make vague statement like "they're legends." Others have process but fail to ship and we again don't fully understand why. Although our tools are concrete, our application of them seems to be a mystery with very little hard research or true science behind it.
  - plinkplonk 17y ago
    "My comment said absolutely nothing in support of or in refutation of statements along those lines," Oh I never said or implied you did. That is why I phrased my comment as "Another question . I unreservedly apologise if anything i wrote here I gave that impression. ", consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming." but those are not the people being addressed as "stone Age programmers". The people interviewed in CaW (which is how this hole brouhaha started in the first place) and Joel Spolsky are hardly "Stone Age" programmers just because they don't "evolve design" using TDD. As per Bob Martin , people who don't use TDD are "Stone age". People who question the value of TDD are "deranged". This sounds like religious fanaticism to me. To extend your original comment. "Person A claims "Process P is good!" Person B retorts "Sez you, can't prove it to my satisfaction!"" Person A replies "That is because you are a deranged Stone Age Programmer".

Semiapies 17y ago

When people start throwing around terms like "derangement syndrome" to refer to those who don't buy their ideas, that's a warning indicator.

kashif 17y ago
this is such a smart and enlightening comment - why has it just got 8 votes? (sarcasm)
ZeroGravitas 17y ago
I get a similar feeling when the word "zealot" gets thrown around.

thunk 17y ago

It seems to me that TDD works against both bottom-up and exploratory programming. Bottom-up programming develops tiny functions that are each trivially correct. These tiny functions form a base on which the next layer of tiny functions is built, and so on -- each layer written in the layer below. Most of these tiny functions begin life as tests, which are then generalized and encapsulated in a function. So in that sense BUP is sort of test-driven. But if it's necessary to write further tests to determine the correctness of these tiny functions, You're Doing It Worng. In exploratory programming you're not sure where you're going yet. The development path during exploratory programming looks like a crooked tree with lots of stubby dead-end branches. If you first developed tests for each of those stubs, only to find that that's not where you're going, you may never stop writing tests, assuming you even know what to test for. Plus, testing during exploration would kill the flow. You can't be a pioneer and a settler at the same time. I'm not saying not to test. And I'm not saying not to use TDD. TDD is probably great for top-down projects where you already know where you're going. But if it preempts the process by which many of the great programs were written -- if it preempts hacking -- then it's not the final word.

kristiandupont 17y ago
My experience is that TDD helps you write code that consists of tiny functions that are trivially correct. For the simple reason that this is the easiest way to write a test for. As for the exploratory programming argument, a key element in TDD is to write one test and the make it succeed. You are not supposed to write a lot of tests and then the code that makes them succeed. So unless you are so exploratory that you don't even know what you want your next function to do, this would not hinder you either.

jlouis 17y ago

I read "An Initial Investigation of Test Driven Development in Industry" by Boby George and Laurie Williams, and "Realizing quality improvement through test driven development: results and experiences of four industrial teams" by Naggapan et al. and I am not too impressed. The first study has 6 TDD teams and 6 control teams. Our null hypothesis is that these teams produce exactly the same quality of code, measured in defect rate. Now, any sane study would take a look into (statistical) power, ie the probability the test will reject the null hypothesis. If we further look at the box-plots, we see that the overlap between the "treatment" and control group samples is pretty big. Hence, we will need to test for any statistic significance before coming to the conclusion. In fact, it may even be that we can reject the idea that writing in a TDD style takes 16% more time on average. The second study is much worse. It quotes studies secondhand, only has 4 studies (with 4 "controls" being completely different systems). It generally fails to the problem of lurking variables, or confounding: That some unmeasured external factor affects the study. For instance, it is put forth that the Microsoft teams in the study employ many methods for minimizing program defects including static analysis. The other grande problem here is that there are no assesment of the control-group systems at all. They will, invariably differ in more than one variable and the question is: Have we measured to benefits of TDD or have we measured the benefits of something else? We can't say for sure. Finally, we only test statically typed languages with a subtyping construct (Java, C#, C++). I think (and I stress it is a hypothesis) dynamically typed languages would benefit considerably more with a TDD system in place, and I hypothesize more expressively typed languages (ML, Haskell) would benefit less. But I have no good idea of how to design an experiment that can show this hypothesis true.

joe_the_user 17y ago
Actually, now that you point me toward the real data... It makes me even more skeptical of TDD. I would expect most projects using just about any new methodology to look notably better in tests through the programmer's equivalent of the placebo effect: a new methodology imparts energy and optimism compared to just doing the same old. This would lead me to deduce that once the "shine" wears off TDD, it would actually give no improvement in quality and result in considerably longer development time...