5 ms·

TDD won't help you solve problems that you don't already know how to solve: http://ravimohan.blogspot.co.uk/2007/04/learning-from-sudoku-solvers.html http://rav

TDD won't help you solve problems that you don't already know how to solve: http://ravimohan.blogspot.co.uk/2007/04/learning-from-sudoku-solvers.html http://ravimohan.blogspot.co.uk/2007/04/learning-from-sudoku... In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it. Of course somebody is going to chip in here and say that if the above is happening then you are writing your tests at the wrong level of abstraction and that you should be testing at the point where you can create a stable API that remains unchanged while you throw away the speculative implementations underneath. But if you are tacking a new problem you have no idea where those boundaries lie. It's hard enough to determine that for problems that you already understand and have working solutions for, never mind ones you don't. TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all. Either that or you just write vast vast numbers of trivial tests for every single function that you write (which does seem to be what some TDD tutorials advocate), but then you are creating a huge maintenance burden for the future where any change to the code is going to necessitate rewriting dozens of tests. I would argue that TDD only works if you are solving problems that you are very familiar with. By the time you are writing your 5th CRUD web app you probably have decent mental model of the parts you need before you start. I feel that a lot of the love for TDD comes from people who spend most of their time working on problems they already solved dozens of times before, but they learned how to solve them before they discovered TDD. For people first learning how to code it is absolute death. It discourages people from doing the most important thing they need to learn: throwing code away.

haar 13y ago
If you were to ask anyone in my office, you'd hear how much I praise TDD as a software development practice. However, when I'm absolutely clueless as to how to even begin to tackle a technical problem (usually due to not enough understanding of the overall system requirements), then I start by spiking the problem - trying to get a feel as to everything that's required. This initial spiking I wouldn't test-drive, as it's a learning process and not production-ready code. After I've got a strong enough grasp as to the overall requirements, then I throw away this spiked code to start fresh, working along similar lines to 'programming by wishful thinking' whilst keeping the original spiked out structure in mind. By following a strong TDD philosophy, it's easier for me to throw away my initial 'spiked out' code base for a couple of reasons: 1. Personal experience of how much better a test-driven code base can be than ones I've later tried to wrap tests around/dismantled in the process. 2. Having the understanding from the get-go that this code is not 'production-ready', and should not be considered reliable, but as a very fast learning experience. 3. Knowing that I've spent a small time prototyping as preparation, and having this laid out as an overall part of the process means I'm not considering this time 'wasted'.
giulianob 13y ago
From my experience, I've found that TDD is mostly valuable where writing the test will make writing the code more efficient. A lot of the times its because manually testing it to get the code right will add significant amount of time and then you will still need to unit test anyways. If I don't know exactly what I need to do, sometimes I will write the code to get a proof-of-concept working then just comment it out. Then I write the tests and uncomment the code as the tests are written to make sure I have proper coverage and I haven't missed anything. I hate writing tests that require me to do any kind of boilerplate code. Tests should (theoretically) only set up and exercise whatever is needed for that test. I think if you are able to do that (I've had success with AutoFixture for .NET as an example) then you'll feel a lot less bad about throwing away tests and they will also be much much faster to write. When writing tests is tedious or brittle, then I feel they are a chore and am less likely to write them to begin with.
trothoun 13y ago
sometimes you can't even really express what the problem is, until you've taken a stab a solving it. The problem might be something like 'This feature is awkward and doesn't feel quite right.' When I'm doing exploratory coding, I need to be able to feel my way through a problem. I have yet to learn to do that from the vantage of tests, and am somewhat skeptical that I ever will.
- RougeFemme 13y ago
  You can't really express it, but you still have an idea of what the problem is. . ."solving it". . .you have an "it" in your mind. Also, in many cases, when you are working on a "less technical" feature, there are others who can help express what the problem is. . .user, designer, architect, etc. It's often really tough for a developer to express what the problem is in a vacuum.
drewhk 13y ago
Absolutely! In the past I worked on an image recognition project where I implemented, tried and thrown away 14 distinct algorithms/approaches (not counting variants!) for a particular problem. TDD is unfit for this because you tend to commit yourself to the set of tests instead of throwing all of them out (all of the approaches were fundamentally different and I had soft real requirements -- so optimized representations were a must). Of course we had some unit tests, but not written in TDD style since the goal was to find the correct approach -- not the correct implementation.
- jasonlotito 13y ago
  If you are trying to do 14 different things, I don't know why you'd expect TDD to solve that for you. That's a different problem.
  - drewhk 13y ago
    Not 14 different things. 14 different approaches to solve the same image recognition problem. Please do not jump to conclusions. I did not say TDD is not useful at a _later_ stage, but at the research phase it is simply more of a burden than help, because the "refactor" step ends up as "drop everything".
    - jasonlotito 13y ago
      > Not 14 different things. > 14 different approaches So... if you are trying to take 14 different approaches... TDD isn't there to solve it for you. That's what I said. But then that's planning. And when you've found your approach, you write it with TDD.
alinajaf 13y ago
> TDD won't help you solve problems that you don't already know how to solve: I talk a lot about web application security. The following line from flesh and blood developers on multiple occasions: "Shouldn't your tests catch security vulnerabilities?" To which I always respond: "Yeah sure, they'll catch the vulnerabilities that you already thought of" I've personally seen codebases with test suites that cost (even with the most conservative estimates of time spent and day rate) upwards of 300,000 USD with 100% coverage that allowed an anonymous client to drop the users table. If a lack of security bugs (and therefore bugs in general) is any indicator of software quality, TDD has yet to impress me in the wild. My development process has always been to prototype out three or four implementations, evaluate them against the known use cases and then settle on a final API design. Then I'm ready to do TDD. Glad to know I'm not the only one who works roughly this way. It also bugs me a little that many otherwise intelligent developers use TDD as their sole software design tool much of the time. The exact term they use for this is using tests to "drive out" a design. While I understand this may work in textbook examples, in real life I've seen the concept being used to justify designs that clearly aren't fit for purpose.
- InclinedPlane 13y ago
  That illustrates a very serious problem with TDD, the idea of coverage. 100% code coverage is not 100% coverage of every different possible code path. In fact, for most software systems it is not even remotely feasible to achieve that level of code path coverage. The "100%" figure can easily lull less experienced devs into thinking they have sufficient test coverage when nothing could be further from the truth.
  - mbrock 13y ago
    However, TDD encourages – ideally – a style of programming where 100% branch coverage is likely to mean the program is correct. That's part of the idea of the "unit" in "unit test." You decompose the program into units for which correctness is relatively simple – and testable. How to do this properly is not obvious, and for teams on which TDD is imposed without this understanding, it will all become a bizarre and useless ideology. When it is understood, the resulting program will have properties that resemble those of purely functional programs, especially composability and correctness.
jasonlotito 13y ago
> TDD won't help you solve problems that you don't already know how to solve So, those Sudoku testing article fails because the author is breaking rules of TDD. I mean, the first test: test_cells, does so much more than it needs to do to pass the test. He's designing first, and then testing. That's not how TDD works. If you can't see why that first test is a bad test, you aren't qualified to judge the merits of TDD. > Of course somebody is going to chip in here and say … you should be testing at the point where you can create a stable API that remains unchanged That's not true. At least, not the way you phrase it. You are, of course, forgetting that TDD involves 3 steps. Writing a test, Passing that test, and then refactoring. That refactoring includes refactoring the test. API's can, and should change to evolve. That's part of refactoring. You pretty much repeat the same arguments against TDD that I see time and time again. Arguments born out of misunderstandings. TDD isn't easy. It's challenging, it requires practice and effort. It does take time. > TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all. Yes, but nothing about TDD prevents you from easily making changes as needed. Again, Refactoring is apart of TDD. You say something like this multiple times, making it seem like refactoring or changing code is some painful process. It's not. It's trivial. If it's not, your process has a serious problem.
- lifeisstillgood 13y ago
  > That's not how TDD works if you are looking at Ron Jeffries Doyen of the Agile set, and saying "well of course he did not get the results, he was doing it wrong" the we are well into id-protecting, religious territory. if one of the leading exponents gets the process wrong there is something wrong with applying the process to the real world TDD really seems to me to be two parts: - Design up front, disguised as writing an API - Ensuring you write repeatable tests before commuting
  - dragandj 13y ago
    I didn't check with the literature, but in my opinion, the biggest misunderstanding of TDD is that lots of people take it to mean literary Test-First Design. As I understanding, in Test-DRIVEN-Design, you are writing the implementation code in PARALLEL to testing code. You write a tiny bit of test, then a tiny bit of implementation to satisfy the test. If you notice something you'd missed previously, you refactor BOTH tests and implementation. You repeat and expand a bit...
majelix 13y ago
Yes, if you can't solve a problem, you certainly can't solve a problem in a supportable manner. Agile has the concept of Spikes (http://www.scaledagileframework.com/spikes/ http://www.scaledagileframework.com/spikes/) for researching new problems. The key point that code is thrown away after completion -- this gives you the freedom to ignore normal methodologies and not have to worry about technical debt. Presumably, you then go back with this new knowledge and only now concern yourself with writing supportable code.
yummyfajitas 13y ago
When tackling a new problem, you still need to build a lot of utility functions that you know how to write. Tests help here. For example, I'm building a statistical model. I need to load a bunch of data from files, transform it in straightforward ways, then run my model against it. TDD doesn't help me come up with the right model, but it does help me not screw up while loading the files. When my model fails to be predictive, I know the model (rather than the loader) is at fault.
ollysb 13y ago
TDD does allow you scope for exploration work. If you watch Kent Beck's TDD screencasts[1] you'll see him follow a workflow where he's exploring an unfamiliar protocol. What's interesting is that he first frames his exploration within a particular test e.g. delete item using protocol, and then writes implementation details inside the test itself until he figures out how it works. He then refactors the test so that the implementation code is moved into the driver that he's developing and the test itself becomes a proper spec. He's working in java but I tend to follow this workflow in ruby. With ruby you can fire up a REPL in the middle of your test code and gradually add to the test as you figure things out. http://pragprog.com/screencasts/v-kbtdd/test-driven-development http://pragprog.com/screencasts/v-kbtdd/test-driven-developm...
benihana 13y ago
>In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it. This is a giant strawman. Who is saying to write tests for your prototypes? You seem to be confusing a bunch of different methodologies here. You're talking about rapid prototyping to understand a domain before building a long term solution. That isn't TDD, and it would be foolish to write tests for an exploratory exercise that you're planning to throw away. >TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all How do you figure? If you've done a rapid prototype like you're advocating, you should have a better understanding of where boundaries lie and how the API should work. Further, you seem to be working under the impression having tests make it hard to change things. If anything, I've noticed the opposite - tests make it much easier to change code, especially when you test behavior, not implementation.
- ajross 13y ago
  I think the point is more the the final "prototype" is the "implementation". The core fallacy of TDD is that there is a single moment where you start work. True iterative development doesn't work that way. Which is not to say that both TDD and iteration don't have their uses. But they work together poorly. And the problems best solved by TDD seem boring to some of us.
  - marxidad 13y ago
    TDD is supposed to work WITH iteration. You have one feature, you write the test for it, you implement it, pass the test, and go on to the next feature, etc.
    - ajross 13y ago
      Implementing features piecewise is not remotely the same thing as iterative development. The point to iteration is that you develop your understanding of the problem along with the solutions, many of which get thrown away.
mkolodny 13y ago
I was taught that you should solve the problem first, and then write the code. TDD doesn't change that; it forces you to at least define the problem before you take a stab at writing the code. Of course, as you delve deeper into the problem, you often learn something that changes your understanding of the problem. That's where the third step of TDD comes in - refactor. TDD doesn't make people reluctant to refactor their code; it forces them to do it. Refactoring is a vital part of TDD.
- seanmcdirmid 13y ago
  > I was taught that you should solve the problem first, and then write the code It is sad that in this day and age waterfall is still taught as a serious methodology.

haar 13y ago

If you were to ask anyone in my office, you'd hear how much I praise TDD as a software development practice. However, when I'm absolutely clueless as to how to even begin to tackle a technical problem (usually due to not enough understanding of the overall system requirements), then I start by spiking the problem - trying to get a feel as to everything that's required. This initial spiking I wouldn't test-drive, as it's a learning process and not production-ready code. After I've got a strong enough grasp as to the overall requirements, then I throw away this spiked code to start fresh, working along similar lines to 'programming by wishful thinking' whilst keeping the original spiked out structure in mind. By following a strong TDD philosophy, it's easier for me to throw away my initial 'spiked out' code base for a couple of reasons: 1. Personal experience of how much better a test-driven code base can be than ones I've later tried to wrap tests around/dismantled in the process. 2. Having the understanding from the get-go that this code is not 'production-ready', and should not be considered reliable, but as a very fast learning experience. 3. Knowing that I've spent a small time prototyping as preparation, and having this laid out as an overall part of the process means I'm not considering this time 'wasted'.

giulianob 13y ago

From my experience, I've found that TDD is mostly valuable where writing the test will make writing the code more efficient. A lot of the times its because manually testing it to get the code right will add significant amount of time and then you will still need to unit test anyways. If I don't know exactly what I need to do, sometimes I will write the code to get a proof-of-concept working then just comment it out. Then I write the tests and uncomment the code as the tests are written to make sure I have proper coverage and I haven't missed anything. I hate writing tests that require me to do any kind of boilerplate code. Tests should (theoretically) only set up and exercise whatever is needed for that test. I think if you are able to do that (I've had success with AutoFixture for .NET as an example) then you'll feel a lot less bad about throwing away tests and they will also be much much faster to write. When writing tests is tedious or brittle, then I feel they are a chore and am less likely to write them to begin with.

trothoun 13y ago

sometimes you can't even really express what the problem is, until you've taken a stab a solving it. The problem might be something like 'This feature is awkward and doesn't feel quite right.' When I'm doing exploratory coding, I need to be able to feel my way through a problem. I have yet to learn to do that from the vantage of tests, and am somewhat skeptical that I ever will.

RougeFemme 13y ago
You can't really express it, but you still have an idea of what the problem is. . ."solving it". . .you have an "it" in your mind. Also, in many cases, when you are working on a "less technical" feature, there are others who can help express what the problem is. . .user, designer, architect, etc. It's often really tough for a developer to express what the problem is in a vacuum.

drewhk 13y ago

Absolutely! In the past I worked on an image recognition project where I implemented, tried and thrown away 14 distinct algorithms/approaches (not counting variants!) for a particular problem. TDD is unfit for this because you tend to commit yourself to the set of tests instead of throwing all of them out (all of the approaches were fundamentally different and I had soft real requirements -- so optimized representations were a must). Of course we had some unit tests, but not written in TDD style since the goal was to find the correct approach -- not the correct implementation.

jasonlotito 13y ago
If you are trying to do 14 different things, I don't know why you'd expect TDD to solve that for you. That's a different problem.
- drewhk 13y ago
  Not 14 different things. 14 different approaches to solve the same image recognition problem. Please do not jump to conclusions. I did not say TDD is not useful at a _later_ stage, but at the research phase it is simply more of a burden than help, because the "refactor" step ends up as "drop everything".
  - jasonlotito 13y ago
    > Not 14 different things. > 14 different approaches So... if you are trying to take 14 different approaches... TDD isn't there to solve it for you. That's what I said. But then that's planning. And when you've found your approach, you write it with TDD.

alinajaf 13y ago

> TDD won't help you solve problems that you don't already know how to solve: I talk a lot about web application security. The following line from flesh and blood developers on multiple occasions: "Shouldn't your tests catch security vulnerabilities?" To which I always respond: "Yeah sure, they'll catch the vulnerabilities that you already thought of" I've personally seen codebases with test suites that cost (even with the most conservative estimates of time spent and day rate) upwards of 300,000 USD with 100% coverage that allowed an anonymous client to drop the users table. If a lack of security bugs (and therefore bugs in general) is any indicator of software quality, TDD has yet to impress me in the wild. My development process has always been to prototype out three or four implementations, evaluate them against the known use cases and then settle on a final API design. Then I'm ready to do TDD. Glad to know I'm not the only one who works roughly this way. It also bugs me a little that many otherwise intelligent developers use TDD as their sole software design tool much of the time. The exact term they use for this is using tests to "drive out" a design. While I understand this may work in textbook examples, in real life I've seen the concept being used to justify designs that clearly aren't fit for purpose.

InclinedPlane 13y ago
That illustrates a very serious problem with TDD, the idea of coverage. 100% code coverage is not 100% coverage of every different possible code path. In fact, for most software systems it is not even remotely feasible to achieve that level of code path coverage. The "100%" figure can easily lull less experienced devs into thinking they have sufficient test coverage when nothing could be further from the truth.
- mbrock 13y ago
  However, TDD encourages – ideally – a style of programming where 100% branch coverage is likely to mean the program is correct. That's part of the idea of the "unit" in "unit test." You decompose the program into units for which correctness is relatively simple – and testable. How to do this properly is not obvious, and for teams on which TDD is imposed without this understanding, it will all become a bizarre and useless ideology. When it is understood, the resulting program will have properties that resemble those of purely functional programs, especially composability and correctness.

jasonlotito 13y ago

> TDD won't help you solve problems that you don't already know how to solve So, those Sudoku testing article fails because the author is breaking rules of TDD. I mean, the first test: test_cells, does so much more than it needs to do to pass the test. He's designing first, and then testing. That's not how TDD works. If you can't see why that first test is a bad test, you aren't qualified to judge the merits of TDD. > Of course somebody is going to chip in here and say … you should be testing at the point where you can create a stable API that remains unchanged That's not true. At least, not the way you phrase it. You are, of course, forgetting that TDD involves 3 steps. Writing a test, Passing that test, and then refactoring. That refactoring includes refactoring the test. API's can, and should change to evolve. That's part of refactoring. You pretty much repeat the same arguments against TDD that I see time and time again. Arguments born out of misunderstandings. TDD isn't easy. It's challenging, it requires practice and effort. It does take time. > TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all. Yes, but nothing about TDD prevents you from easily making changes as needed. Again, Refactoring is apart of TDD. You say something like this multiple times, making it seem like refactoring or changing code is some painful process. It's not. It's trivial. If it's not, your process has a serious problem.

lifeisstillgood 13y ago
> That's not how TDD works if you are looking at Ron Jeffries Doyen of the Agile set, and saying "well of course he did not get the results, he was doing it wrong" the we are well into id-protecting, religious territory. if one of the leading exponents gets the process wrong there is something wrong with applying the process to the real world TDD really seems to me to be two parts: - Design up front, disguised as writing an API - Ensuring you write repeatable tests before commuting
- dragandj 13y ago
  I didn't check with the literature, but in my opinion, the biggest misunderstanding of TDD is that lots of people take it to mean literary Test-First Design. As I understanding, in Test-DRIVEN-Design, you are writing the implementation code in PARALLEL to testing code. You write a tiny bit of test, then a tiny bit of implementation to satisfy the test. If you notice something you'd missed previously, you refactor BOTH tests and implementation. You repeat and expand a bit...

majelix 13y ago

Yes, if you can't solve a problem, you certainly can't solve a problem in a supportable manner. Agile has the concept of Spikes (http://www.scaledagileframework.com/spikes/ http://www.scaledagileframework.com/spikes/) for researching new problems. The key point that code is thrown away after completion -- this gives you the freedom to ignore normal methodologies and not have to worry about technical debt. Presumably, you then go back with this new knowledge and only now concern yourself with writing supportable code.

yummyfajitas 13y ago

When tackling a new problem, you still need to build a lot of utility functions that you know how to write. Tests help here. For example, I'm building a statistical model. I need to load a bunch of data from files, transform it in straightforward ways, then run my model against it. TDD doesn't help me come up with the right model, but it does help me not screw up while loading the files. When my model fails to be predictive, I know the model (rather than the loader) is at fault.

ollysb 13y ago

TDD does allow you scope for exploration work. If you watch Kent Beck's TDD screencasts[1] you'll see him follow a workflow where he's exploring an unfamiliar protocol. What's interesting is that he first frames his exploration within a particular test e.g. delete item using protocol, and then writes implementation details inside the test itself until he figures out how it works. He then refactors the test so that the implementation code is moved into the driver that he's developing and the test itself becomes a proper spec. He's working in java but I tend to follow this workflow in ruby. With ruby you can fire up a REPL in the middle of your test code and gradually add to the test as you figure things out. http://pragprog.com/screencasts/v-kbtdd/test-driven-development http://pragprog.com/screencasts/v-kbtdd/test-driven-developm...

benihana 13y ago

>In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it. This is a giant strawman. Who is saying to write tests for your prototypes? You seem to be confusing a bunch of different methodologies here. You're talking about rapid prototyping to understand a domain before building a long term solution. That isn't TDD, and it would be foolish to write tests for an exploratory exercise that you're planning to throw away. >TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all How do you figure? If you've done a rapid prototype like you're advocating, you should have a better understanding of where boundaries lie and how the API should work. Further, you seem to be working under the impression having tests make it hard to change things. If anything, I've noticed the opposite - tests make it much easier to change code, especially when you test behavior, not implementation.

ajross 13y ago
I think the point is more the the final "prototype" is the "implementation". The core fallacy of TDD is that there is a single moment where you start work. True iterative development doesn't work that way. Which is not to say that both TDD and iteration don't have their uses. But they work together poorly. And the problems best solved by TDD seem boring to some of us.
- marxidad 13y ago
  TDD is supposed to work WITH iteration. You have one feature, you write the test for it, you implement it, pass the test, and go on to the next feature, etc.
  - ajross 13y ago
    Implementing features piecewise is not remotely the same thing as iterative development. The point to iteration is that you develop your understanding of the problem along with the solutions, many of which get thrown away.

mkolodny 13y ago

I was taught that you should solve the problem first, and then write the code. TDD doesn't change that; it forces you to at least define the problem before you take a stab at writing the code. Of course, as you delve deeper into the problem, you often learn something that changes your understanding of the problem. That's where the third step of TDD comes in - refactor. TDD doesn't make people reluctant to refactor their code; it forces them to do it. Refactoring is a vital part of TDD.

seanmcdirmid 13y ago
> I was taught that you should solve the problem first, and then write the code It is sad that in this day and age waterfall is still taught as a serious methodology.