I think I pointed that out in my essay when I discussed how one limitation with TDD is that tests must fail, then change the code to pass the test. This doesn't let you add tests which are expected to pass but which are used to assess the validity of the code.
One way is make the tests fail, such as by changing the comparison from the expected value to something else, or by forcing an exception at the end of the code. Once the test fails then you know the test is being executed against the expected code, so then change it to make it pass. This is done without modifying the code. It's similar to what you would have to do if you depend on a third-party package which might be buggy. Code coverage is another way to check if the test is actually exercising the code paths you think it should. And yes, in some cases I will insert failures into the code itself, to convince myself that the test is also working, but that's only one of several possibilities.
While you make only a single iteration when developing APIs, I find that I many more than that to get the API to feel right, and that API development needs feedback based on implementation. I'll end up with a bunch of semi-tested but reasonable code, which I then test through unit tests, guided by my experience and assisted through code coverage (either manual or automated). It is much easier to make this code work right than it would be to start from scratch.
I'm also not sure that I would say "post hoc", since it really is a mixture of different types and quality of tests all throughout the development process.
To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.
I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.
I'm not aware of any other technique that does that.
Because there isn't one, in the non-trivial case but there are many that try to help - from static typing to design-by-contract and of course a zillion other testing methodologies. In fact, chances are very good that what you do is not strictly unit testing. And that's fine and good. TDD didn't invent testing.
http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-not-to-write-unit-tests.aspx http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-not-...
Another rather irritating thing is the insistence that TDD is a design methodology. If you haven't yet, read the informative case of an XP adherent trying to write a sudoku solver with TDD. Here's a blog post that links to all the relevant articles - the XP'ers attempt, Peter Norvig's solution and a reference to Coders at Work where Norvig talks about it a bit.
http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s...
Long story short, if you don't know how to solve a problem and reason about a problem TDD won't help you. Neither will Design Patterns, UML or any particular fashionable technique, of course.
I think perhaps in my quest for concision I failed to communicate. I certainly didn't mean to imply that TDD could provide certainty for the entire system.
But I have to differ with you in one key point. TDD specifically proves that the code does not pass your test before you made the change. The red bar is, in my mind, the most important part of the practice.
Given that you have a failing test, you make a change, and then you have a passing test, what more do you feel is required for proof that the change did what you expect? It's possible that "proof" is too strong a word, but I find that process gives me sufficient confidence.
(Side note: I'm honestly trying to figure out the best way to write code. I don't think TDD is the silver bullet, but I think it's a useful tool in the mechanical part of development.)
We're using 'pass' in slightly different senses, what I meant is (and this applies to any test methodology) - in the strictest sense, getting the expected outputs from a test-suite, making a change and getting the expected outputs from a test-suite proves that you are getting the expected outputs from your test-suite. It may do a lot of other things but typically proving correctness is not one of them.
Side note: I'm honestly trying to figure out the best way to write code.
I think that's key and merits more than a side note. One of the reasons just about any methodology can boast of success stories is that given a concerted, focused and rational effort to improve quality, quality tends to improve, regardless of methodology. Such efforts are quite hard to get right, especially for organizations but when they succeed, there's little empirical evidence the primary cause is the methodology.
So if TDD helps frame your own efforts to be a better programmer, by all means, TDD away. The TDD griping comes from the often overblown claims of TDD proponents that TDD improves quality. 'Honestly trying to figure out the best way to write code' is a more valuable starting point than TDD.
I think the important realization in here is: TDD is a /tool/. If we fix the *DD == tool, we can reason that a good programmer has a toolbox in his head, containing a number of such tools, that is, a number of development and driving strategies which can be selected according to the problem at hand.
This is at least what I am working on right now. I have a solid grip on TDD, and I begin to understand where TDD works and where it does not (e.g. simple datastructures are nice, and things like katamorphisms can be well-tested pointwise. Basically the "middle tier" and "inner plumbing" of an application, but complicated algorithms, IO and such are problematic.
On the other hand, I am currently investigating the refinement calculus in order to derive programs from specifications. It is true that RDD (for consistencies sake: refinement driven development) is really, really hard when you try it on big problems and IO is again an evil problem lurking in the dark, but on the other hand, refining invariants has given me some really elegant and clean recursive solutions to some really tricky problems. I have to admit, it took around an afternoon to understand the result from refining, but I was able to throw just about any input I could think of at the problem and it just worked.
Third, there is probably some sort of KDD, knowledge driven development. Knowledge driven development occurs if you see a problem and just know the solution. In that case, you can just write it down, fix a few bugs and be done (or receive really obscure wrong outputs :)). Compare Norvigs sudoku solver for some pure knowledge driven development.
I think this makes sense and at least fits my current observations and explorations in the land of improving ones code-writing.
> I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.
Would it have helped if the OP had said "made me more confident" instead of "proves"?
It would not have helped at all since it completely changes the meaning. 'proves' is a statement about the code, if true then we can talk about the influence of TDD on the code. 'made me more confident' is a statement about the author's state of mind - it lets us talk about the effect of TDD on the author's state of mind, which is a lot less interesting. Re-reading my code carefully or telling myself I'm awesome might make me more confident in my code changes but you rarely hear much about Careful Inspection Driven or Self-affirmation Driven Design.
I very much agree with this. I for myself think writing tests first or rather during development for well testable units as sweeping a number of small pebbles out of my way, so tripping due to trivial typos is less likely. Certainly, there might still be big rocks to encounter and I might still fall due to a large crack in the ground, but I certainly won't fall over due to small slippery pebbles.
In other words, I think TDD can be an amazing help at _implementing_ a solution after it was designed and planned and I think the ability to change code while the tests lower the probability of existing assumptions breaking unnoticed allows one to improve the existing design of a solution easily and significantly, while I also agree that designing and solving with tests only is awkward.
In case you want better examples where test driven development will fall flat on its face, take a look at the problem sets on http://uva.onlinejudge.org/ http://uva.onlinejudge.org/ . You have 1-6 testcases in each problem, so there are enough test cases? :)
To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.
I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.
TDD does not prove your change did what you expect, at all. In that you are correct. However, TDD does provide a way to alert you if your change caused a side-effect you were not expecting it to. ie, passing tests does not indicate success, but failing tests alerts you to unexpected failures in edge cases you may not have considered otherwise.
That doesn't invalidate what you said at all, and in fact I agree with it, and think too many people probably equate passing tests with successful feature implementation, which is not ensured. But the fact that it does not serve the purpose you state, does not mean it does not serve any purpose at all.
I think there is some value in it, I just don't know that it is what many of the proponents try to push it as.
it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect
This advantage is less useful the better you know the system and the better you are as a programmer. One experienced programmer I know (30+years programming) said "When I know what to test I don't need the tests." When I started programming it seemed like I spent most of my time getting the code to do what I want, but that’s morphing so I spend more time trying to understand what I need the code to do. IMO, TDD is like using a good debugger, they can be awesome tools when you have a vague idea of what's going on, but once you really understand the code base you they slowly become less useful.
PS: The guy I am talking about was one of those "old masters" you occasionally hear about. One of my favorite stories about him was: "He once spent a week cleaning up someone’s code and sent it off to testing with some misgivings, which it passed and then failed in production. Apparently the testers had learned to ignore his code as it was normally a waste of their time to look at it, and after a few years this ended up biting them is the ass." So clearly testing is useful, but you need to balance cost and benefit.
"I'm not aware of any other technique that does that."
Well, as pointed out by others, TDD doesn't actually prove that.
There are however methods that do, which fall under the general heading of formal methods or program proof. Read "A Discipline of Programming" by Dijkstra for a reasonable introduction. The techniques work, but are laborious. Modern theorem provers can help, but it still involves considerable effort. Which implies that such methods likely won't provide a return on investment unless the cost of errors in the program is extremely high (avionics, medical control).
I prefer to think of TDD as similar to working a problem backwards and forwards in elementary algebra. You may still have gotten the answer wrong, but you'll have to have made a mistake twice, so it helps your odds quite a bit.