11 ms·
No. TDD failed for engineering reasons in addition to economic ones. The main failing of TDD is the assumption that you know all of the tests that you will ne
by taeric 9y ago
No. TDD failed for engineering reasons in addition to economic ones.
The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software. This is true in bridge building as much as it is in anything else. Until you fully know the domain of what you are building, you cannot possibly know all of the things you need to test for.
To that end, if TDD were focused around building things to break them and analyze them, it would be wise. However, in software it is often taught to build a wall of failing tests, and you can then use that as a sort of burn down chart for progress. Note that you don't learn about the software you are building from this burndown. Instead, you only get a progress report.
And obviously, this is all my opinion. I'm highly interested in exploring the ideas. And I'm highly likely to be wrong in my initial thoughts. :)
- s73ver 9y agoI've never heard TDD being that you write all of your tests up front, before you've written any code. I've always heard it as: When you're writing a class/unit, you write a few tests for what that unit is going to do. You then make those tests pass. You add some more tests, make those pass, and so on and so on.
- taeric 9y agoYou are still writing the tests first, with no real learning from them other than "they are now passing." It is thought this is like engineering. You design an experiment and then pass it. However, that is not how experiments work. You design an experiment, and then you collect results from it. These may be enough to indicate that you are doing something right/wrong. They are intended, though, to be directional learnings. Not decisions in and of themselves. So, I probably described something in more of a strawman fashion than I had meant. I don't think that really changes my thoughts here, though.
- sb8244 9y agoYou can throw away the test if you don't find value in it. This can be very useful for fleshing out an idea.
- taeric 9y agoBut this advice should be extended further. Always be willing to throw away the code and the tests. Take the learning. Be ware, though, that you are not throwing your customers for a bad ride. As soon as you have customers, it is nigh impossible to throw away the code without neglecting them. And they are your ultimate responsibility. Not the code.
- sb8244 9y agoExactly right, it can be very hard to make changes (even the right changes) once it's in customer hands. Sometimes it is easy, sometimes it is not. However, it's always easy to throw out code that isn't yet committed.
- nostrademons 9y agoYou're often learning the optimal API as well. I've had several experiences where I write a library, write some tests for it, and then realize that the library's API is inconvenient in ways X, Y, and Z, and that I could have a much simpler API by moving a few tokens around. When I write the tests first, I tend to get a fairly usable API the first time around, because I'm designing it in response to real client code. (This all depends on having a clear enough idea of what the library needs to do that I can write realistic test code...I've also had the experience where I write the tests, write the library, and then find that the real system has a "hidden" requirement that isn't captured by the tests and requires a very different way of interacting with the library.)
- taeric 9y agoSo, this is a slightly different mantra. Working backwards from the customer is an important thing. And if you know that customers will be using your product programmatically, then yes, building the API first is important. But note, in that world, you aren't necessarily building tests of your software. You are building specifications for your API. I can see how these might be seen as similar. And for large portions of your code, they can be. However, for the vast majority of your code, it exists to help out other parts of your code. Not to be used as a library/framework by others.
- mayoff 9y agoIt’s funny that you say “I'm designing it in response to real client code.“ To me, a test is not real client code. Real client code is code that calls the API in service of the user. E.g. the real client code for the Twitter API is in your preferred Twitter client, not in the tests that run against the Twitter API.
- nostrademons 9y agoThat's the caveat I mention in my second paragraph. But yes, if the tests are well-designed and built with actual real-world experience, I do treat them like real client code. Someone looking to use the library should be able to read the unit tests and have a pretty good idea what code they need to write and what pitfalls they'll encounter. And when the library is redesigned or refactored, the tests are first-class citizens; they aren't mindlessly updated to fit the new code, they're considered alongside other client code as something that may or may not have to change but ideally wouldn't.
- ericgunnerson 9y agoAuthor here... This discussion on what happens - or doesn't happen - after the tests are passing is precisely the point of the article. The idea of TDD is that every test you look at the existing code and improve it. The majority of developers I've seen trying TDD do this with small code but do not with significant codebases, and you end without the benefits.
- robotresearcher 9y ago> with no real learning from them other than "they are now passing." You don't learn much the moment the test passes. But designing and writing the test requires understanding the requirements in detail, which is often a learning process.
- PretzelFisch 9y agoIf you wrote the test after you coded your unit,you would not know why it passed. It could be a false positive. When you move on to the next part of the requirement, you can ensure all your tests still pass. Sometimes the tests give you the starting point of the problem, "what is the simplest way to get result x" with a way to rapidly get feed back that all the requirements are ment as the structure takes form and you refactor all the code.
- bluGill 9y agoI sometimes write tests that I know will pass. A common example is when the previous version of an algorithm had a bug in some special case. My new algorithm written from scratch doesn't have that special case at all so there is no test for it. However because the last version had that bug there is a lot of fear so I will write a test that it works even though it is inherent in my new algorithm that it works and at no point in the TDD cycle is that test the correct one to write.
- Sir_Substance 9y ago>I've never heard TDD being that you write all of your tests up front, before you've written any code. That's interesting, because as a non-TDD practitioner who occasionally gets it evangelized to me, that's certainly how I was told to go about it by multiple people. Possibly TDD has a marketing problem?
- goldenkey 9y agoTDD doesn't have a marketing problem -- all these sponsored wigs evangelized it for the last 20 years, there's at least a dozen at every conference, many getting paid thousands to give their gritty speeches. TDD as these maggots explained it is: write your tests first, write your code, make sure the tests pass, rinse, repeat.
- logfromblammo 9y agoTDD is writing code to automatically validate that a specification/requirement has been met, before writing the code that meets the requirement. It does one requirement at a time. You might add a specification that the code has to create a list of strings from a data set. You write a test that takes a data set as input and checks the output to see if it is the expected result. This fails, because the code hasn't been written yet. Then you write the simplest function possible to make the test pass. Once it passes, you check to see if anything can be refactored. You can refactor as much or as little as you like, provided that all the tests still pass. Once you're satisfied, you write another test for another requirement. It's like a spinning ratchet wheel, where the test suite is the pawl. It keeps you from backsliding, but you can't spin too fast, or it flies off and won't engage with the wheel. Turn-click-turn-click-turn-click. Writing all your tests up front is one of the worst ideas I have ever heard.
- kevwil 9y agoWow, if that's how people are describing it, no wonder it's seen negatively. That seems very waterfall-ish and horrible. Thankfully that's not TDD as originally intended.
- thaumasiotes 9y agoIt's a very common way to evangelize TDD. Ruby Koans is a tutorial entirely based on that principle.
- ZoFreX 9y agoBut Ruby Koans is a tutorial on Ruby, not on TDD.
- sb8244 9y agoYou are correct. Not only this, but it's a blatant violation of TDD. TDD is having a single test go red->green, not a suite.
- tomnipotent 9y agoExactly this. It's Red, Green, Refactor. 1. Write test; build fails because code under test DOESN'T EXIST 2. Write the least amount of code needed for the test to pass 3. Refactor & repeat process ad infinitum The Wikipedia entry for TDD also states this explicitly [0] This is a differentiating feature of test-driven development versus writing unit tests after the code is written: it makes the developer focus on the requirements before writing the code, a subtle but important difference. [0] https://en.wikipedia.org/wiki/Test-driven_development#Test-driven_development_cycle https://en.wikipedia.org/wiki/Test-driven_development#Test-d...
- andybak 9y ago> I've never heard TDD being that you write all of your tests up front, before you've written any code. That's the only way I've ever heard it described. I thought that's why it's called "Test Driven"
- skookum 9y agoCan you provide a reference? I'm not a big proponent of TDD but have read up on it and experimented with it some, and none of the TDD literature I've read suggests writing all the tests up front. Virtually all the mainstream TDD literature and talks advocate for the "write a test, make it pass, lather, rinse, repeat" model of development.
- taeric 9y agoI think this is a case of two ways to hear "write all the tests first." If you read that as "write every single one of your tests, then write the code", then you see it as a strawman and not the point of TDD. If you read that as "before you write any specific line of code, you write the test for it first", then you are likely to see that as the same as TDD. I intended the second reading. I can see how both are accurate ways to read it, though. Apologies for the confusion.
- andybak 9y agoAh. Exactly the misunderstanding I was labouring under. Turns out we are all in agreement after all.
- deleted 9y ago[deleted]
- s73ver 9y agoI mean before you've written any code at all. Like, you have to have the tests for all units created before you start writing the code for any unit. That's what the person I responded to made it sound like. Writing tests before you work on the unit you're working on shouldn't be a problem because you should know what that unit is supposed to do. If you don't, then it doesn't matter if you do TDD, Agile, BDD, or whatever. You're not going to be able to do it right. Go have a conversation with someone to find out what that is supposed to do.
- annabellish 9y ago> However, in software it is often taught to build a wall of failing tests People teaching that are not teaching TDD. They may be letting the tests drive the development, but TDD specifically is _all_ about the second-to-second cycle of moving between test and code. If you write all the tests up front then refactoring is expensive and you've locked in your design way too early.
- amirouche 9y agoI don't do the TDD like a waterfall. I don't do upfront all the tests, there is a discussion between the code (what express the business need) and the tests that "wraps" it inside a safe zone (what ensures that the behavior is the one intended). As such, as the idea grow, both the code and the test grow at the same time. Look at [0] to have an idea of how the tests extend the safe zone along the code. Also, I much prefer a lot of nice unit tests over a lot selenium tests (which takes a lot of time to run). [0]https://github.com/charlesthk/python-resize-image/commits/master?after=e4a7e5a8f69ebf491ef069c13530d159594e2c87+69 https://github.com/charlesthk/python-resize-image/commits/ma...
- taeric 9y agoRereading my post, I see where it sounds more strawman of the "entire wall of tests." I did not mean it to be the final test suite is written first. Instead, I intended that to be the wall that you will be passing in a given cycle. So, for a small development cycle, you add in some new tests that will be this cycle's "wall" of failing tests that you flip to passing. And don't take this to be an indication that I think integration tests are somehow superior. They all have advantages and there is a massive cost benefit tradeoff that has to be considered at all times. It is easy to talk in strawmen. Such that I can confidently say if you have a set of tests that always fails together, that should just be a single test. However, I do not feel that statements like that are informative, precisely because it is a judgement call masquerading as a rule.
- jaggederest 9y ago> So, for a small development cycle, you add in some new tests that will be this cycle's "wall" of failing tests that you flip to passing. That's not right either. There is no "wall". Writing tests helps you write the code which helps you write better tests. It's mutual recursion, not a one-way waterfall.
- tieTYT 9y agoThat's not how you're supposed to do it. Write one test, get it to pass, repeat.
- enraged_camel 9y agoYes, precisely. TDD works for experienced developers who know the problem domain well. It does NOT work for that new junior dev you hired straight out of college. They need to write functions and features first and see the spectacular ways in which those functions and features can fail (in a dev/test environment, hopefully), and then use that experience to switch over to TDD when they write a similar feature/function in their next project.
- deleted 9y ago[deleted]
- the_af 9y ago> TDD works for experienced developers who know the problem domain well This is important: it's not only junior devs that fail -- TDD also does NOT work for experienced developers who do NOT know the problem domain well, as the infamous Sudoku solver debacle showed, where Ron Jeffries got too caught up in TDD/refactoring to make any serious progress in actually solving Sudoku.
- spronkey 9y agoAmen. And this is what spikes are for :) I TDD, but I don't TDD my spikes. Sometimes I write an almost working system before I start actually working...
- lliamander 9y agoIt's worth noting that Kent Beck ("Father of TDD") himself said that TDD doesn't always make sense. For him, the important thing has always been having a tight feedback loop for determining if he was still on the right track. For instance, when he was working on an unstructured log parser, there was no way TDD would work because the logging stream didn't have a pre-defined structure. Instead, he developed a process where he could quickly inject his parser into the logging stream, visually inspect the results, tweak, and then test again. Others (Such as Martin Fowler and Uncle Bob) have also emphasized the value of automated unit testing integrated with a CI pipeline as a way of ensuring that you didn't break existing functionality. As always, what's important is not the specific practice, but rather the engineering principles behind it. The key questions being: * Am I on the right track? * Did I break existing functionality? * How quickly and frequently can I get answers to the above two questions?
- YZF 9y agoI absolutely agree. It's also nice to know: * What's my confidence that the code will work correctly under different situations. * How maintainable the code is and related how easy is it to make changes. * How much time/effort does it take to create. It's funny (or sad) how the XP guys like Kent are always talking about doing the right thing and not being religious. This stops working when the people using it don't know what the right thing is any more. They write terrible tests, they make bad changes to their code to support those tests, it ends up a big un-refactor-able ball of mud that takes longer to create and is more difficult to maintain. If you're writing a shortest path algorithm or a JPEG decoder then you absolutely should code up some tests to check your code. The tests are relatively simple. They don't need to know anything about the internals of your code. They help you check quickly whether your code is correct and don't interfere with any refactoring. You can rewrite your JPEG decoder from scratch without needing to touch the tests at all.
- convolvatron 9y agoi'm not a fan of tdd, and a huge proponent of testing in general. what you say about making bad changes to support tests really strikes a chord. in tdd environments the test harness is used as some kind of crutch to avoid having to have difficult discussions about what the product should* be doing - what makes sense. the test acts as some kind of oracle in the regard, even though it was written by an intern last spring who didn't really have any idea what was supposed to happen either. i'm basically afraid to say 'the test is wrong' anymore. i always get back a blank look of incomprehension or even outright fear.
- yebyen 9y ago> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software. You know what makes this work for me (I am a TDD & BDD dev, and I don't think it's dead) is the fact that you can reorder commits. When I hit a case like this, where the correct code is easier to write than the correct test, I still test it. I write the test after, then I reorder the commits once the test passes, and make sure it fails predictably in the way you were expecting it to fail. Sometimes you find your test doesn't fail correctly with the codebase you had before the feature was implemented, and then you amend the commit and use rebase or cherry-pick. Adding BDD to the mix helps a lot too. Sometimes you can write the whole test up-front! Sometimes you can write pseudocode for the test, but you don't have enough information about the details of the implementation to write what you'd call a complete test before the feature is born. Usually you can at least get a vague, english-language description of what the feature is supposed to do. Then you have, at least, a foundation for your test (which can ultimately be upgraded to an executable proof/regression test), and you have a way to measure how far from the vague, usually simple, plain-english description of the feature you've had to go in order to produce "correctly behaving" code. (And you can amend the behavioral spec to correctly explain the edge cases.) I am in the habit of telling my team members with less TDD and BDD experience, who are just trying to get the hang of building up a codebase with strong regression tests, that they should always try to write a Cucumber Scenario before they wrote the feature, even if they don't write step definitions until (sometimes a great while) after the feature is done. I agree that the burn-down chart is not the end goal of TDD, and you should not optimize your process simply around making sure you have a progress chart and a great estimate up-front. Complete coverage of the codebase is more important! And surely there are other things even more important than that.
- charlieflowers 9y ago+1, that's a pretty cool idea >> I write the test after, then I reorder the commits once the test passes, and make sure it fails predictably in the way you were expecting it to fail.
- yebyen 9y agoYes! When you know how to use git, you can honor the spirit and basic practice of TDD without following it to the letter.
- dang 9y ago> the assumption that you know all of the tests that you will need before you know your software I'm not a TDDer but must object on their behalf: not only is that not what they said, it couldn't be further from it! What they taught was to work in short iterations and work out a bit of requirements, design, and implementation in each one. If you needed knowledge you didn't have yet, they would advise you to make each cycle shorter until you did. Sometimes they'd take this to absurdity, breaking even simple features into design-test-implement cycles so microscopic that you would feel bored like a kid in class who can already see the next dozen steps and has to sit there waiting for the teacher to catch up. The TDD approach was at the extreme tip of trying to get away from waterfall methods, so it would be quite some irony if it reduced back to that. Edit: looks like I misread what taeric meant! But as McLuhan said, 'you don't like those ideas, i got others'... In my experience there were two problems with TDD. One is that the micro-iterative approach amounted to a random walk in which features agglutinated into a software blob without any organizing design. Of course that's what most software development does anyway. But if you remember what Fred Brooks et. al. taught about having a coherent design vision for software that can organize a codebase at a higher level, the way iron filings are ordered by a magnetic field, that is a key anti-complexity vector which the TDD/XP/agile projects I saw all lacked. (Original XP had a notion of 'metaphor' that pointed vaguely in this direction but no one knew what it meant so it had no effect.) The lesson was that a coherent design vision doesn't evolve out of bit pieces, and there is an important place for the kind of thinking that got dismissed as 'top-down'. (Edit: by far the deepest work on integrating design coherence into TDD/XP approaches was Eric Evans' stuff on domain models and ubiquitous language. Eric is my friend but I knew he had the best stuff on this before that :)) (Edit 2: Eric points out that the people who created TDD were software design experts for whom coherent design was as natural as breathing air. The admonition against too much design thinking in TDD was intended for people like themselves, for whom the opposite mistake was never a risk, and didn't have unintended consequences until later.) The other problem was that practices that started out as interesting experiments ossified into dogma so damn quickly that people started acting like they'd cracked the code of how to build software. A soft zealotry emerged that was deeply out of sync with the playful, adaptive spirit that good software work needs. This was unintentional but happened anyway, and there's a power law in how much creativity it kills: each generation that learns the Process is an order of magnitude less able to think for itself within it, and so must break free. I don't think the creators of the approach were any more at fault for this than the rest of us would have been in their shoes. The lesson is that this ossification is inevitable, even if you know all about the danger and are firmly resolved to avoid it. If there is any real breakthrough to be had at the 'how to build software' level, it is probably discovering some kind of antifreeze to put in there from the beginning to make sure this doesn't happen. Maybe there's a way to interrupt the brain and social circuitry that produce it. Failing that, I think that each project or maybe (maybe!) organization should hash out its own process from first principles and the prior experience of team members. That's ad hoc and involves a lot of repeating past mistakes for oneself but at least it isn't idiotifying. It's how I like to work, anyhow, and IMO that's what all software processes reduce to.
- api 9y agoI guess another way to put it is that TDD is an attempt to do old school "waterfall" in code form. Way back before anyone really understand how software engineering should be done, people would begin a software product by attempting to pre-specify ahead of time every nuance of the system. They'd write these massive books of requirements. Then they'd get to work coding to these requirements. This rarely worked out well and usually resulted in huge amounts of wasted effort. When you get coding you realize the spec is either wrong or there's a much, much better way to do it. Do you do it the bad way to stick to the spec or do you deviate from the spec? The other problem with this approach is that the requirements are often as hard to write as the code and contain as much information. Why not just write the code? TDD is just "the spec" written in code instead of as a massive design document. Attempting to achieve 100% test coverage often means writing a test set that is a kind of negative image of the code to be written and is equal in complexity. As such the tests often end up containing bugs, which is the same problem as the classical "when we got going we found that the spec was wrong." Do you code to the bugs in the tests? This is not to say tests should be skipped. Automated tests are great. I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice. Test where you can. Test where you must. Keep going. Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula. That categorically does not work outside domains that are purely deterministic in nature, and those can be fully automated so humans aren't needed there at all. There is no formula, or alternately the formula is general intelligence continuously applied. Edit: The same evolution has happened in business. Nobody writes a "business plan" anymore. As soon as you get into business you have to toss the plan so why waste your time. Planning is a false god.
- benji-york 9y agoI have never seen anyone propose writing all the tests before writing a single line of code.
- jacques_chester 9y agoYour description does not resemble TDD as I learned it and practice it. > As such the tests often end up containing bugs All code contains bugs, but we do not abandon programming. Having statements of behaviour from outside and inside the solution increases the chances that one will reveal a meaningful difference from the other. > I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice. I work for Pivotal, one of the most outspokenly pro-TDD shops in the industry. I have, quite literally, never seen or heard any engineer mention test coverage ever. > Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula. I see them as a starting point. The expert mind sees possibility, but the novice mind needs certainty.
- danthemanvsqz 9y agoIn TDD you don't build a wall of tests, you build your tests as you build your code. Test a little, code a little, refactor, repeat.
- kybernetikos 9y ago> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software. I don't think this is true. TDD is also taught as a way of exploring a space and a codebase. And there is no assumption that the tests created during TDD are the only tests you will need. They're the tests needed to drive out the desired behaviour. You'll also need tests to check edge cases, tests that operate at a higher level, tests that integrate different systems, nonfunctional tests, etc.... In a lot of ways they are a discovered, executable spec. > To that end, if TDD were focused around building things to break them and analyze them, it would be wise. However, in software it is often taught to build a wall of failing tests, and you can then use that as a sort of burn down chart for progress. I've never come across TDD taught that way. Hard core TDD as I know it is RED, GREEN, REFACTOR. You never want more than one failing test, and you don't generalise until you've triangulated based on creating tests that force you to. I don't personally use TDD on all my projects because often it feels fussy and constraining when you are in exploration mode, but I also don't think it failed. I know people who find it valuable even for exploration. I'm also a big fan of using it for certain kinds of things - e.g. building a collection class.
- ngoede 9y agoThat is not at all the assumption. The assumption is that you have some idea what you want a particular piece of code, method, or class to do and you write a test for a small piece of that and then after it passes you do that again. At no point does TDD suggest you should know all the tests you want upfront. In fact I would expect if you know that then you don't need TDD. TDD is about learning what tests you need and what you want the object or code to look like one small piece at a time.
- Udik 9y agoAren't you two saying the same thing? TDD works when the requirements are well defined upfront, which in the case of fixing existing system is provided by the system itself. But writing new programs is exploratory in nature: you don't know the exact requirements or shape your software will take, and TDD shifts a lot of precious money, time and attention from the what to the how. Which to me, I agree with you, is an engineering issue before being an economic one.
- deleted 9y ago[deleted]
- th4dv 9y ago> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software One of the main failings of TDD is that developers think they need to know all of the tests up front.
- ajeet_dhaliwal 9y agoI think the failures are due to a combination of engineering, economic and human reasons. TDD is counter intuitive to how most people think and work. They think of the features first, implement them and then once it's working want to have tests to ensure it stays working, especially with continuous delivery, testing is mostly about checking for regression as high frequency iterations are done. Even if things starts out ok, often at some point, some person in the team breaks the flow. You have to be quite rigid about procedure and in my experience most teams don't have the discipline for it. This is especially true when there are business pressures in some companies that put pressure on the development team. We've all experience instances where a program manager or producer asks for estimations and an inexperience developer only considers the coding time. To be honest, unless you're working at one of the top tech firms where everyone is interested in solid engineering and there's adequate revenue to enable this, I still find even basic automated testing in general is still a struggle to have in some companies. The reason for this is because the increased development cost is difficult to measure and so it's absorbed as a cost. They don't realize how much development is costing because they cannot compare it to the alternate reality where they did things a better way. Whereas feature issues are obvious to see for everyone.
- bm2yogi 9y ago"The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software." If that's what TDD is, then it can only fail. Fortunately, TDD does not require you to write all your tests up front (see "The Three Rules of TDD"). I spend a lot of my time interviewing people and talking with people about TDD. In the vast majority of cases, I find people who dislike TDD are often doing "Not TDD, but called TDD." If I were to write software that way, I too would consider that TDD did not live up to expectations.