10 ms·

TDD Doesn't Work

153 points by narfz 10y ago

liquidise 10y ago
Commenting on TDD stories here is historically a bad practice, but i'll add my input here. I have never let my teams go full TDD. The reason is that in all my experience, TDD sacrifices a lot of velocity for the sake of automated tests. When i hear about the reduction in total bugs injected, it is a "duh" moment. The fastest way to make a team inject 30% fewer bugs is to have them write 30% less code. That isn't snarky, it's true. Automated testing is one of the many tools available to software engineers. And it is a valuable one. Unfortunately, TDD is too much of a good thing. It relies so heavily on automated testing that it ventures far into the realm of diminishing returns. Once, in an argument about TDD, i said it was akin to having someone build a shed. But upon checking in on them, you saw they were using a hammer to smash screws into boards. When you ask them what they are doing, they tell you it is Hammer Drive Construction. It is perhaps overly harsh, but it reinforces the point: tools have a place. Automated tests really shine on mission critical logic that does not get rewritten often. Use it where it makes sense. I wouldn't recommend using it ubiquitously. Then again, i also recommend having fun coding. So i suppose the actual message here is: do what makes you successful, not what comments or studies say.
- keithnz 10y ago
  so what's your approach to ensuring the software you deploy is correct?
  - pka 10y ago
    Having a good type system would be a good start :)
    - koder2016 10y ago
      Don't upset the scripters ;).
    - keithnz 10y ago
      I agree :)
  - YZF 10y ago
    As someone I know likes to say, write better code? It's a given that software needs to be tested. The processes around that are a classical "it depends" question. Most likely any software you deploy will never be "correct" (whatever that means). The quality of that software depends on many variables and it's up to you to try and tweak them while optimizing for things like cost, time etc. Whether it's worthwhile to write the tests ahead of time or after the fact or to do them manually or automatically or any other permutation is just not a question that can be answered in a way that applies to all situations.
    - st3v3r 10y ago
      That's like saying, if you're poor, you should buy more money.
    - gozur88 10y ago
      >It's a given that software needs to be tested. The processes around that are a classical "it depends" question. Yes. What you need to do to ensure the correctness of flight software for a jet fighter is entirely different than what you need to do to ensure the correctness of an internal tool at your company that automates a task for which you already have a manual process.
  - GrinningFool 10y ago
    How does TDD prove its correctness? TDD suffers from the same limitations as the code - it generally only covers what you could think of. It's a powerful tool, but I think any belief that sufficient test coverage (in most common cases) actively proves correctness is misguided. In the general case, even full test coverage proves only that you've tested for the conditions you expect - but does nothing to verify the correctness of behavior in conditions you didn't expect.[1] To me the benefits of TDD are three-fold: 1. It makes you think of what you're building in more detail before you build it. 2. The methodology puts heavy emphasis on short test-code cycles. 3. (Applies to any methodology that emphasizes coverage) You end up with an acceptable-to-great regression suite[1], and anecdotally it seems people do a better job of at least ensuring tests exist when required to by the methodology. All of these things are equally possible without TDD. Short iterative cycles and additional forethought are perfectly possible without TDD, but they do require more discipline - it is harder to remember to stop after completing a small set of changes without a forcing mechanism. [1] rust is a possible exception here, still wrapping my head around it. [2] the value of this regression suite varies greatly from project to project. A hint that tests are of low value /potentially high cost can be seen when you're finding that minor internal changes either break a large number of tests or reduce coverage to noticeable degree. Particularly in absence of functional changes.
    - cjwoodall 10y ago
      Testing validates that the program is correct according to some base of assumptions. You run into this a whole lot in embedded systems, where mocking hardware is difficult. You can't feasibly unit test against real hardware (most of the time), so instead you unit test against your assumption of what the hardware does and verify you respond according to requirements. Has the benefit of proving correctness of your assumptions, which makes it easier to debug once you insert it in system and things inevitably are not 100% right. It gives you a way to reason about what your code does, what might be different, and then allows you to revise your assumptions and get your new solution in place and tested without the often long wait times to do manual testing on deeply embedded hardware. Sometimes traditional TDD is the answer, sometimes simulations are the answer, and sometimes you need to just get out of your chair and test it out. It is a tool!
    - 10y ago
  - GFK_of_xmaspast 10y ago
    "TDD" is not the same thing as "having unit and integration tests".
  - taneq 10y ago
    You can use automated unit and integration testing without doing "TDD". The overhead isn't even particularly high; you have to test any piece of code you write anyway, so you might as well put in a tiny bit of extra effort and have unit testing.
- deleted 10y ago
  [deleted]
- pooktrain 10y ago
  You say "do what makes you successful" after "I have never let my teams go full TDD". If someone on your team is most successful with TDD, do you still not allow it? re: writing 30% less code - I've found TDD can reduce my percentage of lines of code, as you suggested. Adherence to the "refactoring" part encourages that you reduce duplication, which in my experience has been easier to do with good test coverage.
  - liquidise 10y ago
    You are correct. My use of the singular "you" was more directed toward people in their own projects. For the purposes of a team at a company, you can think of it as a collective "us". We do not use TDD. I would say that the most successful teams i have been a part of focus not on automated testing but instead on other collective practices: informal code reviews, diff analysis of every commit, group discussion of database changes and collective manual testing of other's code. Many people point to the refactoring (or initial code organization) as a benefit of TDD. I find these other practices tend to inspire a more collective ownership of the system. Additionally, and more importantly, they spur a lot of conversation around how and why to organize code certain ways. These learning opportunities are probably the most valuable among young and growing teams.
    - pooktrain 10y ago
      Sure, these practices can help build a healthy engineering culture, and I agree with them all, especially collective manual testing. How do you keep people from doing TDD though? How do you even know they are doing it?
- crdoconnor 10y ago
  I find that automated tests shine pretty much all of the time, provided they're relatively cheap to build, cheap to maintain and not buggy. Where they fall down is when they're more expensive to build than the code under test and they produce false positives/negatives.
  - YZF 10y ago
    I agree. The problem is that almost inevitably they end up getting more and more expensive to build and maintain while at the same time becoming buggy ;) It's a difficult battle to win. I think you need self discipline to keep limiting yourself to an ever evolving subset that is the optimal ROI. This means over time removing tests that don't add as much value any more. Rewriting some other tests. etc. Human nature though is that these keep growing endlessly and become hard to manage just like any other part of software.
    - marcosdumay 10y ago
      Unit tests tend to get more expensive with time (but not always do). Other kinds of test behave differently.
    - spronkey 10y ago
      Hmm, I think this arises from a mindset that doesn't treat tests as part of the code that should be maintained and refactored. You always have to be removing and refactoring tests if you are changing your production code. Changed a sorting algorithm? Good, go and have a look at your tests to see whether there's anything that doesn't need to be tested anymore, or edge cases that need to be tested now that the algorithm has changed. Red -> Green -> Refactor
- gozur88 10y ago
  That's been my experience. I would add that TDD is the antithesis of "agile", since any changes you make to your product will require changes to the tests. Sometimes large changes.
  - flukus 10y ago
    That depends on the change and the tests. Ideally the changes to individual tests should be zero (for irrelevant tests) and changes to the test fixture should be minimal. Having said that, unit tests in the wild have a tendency to be abominably written, so I'm not surprised a lot of people get frustrated changes the tests.
  - alphanumeric0 10y ago
    In my experience, the large changes to my tests were a result of having to make large changes in product behavior.
  - unclebobmartin 10y ago
    Only if you design your tests that way. Tests are just software. If a change to one part of a software system requires massive changes to another part of that same system; then the system is poorly designed. Indeed, that may be the very definition of poor design. So if a change to your production code causes large changes to your test code, then one, or the other, or both are poorly designed. You have neglected the design. You have allowed couplings to proliferate.
- bunderbunder 10y ago
  I think that this all goes straight back to the old "mockist TDD vs classical TDD" debate.
  - convolvatron 10y ago
    could you elaborate? i think this is my ignorance. The one shop that I worked in that insisted on 'mocks' meant that i wrote some code, then ran that code on some inputs, recorded the outputs, and then wrote a harness which validated that those inputs matched the outputs. which meant that changes to the code might result in a failed mock, but didn't say anything about coverage or correctness. i can't imagine a more useless testing strategy. is that what mockist TDD is commonly understood to be?
    - bunderbunder 10y ago
      Yeah, that is. In what's often termed the "classic" approach, you instead lean toward writing more coarse-grained tests, and you don't shy away from integration tests. You don't avoid mocks, but you tend to prefer saving them for situations where it really is hard to force a collaborator to behave in a certain way. (I also try to stay on guard for the possibility that those situations are code smells indicating that your implementation is getting to be too complex and is due for a refactor.) IMO, the main argument against classical TDD is that you tend to get a suite of tests that runs more slowly and has more dependencies on external resources such as the database. IMO, the main arguments against mockist TDD are that you end up with test-induced design damage, and a brittle suite of tests that makes your codebase resistant to refactoring.
metaphorm 10y ago
tl;dr = someone did a study that used a methodology that confirmed that working in small chunks and writing tests as you go is good, but that it's not very important if you write the tests before the small chunk of code or after the small chunk of code.
- aikah 10y ago
  Aren't tests supposed to be a tool to help design API? in that perspective a test should be written first. The problem IMHO is the choice of methodology as there is several kind of tests. Some may be more time consuming when it comes to the set up.
  - digibo 10y ago
    Personally, I think unit tests shine best when you're designing an API. I can swing from hate to love and back about TDD in minutes, but when it comes to thinking about how your code will be used, unit tests (did we stop using that term?) are a tremendously useful tool I have. I guess if all code written could be seen as an API, TDD would be great, but that's not the world I live in.
    - mjevans 10y ago
      This is why I prefer free to implement open standards that are (hopefully) well designed and specific. Inter-operablility and interchangeability of parts means that it's possible to validate an implementation to at least some degree. The best example that I can think of off the top of my head is the OpenGL 4.4/4.5 work that is nearing conformance for the Mesa3D project ( https://en.wikipedia.org/wiki/Mesa_(computer_graphics) https://en.wikipedia.org/wiki/Mesa_(computer_graphics) ); while the functional coverage for the main modern drivers is 'complete' the official conformance testing has already resulted in some bug fixes and additional areas to focus on improving. That real life case study is yet another example of how an API and conformance tests built around that API result in better code and in a more consistent experience that isn't dependent upon a mono-culture implementation.
    - westurner 10y ago
kendallpark 10y ago
I'm definitely in the TDD-is-not-a-one-size-fits-all programming style camp and I'm glad to see a study that supports that conclusion. I was at Railsconf when DHH said his bit in 2014. My office and I followed the subsequent debates between him and Kent Beck (since my dev group was largely pro-TDD). Lots of anecdotal arguments. It's nice to see some more quantitative data on this! In my programming experience I've found that I prefer to write tests AFTER I do development of a new feature. Oftentimes the implementation is in such flux that continually updating the test as I go along is tedious and kills the creative flow. However, when it comes to fixing bugs in existing software, I find it more helpful to write a test that duplicates the bug FIRST, then code the solution. If anything, the reason to recommend TDD is simply to enforce writing tests to begin with. It's so easy to get a feature working and gloss over testing it. EDIT: What's up with liquidise's statement about commenting on TDD stories being bad practice? Do the TDD fanatics downvote to hell everything anti-TDD?
- gohrt 10y ago
  > Do the TDD fanatics downvote to hell everything anti-TDD? Did you notice the bias in that question?
  - kendallpark 10y ago
    I'm sorry the sarcastic inflection does not come through the online text.
- digibo 10y ago
  I read the article, but I haven't read the study. The article seems to show that the study is useless - it proves nothing, it's neither pro-TDD, nor anti-TDD. Taking a seemingly strong side of an argument supported by the study, but not commenting on the linked article itself, can explain the downvotes.
  - kendallpark 10y ago
    So I read the study. Martin Fowler is correct that you can't make huge generalizations from the study. But the importance is that someone is actually attempting to get quantitative evidence in an argument that has been largely anecdotal for the past several years. As flagrant as DHH's keynote was, back in the day, I think he was right to compare TDD to a fad diet. Yeah, there are people that will absolutely swear by it. But we really won't know until actual studies are done.
crdoconnor 10y ago
tl;dr the recent studies proved that that you're testing first or last doesn't matter, provided you're frequently flipping between writing a test and writing code. The author thinks that TDD is preferable because it helps you maintain discipline. I personally think it's worthwhile besides that because it means you design the API before implementing, meaning it is cheaper to fix API design mistakes. IIRC this aspect wasn't actually tested in the studies (API signatures were given up front).
- AnimalMuppet 10y ago
  And that's one of the things TDD gives you - as you write the test, you have to use the API. If that's painful or even just awkward, it's telling you something...
  - marcosdumay 10y ago
    Except that it is not real use. Might be better than nothing, but if you are designing a reusable API, you'll be better using it on some real code.
    - jacques_chester 10y ago
      > Might be better than nothing, but if you are designing a reusable API, you'll be better using it on some real code. Agreed, this is why tests should be driven from the outside in. But at the unit level, tests are often the first contact with production code. Tests are dumb and setting up complicated stateful worlds is painful and tedious. It becomes easier to simplify the production code to simplify the test code, than to just write the first thing that comes to mind.
    - crdoconnor 10y ago
      >Might be better than nothing, but if you are designing a reusable API, you'll be better using it on some real code. This is generally why I take the top down approach to coding: 1) Write ultra high level test 2) Write code that implements it. 3) In that code if I need something lower level, write the API that I want 4) Write a lower level test that mirrors what I just wrote in real code. 5) Implement the code that passes that test & use it at the higher level. etc.
tooFastFuckYou 10y ago
Junk. Rambles on about "TDD" without ever saying WTF it is.
maxxxxx 10y ago
You can also write "Methodology X doesn't work always". All methodologies work well for some situation and for others they don't. In my view TDD is great for a lot simple things and algorithms and you can structure your code in a way that most of the code is inherently testable. But when things are so complex that you don't even know the correct architecture upfront, TDD is a killer.
- spronkey 10y ago
  If things are that complex, sounds like you need to be doing some discovery work (spikes) first to break the problem down. Then you can use TDD again :-D. So, I guess you're right - if you don't know wtf you're supposed to be doing, TDD is a killer. But then, so is anything else.
linker3000 10y ago
For those like me who enjoy HN but aren't s/w developers: https://en.wikipedia.org/wiki/Test-driven_development https://en.wikipedia.org/wiki/Test-driven_development
nercury 10y ago
TDD presents a paradox that requires split-brain thinking: when writing a test, you pretend to forget what branch of code you are introducing, and when writing a branch, you pretend to forget you already knew the solution. It is annoying as hell. You CAN indeed cover all your branches with tests afterwards. You can even give that a fancier name, like "Exploratory Testing". Of course it may be more boring or tedious, but is a perfectly valid way to ensure coverage when needed. TDD was great for popularizing writing test first; However I much prefer the methodology called CABWT - Cover All Branches With Tests. Let the devs choose the way to do it, because not everyone likes these pretend games.
- lisivka 10y ago
  TDD requires you to write FUNCTIONAL test first, not unit tests you are talking about.
  - nercury 10y ago
    I was commenting on the methodology as I heard and watched it explained by the author (Robert C Martin), as well as the way it was presented in his videos. TDD workflow is fine; it's not thinking about the pink elephant (the source code) idea that bugs me.
    - lisivka 10y ago
      Robert Martin is author of Agile manifesto. https://www.quora.com/Why-does-Kent-Beck-refer-to-the-rediscovery-of-test-driven-development https://www.quora.com/Why-does-Kent-Beck-refer-to-the-redisc... The original description of TDD was in an ancient book about programming. It said you take the input tape, manually type in the output tape you expect, then program until the actual output tape matches the expected output.
      - westurner 10y ago
        +1. TDD could be considered as a derivation of the Scientific Method (Hypothesis Testing). https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Hypothesis https://en.wikipedia.org/wiki/Hypothesis Test first isolates out a null hypothesis (that the test already passed); but not that it passes/fails because of some other chance variation (e.g. hash randomization and unordered maps). https://en.wikipedia.org/wiki/Null_hypothesis https://en.wikipedia.org/wiki/Null_hypothesis ... https://en.wikipedia.org/wiki/Test-driven_development https://en.wikipedia.org/wiki/Test-driven_development
c3RlcGhlbnI_ 10y ago
Oh man, it is really funny that he ends it with telling people to read the study. To clarify the linked study is attempting to replicate https://dl.acm.org/citation.cfm?id=1070834 https://dl.acm.org/citation.cfm?id=1070834, THE seminal study in Test Driven Development. Well to be more precise it was replicating an existing replication of that study which failed to replicate the original results. They were trying to modify the design so as to account for issues in the experimental design that may have led to the replicated study being inconclusive. This is significant because if you were not aware of the failed replication, and believed that TDD was supported scientifically as more productive because of that original study, then you SHOULD be reconsidering its place in your development process. If that isn't the case your opinion is unchanged by these particular results(even in the article inspiring this one the author admits that their opinion was already based on a much more thorough analysis, see: http://neverworkintheory.org/2016/10/05/test-driven-development.html http://neverworkintheory.org/2016/10/05/test-driven-developm...). Now what I want to know is why people insist on writing articles in this awful conversation format. It wastes a lot of words to make a simple argument poorly.
- spronkey 10y ago
  Probably because they have a sense of humour, and the conversational style makes it more entertaining to read...?
  - c3RlcGhlbnI_ 10y ago
    Is it entertaining or funny if you identify with the character who is talking down? I mean this particular example doesn't seem to contain any jokes from my reading.
- unclebobmartin 10y ago
  I use that style, from time to time, because I like it. ;-)
cocktailpeanuts 10y ago
Before I go in, I will state that in 99% of the times I'm a TDD hater. Actually I don't even like writing tests after the fact because I just like to build and move on. I could never understand why the ruby on rails tutorial insisted on walking newbies through TDD and skipped all chapters where they start talking about tests when I started learning rails. I still think it's a bad idea to make newbies do all the weird TDD stuff when they don't even know how to build something. I'm so opinionated about this that most people around me know this. And in most cases it works without needing to write any tests. And even if something fails, I can quickly patch it. As long as I wrote the app in a nicely modular way, I've not had much problem. That said, right now I'm working on writing a JS library. And believe it or not, I AM doing TDD right now. I can't believe it myself. I think in cases where the logic involves a lot of intricate details, it's impossible for me to write something without writing tests. I'm not talking about simple web apps. I'm talking about stuff like: template engine, parser, etc. My current setup: I write a test and document it before I write a function. That way I don't get carried away while implementing and know exactly what I'm trying to build. Then I write another function that utilizes that function I just wrote, and so forth. This way I know when the next function doesn't work for some reason I know exactly where something went wrong. Instead going back and debugging every single function used along the way, I know it's the most recent one that's causing the problem. So my conclusion: you probably don't need to write tests for all your stuff, but there are indeed cases where you will NOT be able to proceed without writing tests.
- bitL 10y ago
  Once you release a product that will be used by many customers and developed by many people throughout its lifecycle, which come and go as the time passes, you won't be able to maintain/extend it without a proper testing suite. It's not only about complexity, but also about maintainability. Some tests will also rot in time.
  - cocktailpeanuts 10y ago
    Agreed, in my case what ends up happening is I start out with no tests but soon the project becomes huge and i have to start writing tests if I want to push stuff without fear. The thing is, most large companies have a QA team so this fear is not super tangible to many developers. And small startups are more focused on building stuff quickly (which they should be). I think this is why this topic has been polarizing. Some people feel the need and some people do, depending on which role you're playing in your organization. Nowadays interestingly, even the large companies are moving towards more testing because they can cut QA costs that way.
ambrop7 10y ago
I must be one of the very few people who can write working and mostly bugless code and without writing any kind of test. Writing tests feels like the most wasteful and possibly harmful thing to me (like by people forcing dependency injection etc. where otherwise unneeded). I don't really know what to think of the situation? Is this how it has always been? Do most software engineers really have no idea what they're doing?
- TylerE 10y ago
  How do you KNOW your code is working and bugless?
  - ambrop7 10y ago
    That's a good question. There are different ways: - Play around with the program intelignetly and observe no bugs (you wouldn't believe how many bugs I've found by that method, that have been missed by very formal verification processes). - Don't call it done until you haven't proven to yourself that the code has no bugs. This is an informal process of self-code-review but which involves quite rigorous thinking about the behavior of the code. Often assertions are involved (which all have to be proved).
    - lisivka 10y ago
      How you prove to your peer that you done all of above? How your code is able to pass peer reviews??
      - ambrop7 10y ago
        Well, I haven't claimed to be able to to that, but to write mostly bugless code :) Though, writing insightful comments in the code helps. If anyone has doubt, you explain in a friendly way why they're wrong (if they are). Though, not all reviewers will study the code suffciently to find possible bugs, and also many findings are not the right/wrong kind but more about style/architecture.
awinter-py 10y ago
Careful -- in these studies the subjects are writing tests before writing code. In practice there are 'test-heavy' devs who use factory data and the test suite to run skeleton code with crashpoints, and switch actively between test and imp files. This has tests & implementation being written in parallel vs strict TDD which has us finishing tests before writing program logic. Most test suites depend not just on functional requirements but also on implementation details, so it seems obvious that tests-before-logic development is inefficient.
- lisivka 10y ago
  TDD requires to write FUNCTIONAL test case first for every new feature. Functional test case should not depend on implementation. Integrational and unit tests are.
bitL 10y ago
I thought that TDD morphed into ending up with a regression/integration/conformation test suite instead of using tests as specifications written prior to writing products. And even 100,000s of tests won't help you in very advanced applications like cloud/cluster infrastructure as sometimes it's simply too difficult if not impossible to come up with tests (imagine observer effect when your cluster deadlock happens only in certain rare nanosecond windows and adding a testing framework will make you miss those windows and the problem never happens) and people with mental capacity capable of writing them (e.g. Google/FB-level) are better utilized in writing the product itself.
- jacques_chester 10y ago
  How do the people who write them know that they work?
  - westurner 10y ago
    > How do the people who write them know that they work? Test first isolates that a given test doesn't already pass (without any additional code). Test after (but before committing) also seems to require a more thorough critical analysis. And then someone finally fuzzes the code.
- lisivka 10y ago
  Why you think that debugging and fixing deadlock in cluster in production (of multibillion business) is easier and cheaper than writing of functional test case? Maybe you just prefer trips to angry customer versus boring office work. :-/ http://www.reuters.com/article/us-nasdaq-halt-tapec-idUSBRE97L0V420130822 http://www.reuters.com/article/us-nasdaq-halt-tapec-idUSBRE9...
  - bitL 10y ago
    The thing is that there are problems we simply can't solve in theory nor in practice, yet we use approximate solutions all the time - and that is the case of advanced distributed algorithms. In theory, we simply can't handle real-world asynchronous systems. And when we pretend we have partially synchronous systems and build abstractions around them, they aren't 100% working. Now add in some complex bugs (like getting a distributed deadlock in transacted system involving exactly 7 nodes but not less nor more) and you might start understanding why functional test case might not really be an option to avoid these issues (you can obviously write them but they won't really help you). I worked on such a system, we had 100,000s of tests yet they were clustered around known issues and not issues that happened when e.g. a node went down and up, data were out of sync and sockets between nodes were becoming full due to OS' performance limitations. And moreover, many of these issues start showing up only when you push throughput to the max, e.g. during trading spikes etc. and adding a test that checks invariants would lower the throughput and those issues simply won't show up anymore.
emmelaich 10y ago
(to nobody in particular) Please the article before commenting. It's not totally clear from reading some of the comments that people have actually read the article. It's a good one, please do.
gravypod 10y ago
I prefer to rather then write tests plan out the interactions between all components in large projects. This will show you how all the pieces interact and what cases to need to handle in each functional unit. After this, I sit down an write all the code. After I know the organization of the source, I write out each functional unit of the code one at a time. As I go, I write each bit of test code for my source. After this I integrate every function unit. If a change is needed, I go back to the drawing board and find a better overall organization. This happens often due to either performance constraints or the need to abstract a section further. After this I'd consider embedding a unit test suite. Works great for small to medium projects.
ivanhoe 10y ago
Author is only partially right about TLD being as "doing TDD in your head", since it's (at least for me) in a much more abstract form of a general idea, a concept, of what I want to achieve. When using TDD you need to come up with the very specific results that you will test and you need then to implement those specific tests, to the last line of code. This means that if you make any changes to the logic afterwords, you need to throw away your pre-written tests and write new ones, the time spent on writing them was wasted. TLD is much more flexible and easier to update, no code is thrown away if you change something. Before I start I just need to decide what I'm trying to solve with my current block of code, and then I later write a test to check if I did it properly. Then I do the next block of logic, and the next test. Since code blocks are directly related to the steps in my logic, it's very natural to come up with the tests for them, just test if the things work as you planned it. If in the middle of that work I realize that I need to do something in a completely different way, there's no pre-written tests, so no time was wasted on coding tests that were never going to be used. And, at least to me, this kind of situations happen a lot, I often refactor and improve things as I work on them, so for me TLD is much more suitable approach.
hawkice 10y ago
Let's say I have a new theory, called Understanding Driven Development. The system says: It's a bug if someone needs to change code and they, at any moment, see code they don't understand. Stumbled into the wrong place? Bug filed for better notes on organization. The code you need to touch not understood? Understand what you see before you make a single change. If you change code and don't update docs, or documentation and code out of sync? It's a bug, and changing one to match the other _without detailed understanding_ is a bug too! Now, that seems reasonable. And if a study comes out and says people can't make program changes faster, on average, when participants are given a bit of code identical, but with more (accurate and non-trivial) comments, that doesn't mean UDD doesn't work. It doesn't test it on real, full size applications. The code was the same, despite clarity of code is one of the goals of UDD -- one of the core claims is that UDD gets you better code to begin with. It focuses on a tiny test of something not necessarily core to the UDD mindset. But it's evidence that at least one claim I've made is false. In fact, that study would be enough for me to throw that idea set into the garbage.
mannykannot 10y ago
Test-first was promoted as being the secret sauce that made TDD so much better than anything else, so this is something of a qualified vindication, but I do think (from my own experience) that writing down what I am thinking does help me see flaws that I had overlooked.
deleted 10y ago
[deleted]
mannykannot 10y ago
The author was going somewhere when he began writing about what a developer is thinking about, but, perhaps because he was focused on vindicating TDD, he did not arrive there. A developer who is writing unit tests must have a good idea of the purpose of the target of the tests, so she is thinking about requirements. Furthermore, if she is writing unit tests for small components (which will often be the case on account of everything being done in short cycles) then a lot of that purpose is contingent on other aspects of the design and how it is all supposed to work together: in other words, she is thinking about design. If you don't spend some time thinking ahead about big-picture requirements and design issues, you are in danger of going a long way down a dead end.
adamconroy 10y ago
That blog post could pretty much apply the same arguments to itself. And who knows if Bob's experience is simply correlation not causation. Perhaps Bob is just a smart, meticulous engineer, and it wouldn't matter how he went about his dev work, the quality may be good regardless.
plinkplonk 10y ago
In all discussions about TDD, it is important to distinguish between having having an automated test suite for your code which is run frequently, and writing your code test first - which is what TDD is, by definition. It is possible to advocate for the former, while thinking the latter is consultantware snake oil. (my position, fwiw)
- lobut 10y ago
  Yeah, I agree. TDD is also, you cannot write a single line of code without writing a failing unit test though. Well, I'm not a fan of that either. I mean, sometimes I like writing an acceptance test to begin with and work inwards. People sometimes interchangeably use "TDD" for "testing". Also, just because you do TDD, doesn't mean your code can be great, I've seen people assert pointless things and the unit has gotten so small that people now define them as methods in classes. Which I also think can lead to some crazy maintenance suite of tests. If you're interested, this was a fun discussion about TDD between two professionals: https://www.youtube.com/watch?v=KtHQGs3zFAM https://www.youtube.com/watch?v=KtHQGs3zFAM Jim Coplien and Uncle Bob.
stevenalowe 10y ago
good article; would like to add that a study based on "21 graduate students" is hardly representative of the software developer population...
ckastner 10y ago
Previous 300+ comment thread which referenced the actual paper, not a blog post about it: https://news.ycombinator.com/item?id=12740456 https://news.ycombinator.com/item?id=12740456
nunez 10y ago
I work at ThoughtWorks. TDD is central to everything that we do. That said, like anything else, TDD done to an extreme is probably a bad thing (too much time spent on tests rather than implementation) and it not being done at all is also usually bad (too much time fixing bugs that could have been caught by tests written beforehand). Balance is key.

153 points by narfz 10y ago

Commenting on TDD stories here is historically a bad practice, but i'll add my input here. I have never let my teams go full TDD. The reason is that in all my experience, TDD sacrifices a lot of velocity for the sake of automated tests. When i hear about the reduction in total bugs injected, it is a "duh" moment. The fastest way to make a team inject 30% fewer bugs is to have them write 30% less code. That isn't snarky, it's true. Automated testing is one of the many tools available to software engineers. And it is a valuable one. Unfortunately, TDD is too much of a good thing. It relies so heavily on automated testing that it ventures far into the realm of diminishing returns. Once, in an argument about TDD, i said it was akin to having someone build a shed. But upon checking in on them, you saw they were using a hammer to smash screws into boards. When you ask them what they are doing, they tell you it is Hammer Drive Construction. It is perhaps overly harsh, but it reinforces the point: tools have a place. Automated tests really shine on mission critical logic that does not get rewritten often. Use it where it makes sense. I wouldn't recommend using it ubiquitously. Then again, i also recommend having fun coding. So i suppose the actual message here is: do what makes you successful, not what comments or studies say.

keithnz 10y ago
so what's your approach to ensuring the software you deploy is correct?
- pka 10y ago
  Having a good type system would be a good start :)
  - koder2016 10y ago
    Don't upset the scripters ;).
  - keithnz 10y ago
    I agree :)
- YZF 10y ago
  As someone I know likes to say, write better code? It's a given that software needs to be tested. The processes around that are a classical "it depends" question. Most likely any software you deploy will never be "correct" (whatever that means). The quality of that software depends on many variables and it's up to you to try and tweak them while optimizing for things like cost, time etc. Whether it's worthwhile to write the tests ahead of time or after the fact or to do them manually or automatically or any other permutation is just not a question that can be answered in a way that applies to all situations.
  - st3v3r 10y ago
    That's like saying, if you're poor, you should buy more money.
  - gozur88 10y ago
    >It's a given that software needs to be tested. The processes around that are a classical "it depends" question. Yes. What you need to do to ensure the correctness of flight software for a jet fighter is entirely different than what you need to do to ensure the correctness of an internal tool at your company that automates a task for which you already have a manual process.
- GrinningFool 10y ago
  How does TDD prove its correctness? TDD suffers from the same limitations as the code - it generally only covers what you could think of. It's a powerful tool, but I think any belief that sufficient test coverage (in most common cases) actively proves correctness is misguided. In the general case, even full test coverage proves only that you've tested for the conditions you expect - but does nothing to verify the correctness of behavior in conditions you didn't expect.[1] To me the benefits of TDD are three-fold: 1. It makes you think of what you're building in more detail before you build it. 2. The methodology puts heavy emphasis on short test-code cycles. 3. (Applies to any methodology that emphasizes coverage) You end up with an acceptable-to-great regression suite[1], and anecdotally it seems people do a better job of at least ensuring tests exist when required to by the methodology. All of these things are equally possible without TDD. Short iterative cycles and additional forethought are perfectly possible without TDD, but they do require more discipline - it is harder to remember to stop after completing a small set of changes without a forcing mechanism. [1] rust is a possible exception here, still wrapping my head around it. [2] the value of this regression suite varies greatly from project to project. A hint that tests are of low value /potentially high cost can be seen when you're finding that minor internal changes either break a large number of tests or reduce coverage to noticeable degree. Particularly in absence of functional changes.
  - cjwoodall 10y ago
    Testing validates that the program is correct according to some base of assumptions. You run into this a whole lot in embedded systems, where mocking hardware is difficult. You can't feasibly unit test against real hardware (most of the time), so instead you unit test against your assumption of what the hardware does and verify you respond according to requirements. Has the benefit of proving correctness of your assumptions, which makes it easier to debug once you insert it in system and things inevitably are not 100% right. It gives you a way to reason about what your code does, what might be different, and then allows you to revise your assumptions and get your new solution in place and tested without the often long wait times to do manual testing on deeply embedded hardware. Sometimes traditional TDD is the answer, sometimes simulations are the answer, and sometimes you need to just get out of your chair and test it out. It is a tool!
  - 10y ago
- GFK_of_xmaspast 10y ago
  "TDD" is not the same thing as "having unit and integration tests".
- taneq 10y ago
  You can use automated unit and integration testing without doing "TDD". The overhead isn't even particularly high; you have to test any piece of code you write anyway, so you might as well put in a tiny bit of extra effort and have unit testing.
deleted 10y ago
[deleted]
pooktrain 10y ago
You say "do what makes you successful" after "I have never let my teams go full TDD". If someone on your team is most successful with TDD, do you still not allow it? re: writing 30% less code - I've found TDD can reduce my percentage of lines of code, as you suggested. Adherence to the "refactoring" part encourages that you reduce duplication, which in my experience has been easier to do with good test coverage.
- liquidise 10y ago
  You are correct. My use of the singular "you" was more directed toward people in their own projects. For the purposes of a team at a company, you can think of it as a collective "us". We do not use TDD. I would say that the most successful teams i have been a part of focus not on automated testing but instead on other collective practices: informal code reviews, diff analysis of every commit, group discussion of database changes and collective manual testing of other's code. Many people point to the refactoring (or initial code organization) as a benefit of TDD. I find these other practices tend to inspire a more collective ownership of the system. Additionally, and more importantly, they spur a lot of conversation around how and why to organize code certain ways. These learning opportunities are probably the most valuable among young and growing teams.
  - pooktrain 10y ago
    Sure, these practices can help build a healthy engineering culture, and I agree with them all, especially collective manual testing. How do you keep people from doing TDD though? How do you even know they are doing it?
crdoconnor 10y ago
I find that automated tests shine pretty much all of the time, provided they're relatively cheap to build, cheap to maintain and not buggy. Where they fall down is when they're more expensive to build than the code under test and they produce false positives/negatives.
- YZF 10y ago
  I agree. The problem is that almost inevitably they end up getting more and more expensive to build and maintain while at the same time becoming buggy ;) It's a difficult battle to win. I think you need self discipline to keep limiting yourself to an ever evolving subset that is the optimal ROI. This means over time removing tests that don't add as much value any more. Rewriting some other tests. etc. Human nature though is that these keep growing endlessly and become hard to manage just like any other part of software.
  - marcosdumay 10y ago
    Unit tests tend to get more expensive with time (but not always do). Other kinds of test behave differently.
  - spronkey 10y ago
    Hmm, I think this arises from a mindset that doesn't treat tests as part of the code that should be maintained and refactored. You always have to be removing and refactoring tests if you are changing your production code. Changed a sorting algorithm? Good, go and have a look at your tests to see whether there's anything that doesn't need to be tested anymore, or edge cases that need to be tested now that the algorithm has changed. Red -> Green -> Refactor
gozur88 10y ago
That's been my experience. I would add that TDD is the antithesis of "agile", since any changes you make to your product will require changes to the tests. Sometimes large changes.
- flukus 10y ago
  That depends on the change and the tests. Ideally the changes to individual tests should be zero (for irrelevant tests) and changes to the test fixture should be minimal. Having said that, unit tests in the wild have a tendency to be abominably written, so I'm not surprised a lot of people get frustrated changes the tests.
- alphanumeric0 10y ago
  In my experience, the large changes to my tests were a result of having to make large changes in product behavior.
- unclebobmartin 10y ago
  Only if you design your tests that way. Tests are just software. If a change to one part of a software system requires massive changes to another part of that same system; then the system is poorly designed. Indeed, that may be the very definition of poor design. So if a change to your production code causes large changes to your test code, then one, or the other, or both are poorly designed. You have neglected the design. You have allowed couplings to proliferate.
bunderbunder 10y ago
I think that this all goes straight back to the old "mockist TDD vs classical TDD" debate.
- convolvatron 10y ago
  could you elaborate? i think this is my ignorance. The one shop that I worked in that insisted on 'mocks' meant that i wrote some code, then ran that code on some inputs, recorded the outputs, and then wrote a harness which validated that those inputs matched the outputs. which meant that changes to the code might result in a failed mock, but didn't say anything about coverage or correctness. i can't imagine a more useless testing strategy. is that what mockist TDD is commonly understood to be?
  - bunderbunder 10y ago
    Yeah, that is. In what's often termed the "classic" approach, you instead lean toward writing more coarse-grained tests, and you don't shy away from integration tests. You don't avoid mocks, but you tend to prefer saving them for situations where it really is hard to force a collaborator to behave in a certain way. (I also try to stay on guard for the possibility that those situations are code smells indicating that your implementation is getting to be too complex and is due for a refactor.) IMO, the main argument against classical TDD is that you tend to get a suite of tests that runs more slowly and has more dependencies on external resources such as the database. IMO, the main arguments against mockist TDD are that you end up with test-induced design damage, and a brittle suite of tests that makes your codebase resistant to refactoring.

metaphorm 10y ago

tl;dr = someone did a study that used a methodology that confirmed that working in small chunks and writing tests as you go is good, but that it's not very important if you write the tests before the small chunk of code or after the small chunk of code.

aikah 10y ago
Aren't tests supposed to be a tool to help design API? in that perspective a test should be written first. The problem IMHO is the choice of methodology as there is several kind of tests. Some may be more time consuming when it comes to the set up.
- digibo 10y ago
  Personally, I think unit tests shine best when you're designing an API. I can swing from hate to love and back about TDD in minutes, but when it comes to thinking about how your code will be used, unit tests (did we stop using that term?) are a tremendously useful tool I have. I guess if all code written could be seen as an API, TDD would be great, but that's not the world I live in.
  - mjevans 10y ago
    This is why I prefer free to implement open standards that are (hopefully) well designed and specific. Inter-operablility and interchangeability of parts means that it's possible to validate an implementation to at least some degree. The best example that I can think of off the top of my head is the OpenGL 4.4/4.5 work that is nearing conformance for the Mesa3D project ( https://en.wikipedia.org/wiki/Mesa_(computer_graphics) https://en.wikipedia.org/wiki/Mesa_(computer_graphics) ); while the functional coverage for the main modern drivers is 'complete' the official conformance testing has already resulted in some bug fixes and additional areas to focus on improving. That real life case study is yet another example of how an API and conformance tests built around that API result in better code and in a more consistent experience that isn't dependent upon a mono-culture implementation.
  - westurner 10y ago

kendallpark 10y ago

I'm definitely in the TDD-is-not-a-one-size-fits-all programming style camp and I'm glad to see a study that supports that conclusion. I was at Railsconf when DHH said his bit in 2014. My office and I followed the subsequent debates between him and Kent Beck (since my dev group was largely pro-TDD). Lots of anecdotal arguments. It's nice to see some more quantitative data on this! In my programming experience I've found that I prefer to write tests AFTER I do development of a new feature. Oftentimes the implementation is in such flux that continually updating the test as I go along is tedious and kills the creative flow. However, when it comes to fixing bugs in existing software, I find it more helpful to write a test that duplicates the bug FIRST, then code the solution. If anything, the reason to recommend TDD is simply to enforce writing tests to begin with. It's so easy to get a feature working and gloss over testing it. EDIT: What's up with liquidise's statement about commenting on TDD stories being bad practice? Do the TDD fanatics downvote to hell everything anti-TDD?

gohrt 10y ago
> Do the TDD fanatics downvote to hell everything anti-TDD? Did you notice the bias in that question?
- kendallpark 10y ago
  I'm sorry the sarcastic inflection does not come through the online text.
digibo 10y ago
I read the article, but I haven't read the study. The article seems to show that the study is useless - it proves nothing, it's neither pro-TDD, nor anti-TDD. Taking a seemingly strong side of an argument supported by the study, but not commenting on the linked article itself, can explain the downvotes.
- kendallpark 10y ago
  So I read the study. Martin Fowler is correct that you can't make huge generalizations from the study. But the importance is that someone is actually attempting to get quantitative evidence in an argument that has been largely anecdotal for the past several years. As flagrant as DHH's keynote was, back in the day, I think he was right to compare TDD to a fad diet. Yeah, there are people that will absolutely swear by it. But we really won't know until actual studies are done.

crdoconnor 10y ago

tl;dr the recent studies proved that that you're testing first or last doesn't matter, provided you're frequently flipping between writing a test and writing code. The author thinks that TDD is preferable because it helps you maintain discipline. I personally think it's worthwhile besides that because it means you design the API before implementing, meaning it is cheaper to fix API design mistakes. IIRC this aspect wasn't actually tested in the studies (API signatures were given up front).

AnimalMuppet 10y ago
And that's one of the things TDD gives you - as you write the test, you have to use the API. If that's painful or even just awkward, it's telling you something...
- marcosdumay 10y ago
  Except that it is not real use. Might be better than nothing, but if you are designing a reusable API, you'll be better using it on some real code.
  - jacques_chester 10y ago
    > Might be better than nothing, but if you are designing a reusable API, you'll be better using it on some real code. Agreed, this is why tests should be driven from the outside in. But at the unit level, tests are often the first contact with production code. Tests are dumb and setting up complicated stateful worlds is painful and tedious. It becomes easier to simplify the production code to simplify the test code, than to just write the first thing that comes to mind.
  - crdoconnor 10y ago
    >Might be better than nothing, but if you are designing a reusable API, you'll be better using it on some real code. This is generally why I take the top down approach to coding: 1) Write ultra high level test 2) Write code that implements it. 3) In that code if I need something lower level, write the API that I want 4) Write a lower level test that mirrors what I just wrote in real code. 5) Implement the code that passes that test & use it at the higher level. etc.

tooFastFuckYou 10y ago

Junk. Rambles on about "TDD" without ever saying WTF it is.

maxxxxx 10y ago

You can also write "Methodology X doesn't work always". All methodologies work well for some situation and for others they don't. In my view TDD is great for a lot simple things and algorithms and you can structure your code in a way that most of the code is inherently testable. But when things are so complex that you don't even know the correct architecture upfront, TDD is a killer.

spronkey 10y ago
If things are that complex, sounds like you need to be doing some discovery work (spikes) first to break the problem down. Then you can use TDD again :-D. So, I guess you're right - if you don't know wtf you're supposed to be doing, TDD is a killer. But then, so is anything else.

linker3000 10y ago

For those like me who enjoy HN but aren't s/w developers: https://en.wikipedia.org/wiki/Test-driven_development https://en.wikipedia.org/wiki/Test-driven_development

nercury 10y ago

TDD presents a paradox that requires split-brain thinking: when writing a test, you pretend to forget what branch of code you are introducing, and when writing a branch, you pretend to forget you already knew the solution. It is annoying as hell. You CAN indeed cover all your branches with tests afterwards. You can even give that a fancier name, like "Exploratory Testing". Of course it may be more boring or tedious, but is a perfectly valid way to ensure coverage when needed. TDD was great for popularizing writing test first; However I much prefer the methodology called CABWT - Cover All Branches With Tests. Let the devs choose the way to do it, because not everyone likes these pretend games.

lisivka 10y ago
TDD requires you to write FUNCTIONAL test first, not unit tests you are talking about.
- nercury 10y ago
  I was commenting on the methodology as I heard and watched it explained by the author (Robert C Martin), as well as the way it was presented in his videos. TDD workflow is fine; it's not thinking about the pink elephant (the source code) idea that bugs me.
  - lisivka 10y ago
    Robert Martin is author of Agile manifesto. https://www.quora.com/Why-does-Kent-Beck-refer-to-the-rediscovery-of-test-driven-development https://www.quora.com/Why-does-Kent-Beck-refer-to-the-redisc... The original description of TDD was in an ancient book about programming. It said you take the input tape, manually type in the output tape you expect, then program until the actual output tape matches the expected output.
    - westurner 10y ago
      +1. TDD could be considered as a derivation of the Scientific Method (Hypothesis Testing). https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Hypothesis https://en.wikipedia.org/wiki/Hypothesis Test first isolates out a null hypothesis (that the test already passed); but not that it passes/fails because of some other chance variation (e.g. hash randomization and unordered maps). https://en.wikipedia.org/wiki/Null_hypothesis https://en.wikipedia.org/wiki/Null_hypothesis ... https://en.wikipedia.org/wiki/Test-driven_development https://en.wikipedia.org/wiki/Test-driven_development

c3RlcGhlbnI_ 10y ago

Oh man, it is really funny that he ends it with telling people to read the study. To clarify the linked study is attempting to replicate https://dl.acm.org/citation.cfm?id=1070834 https://dl.acm.org/citation.cfm?id=1070834, THE seminal study in Test Driven Development. Well to be more precise it was replicating an existing replication of that study which failed to replicate the original results. They were trying to modify the design so as to account for issues in the experimental design that may have led to the replicated study being inconclusive. This is significant because if you were not aware of the failed replication, and believed that TDD was supported scientifically as more productive because of that original study, then you SHOULD be reconsidering its place in your development process. If that isn't the case your opinion is unchanged by these particular results(even in the article inspiring this one the author admits that their opinion was already based on a much more thorough analysis, see: http://neverworkintheory.org/2016/10/05/test-driven-development.html http://neverworkintheory.org/2016/10/05/test-driven-developm...). Now what I want to know is why people insist on writing articles in this awful conversation format. It wastes a lot of words to make a simple argument poorly.

spronkey 10y ago
Probably because they have a sense of humour, and the conversational style makes it more entertaining to read...?
- c3RlcGhlbnI_ 10y ago
  Is it entertaining or funny if you identify with the character who is talking down? I mean this particular example doesn't seem to contain any jokes from my reading.
unclebobmartin 10y ago
I use that style, from time to time, because I like it. ;-)

cocktailpeanuts 10y ago

Before I go in, I will state that in 99% of the times I'm a TDD hater. Actually I don't even like writing tests after the fact because I just like to build and move on. I could never understand why the ruby on rails tutorial insisted on walking newbies through TDD and skipped all chapters where they start talking about tests when I started learning rails. I still think it's a bad idea to make newbies do all the weird TDD stuff when they don't even know how to build something. I'm so opinionated about this that most people around me know this. And in most cases it works without needing to write any tests. And even if something fails, I can quickly patch it. As long as I wrote the app in a nicely modular way, I've not had much problem. That said, right now I'm working on writing a JS library. And believe it or not, I AM doing TDD right now. I can't believe it myself. I think in cases where the logic involves a lot of intricate details, it's impossible for me to write something without writing tests. I'm not talking about simple web apps. I'm talking about stuff like: template engine, parser, etc. My current setup: I write a test and document it before I write a function. That way I don't get carried away while implementing and know exactly what I'm trying to build. Then I write another function that utilizes that function I just wrote, and so forth. This way I know when the next function doesn't work for some reason I know exactly where something went wrong. Instead going back and debugging every single function used along the way, I know it's the most recent one that's causing the problem. So my conclusion: you probably don't need to write tests for all your stuff, but there are indeed cases where you will NOT be able to proceed without writing tests.

bitL 10y ago
Once you release a product that will be used by many customers and developed by many people throughout its lifecycle, which come and go as the time passes, you won't be able to maintain/extend it without a proper testing suite. It's not only about complexity, but also about maintainability. Some tests will also rot in time.
- cocktailpeanuts 10y ago
  Agreed, in my case what ends up happening is I start out with no tests but soon the project becomes huge and i have to start writing tests if I want to push stuff without fear. The thing is, most large companies have a QA team so this fear is not super tangible to many developers. And small startups are more focused on building stuff quickly (which they should be). I think this is why this topic has been polarizing. Some people feel the need and some people do, depending on which role you're playing in your organization. Nowadays interestingly, even the large companies are moving towards more testing because they can cut QA costs that way.

ambrop7 10y ago

I must be one of the very few people who can write working and mostly bugless code and without writing any kind of test. Writing tests feels like the most wasteful and possibly harmful thing to me (like by people forcing dependency injection etc. where otherwise unneeded). I don't really know what to think of the situation? Is this how it has always been? Do most software engineers really have no idea what they're doing?

TylerE 10y ago
How do you KNOW your code is working and bugless?
- ambrop7 10y ago
  That's a good question. There are different ways: - Play around with the program intelignetly and observe no bugs (you wouldn't believe how many bugs I've found by that method, that have been missed by very formal verification processes). - Don't call it done until you haven't proven to yourself that the code has no bugs. This is an informal process of self-code-review but which involves quite rigorous thinking about the behavior of the code. Often assertions are involved (which all have to be proved).
  - lisivka 10y ago
    How you prove to your peer that you done all of above? How your code is able to pass peer reviews??
    - ambrop7 10y ago
      Well, I haven't claimed to be able to to that, but to write mostly bugless code :) Though, writing insightful comments in the code helps. If anyone has doubt, you explain in a friendly way why they're wrong (if they are). Though, not all reviewers will study the code suffciently to find possible bugs, and also many findings are not the right/wrong kind but more about style/architecture.

awinter-py 10y ago

Careful -- in these studies the subjects are writing tests before writing code. In practice there are 'test-heavy' devs who use factory data and the test suite to run skeleton code with crashpoints, and switch actively between test and imp files. This has tests & implementation being written in parallel vs strict TDD which has us finishing tests before writing program logic. Most test suites depend not just on functional requirements but also on implementation details, so it seems obvious that tests-before-logic development is inefficient.

lisivka 10y ago
TDD requires to write FUNCTIONAL test case first for every new feature. Functional test case should not depend on implementation. Integrational and unit tests are.

bitL 10y ago

I thought that TDD morphed into ending up with a regression/integration/conformation test suite instead of using tests as specifications written prior to writing products. And even 100,000s of tests won't help you in very advanced applications like cloud/cluster infrastructure as sometimes it's simply too difficult if not impossible to come up with tests (imagine observer effect when your cluster deadlock happens only in certain rare nanosecond windows and adding a testing framework will make you miss those windows and the problem never happens) and people with mental capacity capable of writing them (e.g. Google/FB-level) are better utilized in writing the product itself.

jacques_chester 10y ago
How do the people who write them know that they work?
- westurner 10y ago
  > How do the people who write them know that they work? Test first isolates that a given test doesn't already pass (without any additional code). Test after (but before committing) also seems to require a more thorough critical analysis. And then someone finally fuzzes the code.
lisivka 10y ago
Why you think that debugging and fixing deadlock in cluster in production (of multibillion business) is easier and cheaper than writing of functional test case? Maybe you just prefer trips to angry customer versus boring office work. :-/ http://www.reuters.com/article/us-nasdaq-halt-tapec-idUSBRE97L0V420130822 http://www.reuters.com/article/us-nasdaq-halt-tapec-idUSBRE9...
- bitL 10y ago
  The thing is that there are problems we simply can't solve in theory nor in practice, yet we use approximate solutions all the time - and that is the case of advanced distributed algorithms. In theory, we simply can't handle real-world asynchronous systems. And when we pretend we have partially synchronous systems and build abstractions around them, they aren't 100% working. Now add in some complex bugs (like getting a distributed deadlock in transacted system involving exactly 7 nodes but not less nor more) and you might start understanding why functional test case might not really be an option to avoid these issues (you can obviously write them but they won't really help you). I worked on such a system, we had 100,000s of tests yet they were clustered around known issues and not issues that happened when e.g. a node went down and up, data were out of sync and sockets between nodes were becoming full due to OS' performance limitations. And moreover, many of these issues start showing up only when you push throughput to the max, e.g. during trading spikes etc. and adding a test that checks invariants would lower the throughput and those issues simply won't show up anymore.

emmelaich 10y ago

(to nobody in particular) Please the article before commenting. It's not totally clear from reading some of the comments that people have actually read the article. It's a good one, please do.

gravypod 10y ago

I prefer to rather then write tests plan out the interactions between all components in large projects. This will show you how all the pieces interact and what cases to need to handle in each functional unit. After this, I sit down an write all the code. After I know the organization of the source, I write out each functional unit of the code one at a time. As I go, I write each bit of test code for my source. After this I integrate every function unit. If a change is needed, I go back to the drawing board and find a better overall organization. This happens often due to either performance constraints or the need to abstract a section further. After this I'd consider embedding a unit test suite. Works great for small to medium projects.

ivanhoe 10y ago

Author is only partially right about TLD being as "doing TDD in your head", since it's (at least for me) in a much more abstract form of a general idea, a concept, of what I want to achieve. When using TDD you need to come up with the very specific results that you will test and you need then to implement those specific tests, to the last line of code. This means that if you make any changes to the logic afterwords, you need to throw away your pre-written tests and write new ones, the time spent on writing them was wasted. TLD is much more flexible and easier to update, no code is thrown away if you change something. Before I start I just need to decide what I'm trying to solve with my current block of code, and then I later write a test to check if I did it properly. Then I do the next block of logic, and the next test. Since code blocks are directly related to the steps in my logic, it's very natural to come up with the tests for them, just test if the things work as you planned it. If in the middle of that work I realize that I need to do something in a completely different way, there's no pre-written tests, so no time was wasted on coding tests that were never going to be used. And, at least to me, this kind of situations happen a lot, I often refactor and improve things as I work on them, so for me TLD is much more suitable approach.

hawkice 10y ago

Let's say I have a new theory, called Understanding Driven Development. The system says: It's a bug if someone needs to change code and they, at any moment, see code they don't understand. Stumbled into the wrong place? Bug filed for better notes on organization. The code you need to touch not understood? Understand what you see before you make a single change. If you change code and don't update docs, or documentation and code out of sync? It's a bug, and changing one to match the other _without detailed understanding_ is a bug too! Now, that seems reasonable. And if a study comes out and says people can't make program changes faster, on average, when participants are given a bit of code identical, but with more (accurate and non-trivial) comments, that doesn't mean UDD doesn't work. It doesn't test it on real, full size applications. The code was the same, despite clarity of code is one of the goals of UDD -- one of the core claims is that UDD gets you better code to begin with. It focuses on a tiny test of something not necessarily core to the UDD mindset. But it's evidence that at least one claim I've made is false. In fact, that study would be enough for me to throw that idea set into the garbage.

mannykannot 10y ago

Test-first was promoted as being the secret sauce that made TDD so much better than anything else, so this is something of a qualified vindication, but I do think (from my own experience) that writing down what I am thinking does help me see flaws that I had overlooked.

deleted 10y ago

[deleted]

mannykannot 10y ago

The author was going somewhere when he began writing about what a developer is thinking about, but, perhaps because he was focused on vindicating TDD, he did not arrive there. A developer who is writing unit tests must have a good idea of the purpose of the target of the tests, so she is thinking about requirements. Furthermore, if she is writing unit tests for small components (which will often be the case on account of everything being done in short cycles) then a lot of that purpose is contingent on other aspects of the design and how it is all supposed to work together: in other words, she is thinking about design. If you don't spend some time thinking ahead about big-picture requirements and design issues, you are in danger of going a long way down a dead end.

adamconroy 10y ago

That blog post could pretty much apply the same arguments to itself. And who knows if Bob's experience is simply correlation not causation. Perhaps Bob is just a smart, meticulous engineer, and it wouldn't matter how he went about his dev work, the quality may be good regardless.

plinkplonk 10y ago

In all discussions about TDD, it is important to distinguish between having having an automated test suite for your code which is run frequently, and writing your code test first - which is what TDD is, by definition. It is possible to advocate for the former, while thinking the latter is consultantware snake oil. (my position, fwiw)

lobut 10y ago
Yeah, I agree. TDD is also, you cannot write a single line of code without writing a failing unit test though. Well, I'm not a fan of that either. I mean, sometimes I like writing an acceptance test to begin with and work inwards. People sometimes interchangeably use "TDD" for "testing". Also, just because you do TDD, doesn't mean your code can be great, I've seen people assert pointless things and the unit has gotten so small that people now define them as methods in classes. Which I also think can lead to some crazy maintenance suite of tests. If you're interested, this was a fun discussion about TDD between two professionals: https://www.youtube.com/watch?v=KtHQGs3zFAM https://www.youtube.com/watch?v=KtHQGs3zFAM Jim Coplien and Uncle Bob.

stevenalowe 10y ago

good article; would like to add that a study based on "21 graduate students" is hardly representative of the software developer population...

ckastner 10y ago

Previous 300+ comment thread which referenced the actual paper, not a blog post about it: https://news.ycombinator.com/item?id=12740456 https://news.ycombinator.com/item?id=12740456

nunez 10y ago

I work at ThoughtWorks. TDD is central to everything that we do. That said, like anything else, TDD done to an extreme is probably a bad thing (too much time spent on tests rather than implementation) and it not being done at all is also usually bad (too much time fixing bugs that could have been caught by tests written beforehand). Balance is key.