16 ms·

Why TDD isn't crap

159 points by mzl 9y ago

vog 9y ago
This submission appears to be a reaction to the following submission: "Why Most Unit Testing Is Waste" https://news.ycombinator.com/item?id=15591190 https://news.ycombinator.com/item?id=15591190 (just for the record, i.e. for future readers)
Siecje 9y ago
What is TLA+?
- pc86 9y ago
  It is a formal specification language. [0] https://lamport.azurewebsites.net/tla/tla.html https://lamport.azurewebsites.net/tla/tla.html [1] https://en.wikipedia.org/wiki/TLA%2B https://en.wikipedia.org/wiki/TLA%2B
- nickpsecurity 9y ago
  Author has a nice tutorial on a subset of it with practical examples you might find interesting: http://learntla.com/ http://learntla.com/
jlebrech 9y ago
it's not crap if there's tight coupling between the testing frameworks the programming language and the frameworks used.
weego 9y ago
TDD isn't bad at all for a mature product where you have clear requirements for additional feature development and granular developer tasks. I've just seen a lot of it where a product is still in broad strokes development and developers get stuck between whether having tests written based on early assumptions are correct and the code should conform to those, or whether new ways of thinking that invalidates the early assumptions and tests are the right way and the tests should be changed. From the article "We don’t actually know that much about what good software engineering looks like." sums the issue up nicely. There is no definitive playbook on whether a strategy like this is good or bad. It's a tool that is good if you use it right.
- latch 9y ago
  There's some truth to this, but it's also possible that the team just wasn't very good at writing tests. It's not easy to do well. Changes will break tests, but given that changes are applied one at a time (you're not erasing your ./src folder, dumping in new code and expecting all your tests to pass), then it should be possible to keep the tests up to date. If a few changes break many tests, then tests might be over-specifying, not making use of factories, testing too much, or a lot of ways that are easy to get it wrong. I've seen and heard this a lot, and it's often the result of code that does too much (not cohesive, too much coupling) and tests that correspondingly do too much.
  - FLUX-YOU 9y ago
    Getting good at TDD requires being bad at TDD for a while, which requires someone to write bad test code at some point. But if the entire population as a whole is still having problems with TDD for as long as TDD has been around, then it needs to be a niche methodology. Our decisions to adopt methodologies/technologies can't revolve around the perfect case of a crack A-team of developers who are good at everything they do. Think of the mediocre developers!
- wvenable 9y ago
  Tests allow you to easily refactor implementation and allow you to add features without worrying as much about breaking the existing interface. However, they're terrible if you want to refactor your design and correct problems with the interface itself. Tests freeze the interface. This is really good for whole classes of problems but terrible for other classes of problems. The debate is going to continue endlessly as those on either side are looking at it from entirely different frames of reference.
  - jononor 9y ago
    That is why I almost exclusively tests against external interfaces, since those should be frozen anyway (so others can depend on it). If the system is a web service that means the HTTP API, if it is a JS library it is the public API etc. And conversely almost no tests against internal interfaces, since internals don't matter and should be free to change.
staticelf 9y ago
I think DHH sums TDD up best in this talk (which is absolutely fantastic btw): https://youtu.be/9LfmrkyP81M?t=24m https://youtu.be/9LfmrkyP81M?t=24m
latch 9y ago
Like the author, I subscribe to the less strict view that TDD isn't necessarily about writing the test first, but rather about having test play some part in how the code takes shape. Unlike the author, I absolutely believe that tests are about design. More specifically, they're about identifying coupling so that you can reduce it. The function being "awkward" to use is part of it, but code which is hard to test is almost always going to be hard to maintain. If you do this for enough time, you should naturally start to write code that is less and less tightly coupled. At that point, the value of tests as a design canary decreases, but never completely goes away. This is all broad generalizations. Individual vigilance and giving-a-fuck matter more than anything else. But if you show me some random code, and it's spaghetti, I'll bet every single time that the author doesn't test.
- quicklime 9y ago
  I've always thought that the bulk of the value of tests is for the unfortunate person who has to refactor or extend your code years down the track. By that logic, if you write testable, but untested code, then you're still making it difficult for people to refactor later. This applies even if the code is well designed and uncoupled.
  - latch 9y ago
    Helping with regressions is a big benefit (and not just for years down the road). But that's more about unit testing in general than TDD. Really, we only disagree about which is bigger. I think for the first few years of a developer's career, learning about solid (pun intended) design is hugely valuable.
  - hacker_9 9y ago
    You don't even have to think that far ahead - consider some code you wrote in week 1, that was then modified by a colleague in week 2, now you return to the code in week 3 - what does the code do? Any mental model you had at week 1 is now irrelevant as the code has changed in unknown ways. What is the current intent of the code? Do you feel safe modifying it, despite the fact you no longer know how to manually test all the end points? You only have to miss one to have clients and managers breathing down your neck when the blame game is played as to who broke the app in production.
acqq 9y ago
For those who don't use that acronym: TDD means here Test Driven Development https://en.wikipedia.org/wiki/Test-driven_development https://en.wikipedia.org/wiki/Test-driven_development
hacker_9 9y ago
I expect the reason TDD is so controversial on here is people can't see the long term benefits of tests, and instead only think in the short term. But in the commercial world, code you write can potentially have a lifespan of 30+ years. In this case, making a choice to write tests is the difference between writing a maintainable component in the future vs writing a soul-destroying 'legacy system'. If you agree tests are a good idea, but think TDD is too extreme, consider that TDD simply makes sure you write testable code from the outset. When you have a test wrapping a method, and need to add a dependency, you actually decide to use DI (Dependency Injection) because otherwise your current tests will break / become integration tests. TDD makes you think upfront about about things like mocking, separation of concerns, etc. When you have the mindset that you will absolutely 100% write tests at some point anyway, TDD is actually a faster and more fun way to develop than bolting on tests afterwards.
- miketery 9y ago
  > you actually decide to use DI Whats DI?
  - wahnfrieden 9y ago
    Dependency injection
- Clubber 9y ago
  I forgot who said it but it goes something like, "any code without unit tests is legacy code."
  - zaphar 9y ago
    Which by definition means your tests are legacy code. It's turtles all the way down. For what it's worth I do think Unit Tests provide value.
    - chewbacha 9y ago
      The code is the “test” for your tests ;)
deleted 9y ago
[deleted]
dustingetz 9y ago
Hi Hillel Wayne (Author), I love your writing style and that you are so carefully articulate about what people say, and reading their arguments charitably! The world needs a lot more of that. This article has an inspirational writing style that I will try to leverage in my own writings. PS maybe put your name at the top of your posts :)
- hwayne 9y ago
  Thank you! I was heavily inspired by Dan Luu's blog (https://danluu.com/ https://danluu.com/), so I'd recommend checking that out too.
chewbacha 9y ago
> you’ll need a bunch of servers and a hug if you have microservices I laughed out loud
deleted 9y ago
[deleted]
DanielBMarkham 9y ago
My journey with TDD started with hater, moved to skeptic, and is currently on cautionary-supporter. It's a design methodology, not some new way of unit testing. In fact, I think the more you think of TDD as being testing, the more you're probably missing the point. Modern OO languages are full of hidden dependencies and perverse side effects. The only sane way to write clear and maintainable code is to write the spec first, that is, you code by writing tests, then writing the code to make the tests pass. In this manner your code is always up-to-date with your spec. Where is it a bad idea? Exploratory or academic code, for one. Startup code where there's no clear benefit to maintainability or even knowledge of what the app is supposed to do. Pure functional code is another case entirely. Lately, I've switched to writing small pure FP in microservices, usually with less than 200 lines of code. Writing code like this creates very simple and small pieces of functionality with little hidden state or adverse side effects and a limited cyclomatic complexity factor. I don't see any reason to use TDD here, because there's nothing happening that isn't obvious. (It's a horse of a different color with larger pure FP projects, however. Having said that, one should pay careful attention to whether or not you need to build out huge pure FP execution units in the first place)
- anon1385 9y ago
  >It's a design methodology, not some new way of unit testing. In fact, I think the more you think of TDD as being testing, the more you're probably missing the point. Weirdly it seems to be the loudest advocates of TDD who are the most confused about this.
  - DanielBMarkham 9y ago
    Yes. Every time I read an essay where in one paragraph the author uses "TDD" and the next paragraph "unit testing" to mean the same thing, I want to cringe. I know we're off-the-rails. I've seen many outsourced teams say they're doing TDD and when you look at the code it's obvious it's just the same old unit testing as before. I have no idea how vendors get away with this. It's no less than fraud, really. ADD: I think the danger here is that, even with hardcore TDD boosters, they don't understand why they're doing it. It's a discipline, not an engineering skill (Choosing the tests is the engineering skill). Over time they tend to get lax. After all, the code always does mostly what I wanted it to do, right? So I can look at it and by inspection reason through the execution. At this point, when you don't understand the rationale behind it and you've started to slip-up in your application, TDD has become nothing but some weird way of writing unit tests. Then, sure, you can use the terms interchangeably. But then you've missed the entire point of what you're doing, so might as well just call it "unit test ahead of coding" or something.
chewbacha 9y ago
I’ve been viewing tests as writing the logic twice. That might sound like a waste but it’s kinda like a form asking you to type your new password twice; you may make a mistake the first time, but making the same mistake twice has much lower odds. And really, we’re playing a probability game with ourselves, trying to reduce the probability of typing the wrong thing. Writing it twice is one way to do that.
revmoo 9y ago
Getting to market faster than your competitors is more important than having test coverage, in most industries.
grandalf 9y ago
My view is very pragmatic: Some code is far easier to write quickly and correctly using a TDD approach. In other cases the cost of refactoring with TDD is very high if any significant design or architecture changes occur, so in those cases I find it is useful to stabilize the broad strokes/patterns a bit before investing to much in tests.
jrochkind1 9y ago
This is great.
apo 9y ago
The problem with many TDD critiques is that they offer no alternative. The original presentation: https://www.youtube.com/watch?v=DQBf6li1hww https://www.youtube.com/watch?v=DQBf6li1hww is a case in point. Presenter takes what he believes to be TDD's four main points, some of them strawmen, and mocks them. He does make some good points, but here's the problem: he offers no alternative. If you're not writing tests as you go, that you run before every commit (or before you move onto the next thing), then what is your standard for putting code into a production repository?
- k__ 9y ago
  You're right, I also don't know good alternatives. I wrote my first big API with many tests, hundreds of them. Then the requirements changed and all of them failed. People worked for weeks to get the tests passing again. So just writing many tests up front in a new project doesn't seem to help anyone. Also, I went from feature to bugfix sprints. First I implemented some features, then they got tested by non-devs, then I fixed all the bugs. Often this was faster than the whole tests up front stuff with refinement of tests afterwards. I also saw that >90% of my bugs came from the dynamic nature of my language of choice (JS). So I could imagine, that this feature->bugfix->feature->bugfix cycles could be greatly shortened (especially the bugfix parts) by using a typed language instead.
  - humanrebar 9y ago
    In most statically typed languages, the bugfixing (usually) takes longer because you need to do many more things each time you refactor. An enterprise Java application doesn't typically pass around the equivalent of a JS object. It passes around Users, Admins, and Guests. But then some features in Admins need to be added to Users without affecting Guests. But Guests inherit from Users, so you have to go back and restructure the type hierarchy. But it turns out that User is a Hibernate entity, so you have to make sure you don't change your DB schema on accident. Using a static language does help with some bugs (where it effectively serves as a compile-time test suite), but it can also increase coupling a lot. It's not easy to quantify that tradeoff in abstract. I'd be wary of people overselling aggressive compiler checks leading to productivity boosts.
macawfish 9y ago
You know what's awesome about TDD? It means I, a nobody, can contribute to a huge open source project with lots of moving pieces and be pretty confident that I'm not catastrophically breaking anything and that my feature works as intended. That's awesome!
- humanrebar 9y ago
  Right. Good tests are a passive communication tool. They communicate expectations. Most of the discussion here revolves around different people having different expectations. I think the "please test" people have a better argument mostly because they're advocating for tools for good communication.
  - macawfish 9y ago
    Totally. I used to hate testing. It felt like someone making me do homework I didn't want to do. Then it dawned on me that I was already testing, the hard way, by opening up a console and manually setting up test conditions over and over and over again, and that I could do this much faster and in a reusable way by writing tests and running them. What an epiphany that was! I still open up the console all the time. It's a really useful thing. I think ideally, my testing environment would dump me into a full-fledged console if something went wrong, but this is not something I've taken the time to set up.
- the_af 9y ago
  That seems to do more with automated testing (unit, regression, integration, etc) than with TDD. It's important not to conflate the two because TDD proponents say that TDD is not primarily about testing.
  - macawfish 9y ago
    Thanks for clarifying this. I wasn't aware of the hardcore TDD thing, I moreso just interpreted TDD to be the concept of "write tests as you code".
signa11 9y ago
tdd always reminds me of ron jeffries attempt at solving sudoku, in contrast with peter-norvig's approach...
- hwayne 9y ago
  Ron Jeffries' mistake was thinking that he could just use TDD to solve sudoku, as opposed to research and up-front design. Testing does not substitute for thinking.
jondubois 9y ago
I'm glad that developers are finally having honest discussions about this. Not long ago, it seemed like 100% unit test coverage with TDD was the only valid point of view to have. I'm not a fan of TDD (in the sense of writing the test first) but I do think that some sort of testing-as-you-go is important for back end API work in particular. I don't think that 100% test coverage is a good idea for 99% of business use cases.
LoSboccacc 9y ago
the most important part to me is to write testable code. and you cannot be sure that something is testable unless you do write at least some test to check that objects do in fact work in a mocked environment and that you can replicate user interaction at any level of your application. writing a full test battery after that is probably overkill, but making sure everything can be taken out of the running app and tested singularly is essential to be able at a later date to get a user bug report and convert it into a testable case to narrow down the root cause. a good strategy for that goal is to make use of dependency injection at each layer separation and to make sure that every user generated event can be also triggered pro grammatically - that's especially useful as while relying on something like selenium do work, it's exceptionally costly and aggravating in the long run. being able to isolate the bugged behavior and responsible component is the major advantage that comes with the full TDD implementation, but that doesn't mean you can't have enough of that with a lighter approach
Bluecobra 9y ago
Yes, Transport Tycoon Deluxe is a great game and OpenTTD is even better.
logicallee 9y ago
I haven't shared this on HN but I saw this on another forum and thought this was a really fantastic example of what you can end up with, in the real world, when you are driven by tests. Real-world example: https://i.redd.it/lwin56fisdsz.png https://i.redd.it/lwin56fisdsz.png It doesn't look like a joke to me. It only works over integers so the code is absolutely correct. It also strikes me as the kind of convoluted logic that someone took a really, really long time to come up with, before it finally worked. (As indicated in the comment.) A test can hardly capture what's wrong with this code. But any human can see it instantly. (And it's kind of weird that the programmer didn't.) I think most people can think of braindead decisions that are not really captured by testing.
- UncleMeat 9y ago
  Are there any proponents of serious testing or tdd that don't also promote code review? Why would tests make this code more likely? If anything, I am more confident that I can change it to be better if I have a test suite.
  - ngoede 9y ago
    Yeah the way to write better code is to write better code. I happen to find TDD a useful tool in that but doing things badly is still possible. Someone mentioned DI not getting rid of coupling and I agree. DI is a tool you might use but the way to get rid of coupling is not one simple thing but a process of a bunch of different tools and techniques. You can't just slavishly fallow some process and expect it to fix all your issues. You have to think and do work yourself to fix it.
  - logicallee 9y ago
    >Why would tests make this code more likely? Code like this is basically only possible through testing, because the programmer doesn't understand why it is working: it says so in the comment, which I believe. So if it weren't written against testing (manual or automatic) it simply couldn't be written like this. There are a lot of really broken designs that "work". Testing gets you "working" code. Often you could do much better starting with correct code and then adding testing afterward - which I believe is not the essence of TDD. TDD is about driving the writing of code by tests, and I believe the example I shared is one of these bad outcomes. (There are others.)
agentultra 9y ago
The only thing I disagree with in this article is the leeway given to supposedly legendary programmers who can somehow write bug-free code without tests, specifications, or an inkling of communication with others. First, it's probably not true. Linus Torvalds is not a legendary human being who can write critical systems without a single flaw. He relies on legions of human beings to carefully check and review every line of code before he even looks at it. There are discussions on mailing lists. There are arguments and disagreements. There's a process there. He doesn't just flit his fingers across the keyboard and output amazing, error free code. It probably has tonnes of errors. Linus' philosophy is that errors aren't the end of the world and someone will patch them when they are uncovered. For some use cases that's fine. However there are plenty of applications where a more proactive approach to correctness is necessary: real-time systems, safety critical systems, and yes... even security. Maybe TDD is a misnomer. I think we should call it specification driven development. Unit tests and integration tests are just a weak form of specification. They provide theorems in the form of examples that we try to prove with an implementation. Property based tests give us more examples to quantify assertions over. Model checking can test liveness as well as safety in our high-level designs... how much you need to specify and how thoroughly really should be a factor of the risk and complexity present in the requirements of the system. To use an analogy: blueprints. If you're just building a shed or a small footpath then it's enough to sketch your idea on a napkin. If you're building a house you need to have a more specific and detailed plan that passes by a civil engineer. And if you're building a sky scraper then you need to be thorough and able to convince others of the validity of your designs. (credit for the analogy should go to Leslie Lamport). I think most software projects are at the house level in terms of risk. You could get by with using a dynamic language and a few unit tests if you value productivity more than correctness. That just means you're willing to accept that you will have higher reported error rates and are comfortable with potentially losing customer data or a higher risk for security vulnerabilities. You can lower your risk if the project requires more sensitivity to data consistency or security by using a sound type system and encoding your assertions at the type level, add some property-based tests, and more integration and unit tests. It's a spectrum one should consider. I know we all like to write code and sometimes we even hear ourselves saying, "Well if you wrote the perfect specification you might as well have written the software," but don't be fooled. "Software engineering is the part of computer science which is too difficult for the computer scientist." -- Friedrich Bauer
codemac 9y ago
This is a very poorly researched article, and the previous was as well. With people like Capers Jones and others doing piles of studies 30+ years ago, I'm confused why someone says there are no studies on TDD. My bet is the author doesn't have access to the relevant historical papers, and doesn't know they exist.
- davedx 9y ago
  Could you provide some citations please?
  - codemac 9y ago
    Look up the talk "what we know about software engineering" on YouTube. Also, search for Mills, Cleanroom, and obviously Capers Jones as I mentioned. I'm on my phone so I can't link a lot, but I maintain that the author should do a lit review of software quality in engineering, and they'll get better conclusions
    - hwayne 9y ago
      TDD is only ~15 years old. Cleanroom and correct-by-construction both have some solid studies supporting them, but take very different approaches than TDD does. They tend to be more rigorous and also take longer so again, it's a case-by-case basis thing.
      - codemac 9y ago
        > They tend to be more rigorous and also take longer so again, it's a case-by-case basis thing I'm not sure if we're taking away the same things from these studies, as their whole conclusion is it actually takes less time and costs less, due to early wins in quality. The papers claim it's not a case-by-case thing.
JoeNr76 9y ago
I never was a fan of TDD, until I saw this talk by Ian Cooper: https://www.infoq.com/presentations/tdd-original https://www.infoq.com/presentations/tdd-original The whole idea of testing functions and/or classes separately means tightly coupling your test code to the implementation of the real code, while you should only care about testing the functionality. Nowadays, I try to write tests that test a unit of functionality. And the tests should only change when the functionality changes, not after every refactoring. That said, code coverage is a metric with no inherit value (well, unless it's 0, of course)
- marsrover 9y ago
  Not a word for word quote, but I heard this: > How many of you have a code base that if you refactored, test would break? > Most people raise hand and it made me realize maybe I'm not as stupid as I think I am. I've been trying to understand how to create unit tests that allow me to refactor for years, and have been completely unsuccessful. The only way I can achieve this is using the "classicist" viewpoint of creating many "unit test" and only isolating the architecturally significant boundaries. That doesn't make me happy either, though. But, yeah, good to know (or bad to know) that I'm not the only one that struggles with this. Back to the video I go.
S_A_P 9y ago
I don't think that any of the common development methodologies are crap. I see no problem with writing tests as a way to work out ideas. I see no problem with not doing so. I think the main flaw in these methodologies is that they assume people problem solve in the same way. It only becomes a problem when you have a methodology zealot telling you to work in that way only. I am happy to write unit tests to cover my code. Unit tests can and do prevent bugs/outages in many cases. What I do think is kind of crap is the holy war of methodologies that exists. In my experience, it usually happens that some new CTO or other manager comes in and says, "We are doing it wrong! From this day forward, WE ALL MUST USE TDD/AGILE/WATERFALL/YOURFAVORITEWAYHERE". There is no latitude given, and folks who are not used to the methodology now take 1.5 to 3x longer to complete things, and deadlines slip. Then the push to complete work means that things get written sloppily and tests are not well thought out or people arent fooing their bar with the baz properly. Quality suffers in the short term, but eventually everyone catches on, and life goes back to normal. Then the new CTO shows up...
unnawut 9y ago
I have been attempting at TDD on and off for about 3 years. It never really took off for me. I would spend most of the time writing the tests, then implementing the code, then only to find out that a lot of my tests don't make sense, or are overthoughts that don't really add business nor technical value. However, I think I reached my sweet spot just weeks ago. Here is my optimum workflow now: 1) Write test cases (the one sentences that say what is expected, in plain English) 2) Implement the code 3) Implement the test code The test cases become neatly arranged in bullet-like layout in a test case file. I'm able to read through and be confident that I'm probably covering most if not all the cases that need to be covered. I'm always able to switch between the code and this file to make sure my code covered all that the tests need. Then once the code is done, I come back to the test cases to implement them one by one, catching error by error and seeing my code coming to life. As I code now, I know I have the implementation I'm quite confident of, my tests that have business value are being covered one by one. It's been pleasure since then and I'm more confident of my code and of my time spent efficiently.
- Vinnl 9y ago
  This works very well for me too when I'm reluctant to go full TDD. That said, step 2 isn't necessarily the complete code needed for the tests, so there is some back-and-forth between step 2 and 3, resulting in some test code still being written before the code under test.
- xellisx 9y ago
  I've been trying to get into TDD myself for years. I've read the 2 most suggested books and another book that had "testing" in it. But I still came back with issues. I don't program Java, so I don't understand some of this code, so I can't just convert it over to the languages I know. The examples were of basic stuff like "Lets make something that adds two numbers", "Here's a basic mock, lets not worry about DB queries", etc... It really didn't teach me how to think in TDD, it wasn't specific for the languages I program in, nor did it touch on things like "real time data" coming from a stream source, which I think is "How to think in TDD and how to program to fit it."
p0nce 9y ago
I guess TDD can be great for parsers.
- jononor 9y ago
  In the classical way of setting up the parser, passing some example input and then asserting some things about the output? It's good for covering basic features/usecases. Since setup is always the same, it should be written as a list of input/expectation pairs (data driven). But coverage is typically limited by the effort needed to write the assertions - which grows in parallel with the input complexity. If one also writes a serializer, then one can additionally test for any input the property: `serialize(parse(input)) === input`. This means adding a new test is just dropping in more example inputs (say from bug reports). Now one can go further, and define a set of mutation operators that can act on input to produce a new valid input. Reorder tokens, change data at leaves, delete and insert new data, etc. Now one can generate arbitrary amounts of new test cases based on existing input examples. Other mutation operations can be designed to generate invalid inputs, which should always give an error (never crash, halt or unexpected exception).
jorgec 9y ago
I created an document explaining TDD (test unit in general) is a real case scenario: https://medium.com/southprojects/tdd-a-business-crud-is-it-worth-1e6d03dd6b84 https://medium.com/southprojects/tdd-a-business-crud-is-it-w... tl/dr: In theory TDD sounds great but in a real example, TDD is not magic with a real limited coverage.

159 points by mzl 9y ago

vog 9y ago

This submission appears to be a reaction to the following submission: "Why Most Unit Testing Is Waste" https://news.ycombinator.com/item?id=15591190 https://news.ycombinator.com/item?id=15591190 (just for the record, i.e. for future readers)

Siecje 9y ago

What is TLA+?

pc86 9y ago
It is a formal specification language. [0] https://lamport.azurewebsites.net/tla/tla.html https://lamport.azurewebsites.net/tla/tla.html [1] https://en.wikipedia.org/wiki/TLA%2B https://en.wikipedia.org/wiki/TLA%2B
nickpsecurity 9y ago
Author has a nice tutorial on a subset of it with practical examples you might find interesting: http://learntla.com/ http://learntla.com/

jlebrech 9y ago

it's not crap if there's tight coupling between the testing frameworks the programming language and the frameworks used.

weego 9y ago

TDD isn't bad at all for a mature product where you have clear requirements for additional feature development and granular developer tasks. I've just seen a lot of it where a product is still in broad strokes development and developers get stuck between whether having tests written based on early assumptions are correct and the code should conform to those, or whether new ways of thinking that invalidates the early assumptions and tests are the right way and the tests should be changed. From the article "We don’t actually know that much about what good software engineering looks like." sums the issue up nicely. There is no definitive playbook on whether a strategy like this is good or bad. It's a tool that is good if you use it right.

latch 9y ago
There's some truth to this, but it's also possible that the team just wasn't very good at writing tests. It's not easy to do well. Changes will break tests, but given that changes are applied one at a time (you're not erasing your ./src folder, dumping in new code and expecting all your tests to pass), then it should be possible to keep the tests up to date. If a few changes break many tests, then tests might be over-specifying, not making use of factories, testing too much, or a lot of ways that are easy to get it wrong. I've seen and heard this a lot, and it's often the result of code that does too much (not cohesive, too much coupling) and tests that correspondingly do too much.
- FLUX-YOU 9y ago
  Getting good at TDD requires being bad at TDD for a while, which requires someone to write bad test code at some point. But if the entire population as a whole is still having problems with TDD for as long as TDD has been around, then it needs to be a niche methodology. Our decisions to adopt methodologies/technologies can't revolve around the perfect case of a crack A-team of developers who are good at everything they do. Think of the mediocre developers!
wvenable 9y ago
Tests allow you to easily refactor implementation and allow you to add features without worrying as much about breaking the existing interface. However, they're terrible if you want to refactor your design and correct problems with the interface itself. Tests freeze the interface. This is really good for whole classes of problems but terrible for other classes of problems. The debate is going to continue endlessly as those on either side are looking at it from entirely different frames of reference.
- jononor 9y ago
  That is why I almost exclusively tests against external interfaces, since those should be frozen anyway (so others can depend on it). If the system is a web service that means the HTTP API, if it is a JS library it is the public API etc. And conversely almost no tests against internal interfaces, since internals don't matter and should be free to change.

staticelf 9y ago

I think DHH sums TDD up best in this talk (which is absolutely fantastic btw): https://youtu.be/9LfmrkyP81M?t=24m https://youtu.be/9LfmrkyP81M?t=24m

latch 9y ago

Like the author, I subscribe to the less strict view that TDD isn't necessarily about writing the test first, but rather about having test play some part in how the code takes shape. Unlike the author, I absolutely believe that tests are about design. More specifically, they're about identifying coupling so that you can reduce it. The function being "awkward" to use is part of it, but code which is hard to test is almost always going to be hard to maintain. If you do this for enough time, you should naturally start to write code that is less and less tightly coupled. At that point, the value of tests as a design canary decreases, but never completely goes away. This is all broad generalizations. Individual vigilance and giving-a-fuck matter more than anything else. But if you show me some random code, and it's spaghetti, I'll bet every single time that the author doesn't test.

quicklime 9y ago
I've always thought that the bulk of the value of tests is for the unfortunate person who has to refactor or extend your code years down the track. By that logic, if you write testable, but untested code, then you're still making it difficult for people to refactor later. This applies even if the code is well designed and uncoupled.
- latch 9y ago
  Helping with regressions is a big benefit (and not just for years down the road). But that's more about unit testing in general than TDD. Really, we only disagree about which is bigger. I think for the first few years of a developer's career, learning about solid (pun intended) design is hugely valuable.
- hacker_9 9y ago
  You don't even have to think that far ahead - consider some code you wrote in week 1, that was then modified by a colleague in week 2, now you return to the code in week 3 - what does the code do? Any mental model you had at week 1 is now irrelevant as the code has changed in unknown ways. What is the current intent of the code? Do you feel safe modifying it, despite the fact you no longer know how to manually test all the end points? You only have to miss one to have clients and managers breathing down your neck when the blame game is played as to who broke the app in production.

acqq 9y ago

For those who don't use that acronym: TDD means here Test Driven Development https://en.wikipedia.org/wiki/Test-driven_development https://en.wikipedia.org/wiki/Test-driven_development

hacker_9 9y ago

I expect the reason TDD is so controversial on here is people can't see the long term benefits of tests, and instead only think in the short term. But in the commercial world, code you write can potentially have a lifespan of 30+ years. In this case, making a choice to write tests is the difference between writing a maintainable component in the future vs writing a soul-destroying 'legacy system'. If you agree tests are a good idea, but think TDD is too extreme, consider that TDD simply makes sure you write testable code from the outset. When you have a test wrapping a method, and need to add a dependency, you actually decide to use DI (Dependency Injection) because otherwise your current tests will break / become integration tests. TDD makes you think upfront about about things like mocking, separation of concerns, etc. When you have the mindset that you will absolutely 100% write tests at some point anyway, TDD is actually a faster and more fun way to develop than bolting on tests afterwards.

miketery 9y ago
> you actually decide to use DI Whats DI?
- wahnfrieden 9y ago
  Dependency injection
Clubber 9y ago
I forgot who said it but it goes something like, "any code without unit tests is legacy code."
- zaphar 9y ago
  Which by definition means your tests are legacy code. It's turtles all the way down. For what it's worth I do think Unit Tests provide value.
  - chewbacha 9y ago
    The code is the “test” for your tests ;)

deleted 9y ago

[deleted]

dustingetz 9y ago

Hi Hillel Wayne (Author), I love your writing style and that you are so carefully articulate about what people say, and reading their arguments charitably! The world needs a lot more of that. This article has an inspirational writing style that I will try to leverage in my own writings. PS maybe put your name at the top of your posts :)

hwayne 9y ago
Thank you! I was heavily inspired by Dan Luu's blog (https://danluu.com/ https://danluu.com/), so I'd recommend checking that out too.

chewbacha 9y ago

> you’ll need a bunch of servers and a hug if you have microservices I laughed out loud

deleted 9y ago

[deleted]

DanielBMarkham 9y ago

My journey with TDD started with hater, moved to skeptic, and is currently on cautionary-supporter. It's a design methodology, not some new way of unit testing. In fact, I think the more you think of TDD as being testing, the more you're probably missing the point. Modern OO languages are full of hidden dependencies and perverse side effects. The only sane way to write clear and maintainable code is to write the spec first, that is, you code by writing tests, then writing the code to make the tests pass. In this manner your code is always up-to-date with your spec. Where is it a bad idea? Exploratory or academic code, for one. Startup code where there's no clear benefit to maintainability or even knowledge of what the app is supposed to do. Pure functional code is another case entirely. Lately, I've switched to writing small pure FP in microservices, usually with less than 200 lines of code. Writing code like this creates very simple and small pieces of functionality with little hidden state or adverse side effects and a limited cyclomatic complexity factor. I don't see any reason to use TDD here, because there's nothing happening that isn't obvious. (It's a horse of a different color with larger pure FP projects, however. Having said that, one should pay careful attention to whether or not you need to build out huge pure FP execution units in the first place)

anon1385 9y ago
>It's a design methodology, not some new way of unit testing. In fact, I think the more you think of TDD as being testing, the more you're probably missing the point. Weirdly it seems to be the loudest advocates of TDD who are the most confused about this.
- DanielBMarkham 9y ago
  Yes. Every time I read an essay where in one paragraph the author uses "TDD" and the next paragraph "unit testing" to mean the same thing, I want to cringe. I know we're off-the-rails. I've seen many outsourced teams say they're doing TDD and when you look at the code it's obvious it's just the same old unit testing as before. I have no idea how vendors get away with this. It's no less than fraud, really. ADD: I think the danger here is that, even with hardcore TDD boosters, they don't understand why they're doing it. It's a discipline, not an engineering skill (Choosing the tests is the engineering skill). Over time they tend to get lax. After all, the code always does mostly what I wanted it to do, right? So I can look at it and by inspection reason through the execution. At this point, when you don't understand the rationale behind it and you've started to slip-up in your application, TDD has become nothing but some weird way of writing unit tests. Then, sure, you can use the terms interchangeably. But then you've missed the entire point of what you're doing, so might as well just call it "unit test ahead of coding" or something.

chewbacha 9y ago

I’ve been viewing tests as writing the logic twice. That might sound like a waste but it’s kinda like a form asking you to type your new password twice; you may make a mistake the first time, but making the same mistake twice has much lower odds. And really, we’re playing a probability game with ourselves, trying to reduce the probability of typing the wrong thing. Writing it twice is one way to do that.

revmoo 9y ago

Getting to market faster than your competitors is more important than having test coverage, in most industries.

grandalf 9y ago

My view is very pragmatic: Some code is far easier to write quickly and correctly using a TDD approach. In other cases the cost of refactoring with TDD is very high if any significant design or architecture changes occur, so in those cases I find it is useful to stabilize the broad strokes/patterns a bit before investing to much in tests.

jrochkind1 9y ago

This is great.

apo 9y ago

The problem with many TDD critiques is that they offer no alternative. The original presentation: https://www.youtube.com/watch?v=DQBf6li1hww https://www.youtube.com/watch?v=DQBf6li1hww is a case in point. Presenter takes what he believes to be TDD's four main points, some of them strawmen, and mocks them. He does make some good points, but here's the problem: he offers no alternative. If you're not writing tests as you go, that you run before every commit (or before you move onto the next thing), then what is your standard for putting code into a production repository?

k__ 9y ago
You're right, I also don't know good alternatives. I wrote my first big API with many tests, hundreds of them. Then the requirements changed and all of them failed. People worked for weeks to get the tests passing again. So just writing many tests up front in a new project doesn't seem to help anyone. Also, I went from feature to bugfix sprints. First I implemented some features, then they got tested by non-devs, then I fixed all the bugs. Often this was faster than the whole tests up front stuff with refinement of tests afterwards. I also saw that >90% of my bugs came from the dynamic nature of my language of choice (JS). So I could imagine, that this feature->bugfix->feature->bugfix cycles could be greatly shortened (especially the bugfix parts) by using a typed language instead.
- humanrebar 9y ago
  In most statically typed languages, the bugfixing (usually) takes longer because you need to do many more things each time you refactor. An enterprise Java application doesn't typically pass around the equivalent of a JS object. It passes around Users, Admins, and Guests. But then some features in Admins need to be added to Users without affecting Guests. But Guests inherit from Users, so you have to go back and restructure the type hierarchy. But it turns out that User is a Hibernate entity, so you have to make sure you don't change your DB schema on accident. Using a static language does help with some bugs (where it effectively serves as a compile-time test suite), but it can also increase coupling a lot. It's not easy to quantify that tradeoff in abstract. I'd be wary of people overselling aggressive compiler checks leading to productivity boosts.

macawfish 9y ago

You know what's awesome about TDD? It means I, a nobody, can contribute to a huge open source project with lots of moving pieces and be pretty confident that I'm not catastrophically breaking anything and that my feature works as intended. That's awesome!

humanrebar 9y ago
Right. Good tests are a passive communication tool. They communicate expectations. Most of the discussion here revolves around different people having different expectations. I think the "please test" people have a better argument mostly because they're advocating for tools for good communication.
- macawfish 9y ago
  Totally. I used to hate testing. It felt like someone making me do homework I didn't want to do. Then it dawned on me that I was already testing, the hard way, by opening up a console and manually setting up test conditions over and over and over again, and that I could do this much faster and in a reusable way by writing tests and running them. What an epiphany that was! I still open up the console all the time. It's a really useful thing. I think ideally, my testing environment would dump me into a full-fledged console if something went wrong, but this is not something I've taken the time to set up.
the_af 9y ago
That seems to do more with automated testing (unit, regression, integration, etc) than with TDD. It's important not to conflate the two because TDD proponents say that TDD is not primarily about testing.
- macawfish 9y ago
  Thanks for clarifying this. I wasn't aware of the hardcore TDD thing, I moreso just interpreted TDD to be the concept of "write tests as you code".

signa11 9y ago

tdd always reminds me of ron jeffries attempt at solving sudoku, in contrast with peter-norvig's approach...

hwayne 9y ago
Ron Jeffries' mistake was thinking that he could just use TDD to solve sudoku, as opposed to research and up-front design. Testing does not substitute for thinking.

jondubois 9y ago

I'm glad that developers are finally having honest discussions about this. Not long ago, it seemed like 100% unit test coverage with TDD was the only valid point of view to have. I'm not a fan of TDD (in the sense of writing the test first) but I do think that some sort of testing-as-you-go is important for back end API work in particular. I don't think that 100% test coverage is a good idea for 99% of business use cases.

LoSboccacc 9y ago

the most important part to me is to write testable code. and you cannot be sure that something is testable unless you do write at least some test to check that objects do in fact work in a mocked environment and that you can replicate user interaction at any level of your application. writing a full test battery after that is probably overkill, but making sure everything can be taken out of the running app and tested singularly is essential to be able at a later date to get a user bug report and convert it into a testable case to narrow down the root cause. a good strategy for that goal is to make use of dependency injection at each layer separation and to make sure that every user generated event can be also triggered pro grammatically - that's especially useful as while relying on something like selenium do work, it's exceptionally costly and aggravating in the long run. being able to isolate the bugged behavior and responsible component is the major advantage that comes with the full TDD implementation, but that doesn't mean you can't have enough of that with a lighter approach

Bluecobra 9y ago

Yes, Transport Tycoon Deluxe is a great game and OpenTTD is even better.

logicallee 9y ago

I haven't shared this on HN but I saw this on another forum and thought this was a really fantastic example of what you can end up with, in the real world, when you are driven by tests. Real-world example: https://i.redd.it/lwin56fisdsz.png https://i.redd.it/lwin56fisdsz.png It doesn't look like a joke to me. It only works over integers so the code is absolutely correct. It also strikes me as the kind of convoluted logic that someone took a really, really long time to come up with, before it finally worked. (As indicated in the comment.) A test can hardly capture what's wrong with this code. But any human can see it instantly. (And it's kind of weird that the programmer didn't.) I think most people can think of braindead decisions that are not really captured by testing.

UncleMeat 9y ago
Are there any proponents of serious testing or tdd that don't also promote code review? Why would tests make this code more likely? If anything, I am more confident that I can change it to be better if I have a test suite.
- ngoede 9y ago
  Yeah the way to write better code is to write better code. I happen to find TDD a useful tool in that but doing things badly is still possible. Someone mentioned DI not getting rid of coupling and I agree. DI is a tool you might use but the way to get rid of coupling is not one simple thing but a process of a bunch of different tools and techniques. You can't just slavishly fallow some process and expect it to fix all your issues. You have to think and do work yourself to fix it.
- logicallee 9y ago
  >Why would tests make this code more likely? Code like this is basically only possible through testing, because the programmer doesn't understand why it is working: it says so in the comment, which I believe. So if it weren't written against testing (manual or automatic) it simply couldn't be written like this. There are a lot of really broken designs that "work". Testing gets you "working" code. Often you could do much better starting with correct code and then adding testing afterward - which I believe is not the essence of TDD. TDD is about driving the writing of code by tests, and I believe the example I shared is one of these bad outcomes. (There are others.)

agentultra 9y ago

The only thing I disagree with in this article is the leeway given to supposedly legendary programmers who can somehow write bug-free code without tests, specifications, or an inkling of communication with others. First, it's probably not true. Linus Torvalds is not a legendary human being who can write critical systems without a single flaw. He relies on legions of human beings to carefully check and review every line of code before he even looks at it. There are discussions on mailing lists. There are arguments and disagreements. There's a process there. He doesn't just flit his fingers across the keyboard and output amazing, error free code. It probably has tonnes of errors. Linus' philosophy is that errors aren't the end of the world and someone will patch them when they are uncovered. For some use cases that's fine. However there are plenty of applications where a more proactive approach to correctness is necessary: real-time systems, safety critical systems, and yes... even security. Maybe TDD is a misnomer. I think we should call it specification driven development. Unit tests and integration tests are just a weak form of specification. They provide theorems in the form of examples that we try to prove with an implementation. Property based tests give us more examples to quantify assertions over. Model checking can test liveness as well as safety in our high-level designs... how much you need to specify and how thoroughly really should be a factor of the risk and complexity present in the requirements of the system. To use an analogy: blueprints. If you're just building a shed or a small footpath then it's enough to sketch your idea on a napkin. If you're building a house you need to have a more specific and detailed plan that passes by a civil engineer. And if you're building a sky scraper then you need to be thorough and able to convince others of the validity of your designs. (credit for the analogy should go to Leslie Lamport). I think most software projects are at the house level in terms of risk. You could get by with using a dynamic language and a few unit tests if you value productivity more than correctness. That just means you're willing to accept that you will have higher reported error rates and are comfortable with potentially losing customer data or a higher risk for security vulnerabilities. You can lower your risk if the project requires more sensitivity to data consistency or security by using a sound type system and encoding your assertions at the type level, add some property-based tests, and more integration and unit tests. It's a spectrum one should consider. I know we all like to write code and sometimes we even hear ourselves saying, "Well if you wrote the perfect specification you might as well have written the software," but don't be fooled. "Software engineering is the part of computer science which is too difficult for the computer scientist." -- Friedrich Bauer

codemac 9y ago

This is a very poorly researched article, and the previous was as well. With people like Capers Jones and others doing piles of studies 30+ years ago, I'm confused why someone says there are no studies on TDD. My bet is the author doesn't have access to the relevant historical papers, and doesn't know they exist.

davedx 9y ago
Could you provide some citations please?
- codemac 9y ago
  Look up the talk "what we know about software engineering" on YouTube. Also, search for Mills, Cleanroom, and obviously Capers Jones as I mentioned. I'm on my phone so I can't link a lot, but I maintain that the author should do a lit review of software quality in engineering, and they'll get better conclusions
  - hwayne 9y ago
    TDD is only ~15 years old. Cleanroom and correct-by-construction both have some solid studies supporting them, but take very different approaches than TDD does. They tend to be more rigorous and also take longer so again, it's a case-by-case basis thing.
    - codemac 9y ago
      > They tend to be more rigorous and also take longer so again, it's a case-by-case basis thing I'm not sure if we're taking away the same things from these studies, as their whole conclusion is it actually takes less time and costs less, due to early wins in quality. The papers claim it's not a case-by-case thing.

JoeNr76 9y ago

I never was a fan of TDD, until I saw this talk by Ian Cooper: https://www.infoq.com/presentations/tdd-original https://www.infoq.com/presentations/tdd-original The whole idea of testing functions and/or classes separately means tightly coupling your test code to the implementation of the real code, while you should only care about testing the functionality. Nowadays, I try to write tests that test a unit of functionality. And the tests should only change when the functionality changes, not after every refactoring. That said, code coverage is a metric with no inherit value (well, unless it's 0, of course)

marsrover 9y ago
Not a word for word quote, but I heard this: > How many of you have a code base that if you refactored, test would break? > Most people raise hand and it made me realize maybe I'm not as stupid as I think I am. I've been trying to understand how to create unit tests that allow me to refactor for years, and have been completely unsuccessful. The only way I can achieve this is using the "classicist" viewpoint of creating many "unit test" and only isolating the architecturally significant boundaries. That doesn't make me happy either, though. But, yeah, good to know (or bad to know) that I'm not the only one that struggles with this. Back to the video I go.

S_A_P 9y ago

I don't think that any of the common development methodologies are crap. I see no problem with writing tests as a way to work out ideas. I see no problem with not doing so. I think the main flaw in these methodologies is that they assume people problem solve in the same way. It only becomes a problem when you have a methodology zealot telling you to work in that way only. I am happy to write unit tests to cover my code. Unit tests can and do prevent bugs/outages in many cases. What I do think is kind of crap is the holy war of methodologies that exists. In my experience, it usually happens that some new CTO or other manager comes in and says, "We are doing it wrong! From this day forward, WE ALL MUST USE TDD/AGILE/WATERFALL/YOURFAVORITEWAYHERE". There is no latitude given, and folks who are not used to the methodology now take 1.5 to 3x longer to complete things, and deadlines slip. Then the push to complete work means that things get written sloppily and tests are not well thought out or people arent fooing their bar with the baz properly. Quality suffers in the short term, but eventually everyone catches on, and life goes back to normal. Then the new CTO shows up...

unnawut 9y ago

I have been attempting at TDD on and off for about 3 years. It never really took off for me. I would spend most of the time writing the tests, then implementing the code, then only to find out that a lot of my tests don't make sense, or are overthoughts that don't really add business nor technical value. However, I think I reached my sweet spot just weeks ago. Here is my optimum workflow now: 1) Write test cases (the one sentences that say what is expected, in plain English) 2) Implement the code 3) Implement the test code The test cases become neatly arranged in bullet-like layout in a test case file. I'm able to read through and be confident that I'm probably covering most if not all the cases that need to be covered. I'm always able to switch between the code and this file to make sure my code covered all that the tests need. Then once the code is done, I come back to the test cases to implement them one by one, catching error by error and seeing my code coming to life. As I code now, I know I have the implementation I'm quite confident of, my tests that have business value are being covered one by one. It's been pleasure since then and I'm more confident of my code and of my time spent efficiently.

Vinnl 9y ago
This works very well for me too when I'm reluctant to go full TDD. That said, step 2 isn't necessarily the complete code needed for the tests, so there is some back-and-forth between step 2 and 3, resulting in some test code still being written before the code under test.
xellisx 9y ago
I've been trying to get into TDD myself for years. I've read the 2 most suggested books and another book that had "testing" in it. But I still came back with issues. I don't program Java, so I don't understand some of this code, so I can't just convert it over to the languages I know. The examples were of basic stuff like "Lets make something that adds two numbers", "Here's a basic mock, lets not worry about DB queries", etc... It really didn't teach me how to think in TDD, it wasn't specific for the languages I program in, nor did it touch on things like "real time data" coming from a stream source, which I think is "How to think in TDD and how to program to fit it."

p0nce 9y ago

I guess TDD can be great for parsers.

jononor 9y ago
In the classical way of setting up the parser, passing some example input and then asserting some things about the output? It's good for covering basic features/usecases. Since setup is always the same, it should be written as a list of input/expectation pairs (data driven). But coverage is typically limited by the effort needed to write the assertions - which grows in parallel with the input complexity. If one also writes a serializer, then one can additionally test for any input the property: `serialize(parse(input)) === input`. This means adding a new test is just dropping in more example inputs (say from bug reports). Now one can go further, and define a set of mutation operators that can act on input to produce a new valid input. Reorder tokens, change data at leaves, delete and insert new data, etc. Now one can generate arbitrary amounts of new test cases based on existing input examples. Other mutation operations can be designed to generate invalid inputs, which should always give an error (never crash, halt or unexpected exception).

jorgec 9y ago

I created an document explaining TDD (test unit in general) is a real case scenario: https://medium.com/southprojects/tdd-a-business-crud-is-it-worth-1e6d03dd6b84 https://medium.com/southprojects/tdd-a-business-crud-is-it-w... tl/dr: In theory TDD sounds great but in a real example, TDD is not magic with a real limited coverage.