9 ms·
For many paragraphs, I thought this was a parody of TDD defense, but then it turns out it wasn't.
His 'defense' of the point is basically: Look, when you do TD
For many paragraphs, I thought this was a parody of TDD defense, but then it turns out it wasn't.
His 'defense' of the point is basically: Look, when you do TDD you have to put a lot more work into the tests than you thought! It is not just a simple thing!
Okay, fine, but ... Before embarking on TDD, the programmer had a picture in his head of what the costs+benefits of this change would be. Now you are telling him the costs are WAY higher. So a successful defense would have to then make the case that the benefits are also WAY higher.
But he doesn't. Because the benefits aren't higher, in fact they are lower (as is the case with every well-intended scheme in the history of anything.)
As usual my advice on this is: look at the people who build things you find highly impressive, and study how they did it. This is much more fruitful than reading the output of people who want to spend all day telling you how to program (which leaves very little time for them to build software that is impressive, i.e. they never even test their own ideas!)
I like your usual advice! I imagine there are a lot of very ingenious things hidden within the code of people who have never written an article. Or even just contradictions to articles that they may have written.
I find a lot of articles tend to just build upon common assumptions with a unique twist. Makes sense though, people want to read things that validate their assumptions. People want to write things that people want to read.
TDD is better than no tests most of the time I think. So at least that's a gain altogether.
I do TDD rarely, I like it when I'm on pristine projects, I feel it works when working alone on code you're outside your paying job. Or a library that you don't need to do much exploration programming.
I add tests after the fact for most of my programming, usually because that's how I feel more comfortable. Sometimes it is a net win to figure out how something works by experimenting with different ideas and start having a feel of the software you're creating. Without slowing down for trying to think about correctness every time you write something down.
TDD should not be, "An always on" kind of thing, I feel like most of what software has taught me is "Use where better suited" which this I think is the main reason why I feel I differ with his view which I think it is, use always. I kind of think that people with those bracelets about "always doing TDD" are thinking.
Yeah. To be honest the only unit tests I ever wrote... I wrote after I built something that I understood through and through and is likely to change very little.
I really only did it cause people tend to say... "don't use open source unless it has tests". I even did all the code coverage stuff... got it to 90%+
It was fun and amusing... certainly not a validation of TDD though.
I've used TDD both at my paying job and as part of some kinds of exploration. I agree it's more challenging in both places.
The place where I won't use it is when I'm writing throwaway code. If I'm really being exploratory, then I just go write garbage. And when I've done enough exploration to start writing real production code, I'll start fresh with TDD.
I've tried it the other way, where I retrofit tests to existing experimental code. But because the experimental code was to help me learn something, it generally ends up being poorly designed. It's only once I understand the big picture that I know the write way to express my understanding in code. At first I didn't like throwing out the experimental code, but now I prefer it in that it frees me to be entirely experimental, rather than writing something that's half experiment, half production grade.
It should be noted that TDD specifically means writing tests before you write application code.
For example, at our shop, we don't require devs to do TDD, however we require test coverage.
I don't think anyone would argue that writing tests at all is a bad idea, I think the debate is whether or not your tests should drive your application code.
I don't think anyone would argue that writing tests at all is a bad idea
If we're talking specifically about the kind of low-level unit tests required for TDD, that is still an assumption, though not an implausible one.
Even if unit testing is effective by some measure, it takes a significant investment of time to create and run those tests. TDD also constrains the software design and the development process.
Maybe we would do better to remove those constraints and instead spend that time on some other activity? Maybe some form of code review or walk-through exercise would be effective. Maybe we should be writing higher-level tests. Maybe we could formally prove some key parts of our code are correct. Learning new programming skills might stop us making some mistakes in the first place. Maybe we should even be adopting a new language or tool that would prevent some errors from being possible at all by design.
There are many ways we can try to make our code more reliable, and they all have costs, and sometimes they conflict. Even if one of them is better than nothing in isolation, that doesn't necessarily make it the best possible strategy when you consider the alternatives and opportunity cost.
> As usual my advice on this is: look at the people who build things you find highly impressive, and study how they did it.
Are there any big popular open source projects that follow TDD?
Yes: Cloud Foundry. Small teams, pair programming, TDD. I believe it is now north of a million lines, mostly Go, spread across a lot of complex, interacting subsystems. Teams are located on multiple continents, working for multiple companies.
At Pivotal we typically roll out the latest release versions of Cloud Foundry into production (PWS) in less than 48 hours after a release is cut.
> As usual my advice on this is: look at the people who build things you find highly impressive, and study how they did it. This is much more fruitful than reading the output of people who want to spend all day telling you how to program (which leaves very little time for them to build software that is impressive, i.e. they never even test their own ideas!)
"Uncle Bob" Martin has built a lot of impressive things. Most notably FitNesse [1] the service level test framework. He tests his ideas out and is a great teacher.
I've been doing TDD for years thanks to Uncle Bob, Martin Fowler, and Roy Osherove (who have all built impressive things) and I've leveled up as a result. In fact my entire team has leveled up with this simple discipline.
I've yet to meet an anti-TDD zealot who has actually spent a month developing the art of TDD. The detractors tend to be people who write really brittle tests that are a pain to maintain.
[1] - http://FitNesse.org http://FitNesse.org
I've been using FitNesse heavily at work for the past few years, and I've been surprised by how buggy we've found it.
We have over 30,000 tests and we've had to rewrite part of it because it memory-leaked so badly. Unfortunately the company has strict rules against contributing back to Open Source projects.
I just came to post something similar. Typical "Enterprise Java", much like Jenkins.
Free but very buggy.
Relevant: https://www.youtube.com/watch?v=IRTfhkiAqPw https://www.youtube.com/watch?v=IRTfhkiAqPw
Specifically the Uncle Bob example of OOP run amok.
I don't necessarily think it's even TDD vs not... I'm not big on TDD, but always try to design my code so that it is very module, and as a side effect easier to test. At least beyond some cross cutting edge cases, which are usually easy enough to isolate.
The act of TDD, or test compliance is only that you've looked at everything twice. It doesn't guarantee quality. Writing modular code means less friction over time, and TDD encourages this, but it's not necessarily a requirement imho.
> Because the benefits aren't higher, in fact they are lower
It depends on what you think the benefits are.
> As usual my advice on this is: look at the people who build things you find highly impressive, and study how they did it.
Good idea. Researchers at Microsoft and IBM conducted studies of applying TDD to real, commercial projects of varying sizes, languages, team structure and complexity.
They found that defects found in production were 40 to 90% lower. Development was slower by 15-35%, assuming that you don't count re-work after release towards the overall time spent on development.
http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf http://research.microsoft.com/en-us/groups/ese/nagappan_tdd....
These studies were not conducted with experienced TDDers, nor with TDD advocates on staff. Some of the teams abandoned TDD afterwards.
> But he doesn't. Because the benefits aren't higher, in fact they are lower
I am genuinely curious, what do you mean with this claim? I mean, yes the costs are high [1], but when properly paid, do the benefits still end up smaller than without practicing TDD? Which concrete benefits we miss when doing TDD?
Uncle Bob has stated many of benefits of TDD elsewhere and did not mention them in this post. Many of his public talks on the subject are available online.
[1] https://news.ycombinator.com/item?id=11311646 https://news.ycombinator.com/item?id=11311646
Edit: link to my previous comment from yesterday elaborating on the costs
It is important to read the rest of the sentence in order for this comment to really make sense.
I am talking about any scheme of how to do things that is intended to provide benefit. These all start with "wouldn't it be better if X, because Y" and then a plan is made of how to bring this about.
Well, this plan is inevitably imperfect, so it is either that you don't get all of X, or the reasons Y were not correctly understood or accounted for.
Then, there are always some extra drawbacks that creep in that negate some of the benefits. Usually these drawbacks are very subtle, and they can be hard to notice because they are not things that the plan was trying to address.
In the end, usually the net result is negative: the scheme causes more damage than it provides in benefit. But usually it takes a long time to understand this clearly, because the drawbacks can be subtle (but sometimes they aren't, for example, in TDD, how much extra code you are writing all the time).
If I read this sentiment correctly, this means that any attempt to improve anything ("provide benefit") usually results in net negative. But I cannot imagine that this is the case. Can you elaborate?
For example, let's say I want to improve my car factory efficiency. I introduce a way to keep things running without delays on the production line. Would you say that this plan is futile too? Toyota might disagree.
Would you also say that accountants waste half of their time doing double-entry bookkeeping? Half of the transactions are extra in the same sense.
I mean, I understand your argument on the imperfection of any improvement plan, but this does not say much on the actual topic on hand. Should we discuss concrete trade-offs instead of general sentiments?
Most software out there has been done without TDD.
Sure, but most software is also pretty bad. Most software projects aren't considered successful. [1] And 99% of people advocating TDD have written software without TDD, so they've tried it both ways.
Heck, when anesthesia was introduced, people thought it was a needless luxury and an interference with the pain God intended us to have. [2] And they were right in the way you were right: billions had just lived with the pain, so anesthesia wasn't really necessary. But something can be unnecessary and still be a good idea.
[1] https://www.google.com/search?site=&tbm=isch&q=software+project+failure+statistics https://www.google.com/search?site=&tbm=isch&q=software+proj...
[2] http://www.newyorker.com/magazine/2013/07/29/slow-ideas http://www.newyorker.com/magazine/2013/07/29/slow-ideas
Sure, but most software is also pretty bad.
That seems rather uncharitable.
Billions of people travel in software-controlled vehicles every day, and most of them will get to their destination safely and reasonably efficiently.
When I check my bank account or credit card statement, it is extremely unlikely that anything on it is incorrect, even though I may have been transacting with other parties all around the world, with several completely different organisations involved along the way.
When I make my dinner, it is a safe bet that all my kitchen equipment will function properly, even though everything from my microwave to my refrigerator is software controlled.
If I pick up my wireless landline handset and place a call, chances are nothing short of severe interference is going to prevent me from connecting to the phone of the person I want to speak to, again even though this might involve intricate negotiations between numerous devices and even different organisations in different countries.
The world is full of software that is actually pretty good considering it was made by fallible humans. We don't notice a lot of the time, because the things depending on that software just work.
> But he doesn't. Because the benefits aren't higher, in fact they are lower (as is the case with every well-intended scheme in the history of anything.)
The benefits of X are always lower than any naive adopter thinks? That's quite a claim. Would you care to justify that? Heck, don't even worry about the general case, just explain it for this one.
I've been doing TDD for many years at this point. Right now I'm working on a hobby project where I was intentionally sloppy about testing, just to see how little I could get away with. The answer is: in the long term, I can get away with surprisingly little. But there's a lot of subtlety in exactly how and when the benefits come, and when exactly sloppiness is ok.
Given that I'm still learning about the benefits of TDD despite many years of practice and thought, I'd say there's no reason to think a novice would have a particularly good idea. Maybe their expectations are unrealistically low, maybe high. But more likely it's both and neither. Maybe one's very notion of "benefit" changes over time. TDD has certainly done that for me.
I think he did state that the benefit is also much higher, but it was a bit subtle. He argues that the cost is higher than expected because you have to design the tests and design the system to be testable. He then states that this isn't wasted effort because, "Something that is hard to test is badly designed." And then hammers the point home with the pacemaker story and a bit of ranting.
I agree with him. It took me a few years to realize that if code is hard to test, it is hard to debug. And if it is hard to debug, it is hard to reason about. And if it is hard to reason about, then it is likely to have hidden bugs.
One nuance you might have failed to capture in your characterization of the article is that a lot of the cost is high when the subject is a beginner on the technique. The point is that as you gain mastery it becomes easier, thus encouraging people who are interested in the technique not to give up so fast.
In particular one side-effect of being a beginner is that one spends a lot of mental energy on the technique, leaving less room to focus on other important aspects. That gets better with time.
"I've yet to meet an anti-TDD zealot who has actually spent a month developing the art of TDD. The detractors tend to be people who write really brittle tests that are a pain to maintain."
I'm not an anti-TDD zealot, but I don't personally do it, nor require any of my developers to do it (but they can, if it helps them).
As with everything in software, some things work well for some people, and not so much for others.
Instead of TDD, HDD works best for me. https://www.youtube.com/watch?v=f84n5oFoZBc https://www.youtube.com/watch?v=f84n5oFoZBc
Rich also said, I think we're in this world I'd like to call Guard Rail Programming... 'I can make change because I have tests!' Who does that? Who drives their car around, banging against the guard rails? Do the guard rails help you get to where you want to go?"
http://patrick.lioi.net/2011/11/23/guard-rail-programming/ http://patrick.lioi.net/2011/11/23/guard-rail-programming/
TDD has always seemed to me to be another sketchy, cult-like following in the Agile ecosystem. I've never seen anybody be productive at it, even if they did find some kind of solution to a problem long after they would have doing normal development.
But what do I know, I'm not even a fan of unit tests. I'd rather go to the real-deal and run some acceptance/integration test at 2 in the morning.
> I've yet to meet an anti-TDD zealot who has actually spent a month developing the art of TDD. The detractors tend to be people who write really brittle tests that are a pain to maintain.
OK, sure. I'm basically pro-TDD, seeing as how everyone's taking sides. Yet there's a problem here, which is that (as Martin says here) bad testing is basically a design problem. To write better tests, learn better software design - which is to say, you do not do it (or at least most efficiently) by doing more TDD.
This stuff is not in the brochure. And it should be; because without it, the claims made by TDD evangelists are somewhat misleading. No, it will not make you a better engineer on its own, at all. No, it is not a substitute for getting your whiteboard marker out and drawing a few boxes, or thorough code reviews, or whatever else.
Sandi Metz makes this point in Practical Object Oriented Design in Ruby[0]; and the chapter on testing is right at the end of that book, presumably for the same reasons (get the design sense first, then you'll get a sane red-green-refactor thing going).
Note that I'm not talking about Ian Sommerville here - I'd never heard of him until this whole contretemps, but I'm given to understand he knows a thing or two about design. Perhaps if he really stuck at TDD, Bob Martin's favourite bit would flip[1]. For less experienced programmers, who have not yet felt the pain of maintaining a ball-of-mud and learned a thing or two about separating concerns, TDD is not going to help without ongoing education about design.
[0] http://www.poodr.com/ http://www.poodr.com/ . The design principles are different outside of the OO world, of course; but not the principle of design!
[1] http://blog.8thlight.com/uncle-bob/2012/01/11/Flipping-the-Bit.html http://blog.8thlight.com/uncle-bob/2012/01/11/Flipping-the-B...
Huh. I've seen TDD help people improve their design skills. It certainly has helped me.
It forces a very short feedback loop between acts of design and experiencing the design as a consumer. That gives people immediate feedback on bad design choices, giving them opportunities to see the problems they're creating.
It also forces people to think immediately about the consumer perspective of a design rather than the implementation perspective. Instead of mentally being inside the objects and methods they're building, they have to start thinking about them from the outside.
I agree it's not a substitute for getting up and going to a whiteboard. But, then, going to a whiteboard is not a substitute for TDD. They're mainly working at different levels of design; when I'm at a whiteboard we're generally talking about the relationship of relatively large pieces of a system. Whereas during TDD you're mainly confronting the fine details of design.
Back when TDD started, it was often referred to as Test Driven DESIGN by the same people you cite.
I do it, and I find it valuable. But I think if you're not using it as a design tool, you're missing most of the benefit.
You don't do TDD (IMO) to prove correctness. You do it to achieve a level of composition and design that makes later requirements changes and refactoring less painful. Proving that your stuff actually works is a convenient side effect. But not the primary goal because tests have costs.
I haven't followed Osherove for years, so maybe his thinking has changed, but it used to be common to think of chasing high test coverage numbers as an anti-pattern. 70 to 80% is the sweet spot. You're designing. You're being productive. That last 10, 20 or 30% of coverage gets exponentially more expensive both to develop and maintain, and it provides no additional design benefit. It's only testing for it's own sake.
BDD and the culture that sprang up around it (at least in the Ruby community) was such a disappointment after having done TDD for a few years previous. It's like every known anti-pattern was adopted as a core deliverable and the entire point of the exercise (IMO, solving the "blank page" problem in building systems; Design) was forgotten entirely.
It's my experience that Developers who've adopted this style of "TDD" are cargo culting, and incredibly difficult to work with. Reason didn't get them into the belief. They can't be reasoned out of it. It's religion by that point.
Now that I'm in Scala, and TDD my own code, I'm a very happy developer. I never suffer blank-page issues. I never bother writing tests after the fact. I don't often encounter bugs in such code, and they're almost never fundamental design issues when I do. And my coverage proudly hovers around 70%.
That is how you TDD right IMO. Follow the lessons learned a decade or more ago and avoid the anti-patterns.
Chasing test coverage is not only a good way to light piles of somebody else's money on fire by wasting time for little benefit, but it's actively harmful to the quality of your code base over the long term. You're disincentivized to correct design mistakes, and you're encouraged to over design to enable a level of testing granularity that never paid any concrete benefits for itself in the first place.
I'm a solid proponent of TDD then. But it's like saying I think water is good and you can prove it when most people are trying to sell you "mineral rich" Iron tainted industrial runoff. TDD absolutely can increase costs and complexity in the wrong hands.
The moral of my rambling story is to be Agile I guess. In the original sense, not the consultation services one. To the inexperienced developer I'd say: Try to solve problems, not implement solutions. And above all, never cargo cult. When someone tells you you need 100% coverage, ask them what benefit it provides. When someone tells you to "measure everything", ask them if they've measured the business value of measuring everything and if it outweighed the opportunity cost and dollars sunk into the effort. Be a constant skeptic, because snake oil is everywhere.
Sorry for the diatribe. You took me down memory lane and I guess I feel pretty strongly about TDD.
> BDD and the culture that sprang up around it (at least in the Ruby community) was such a disappointment after having done TDD for a few years previous. It's like every known anti-pattern was adopted as a core deliverable and the entire point of the exercise (IMO, solving the "blank page" problem in building systems; Design) was forgotten entirely.
Could you elaborate?
Speaking for myself, I thought that the main useful thing to come out of BDD was working outside-in and the given/when/then convention.
But I certainly didn't get it on my first pass. Looked at Cucumber, soldiered mightily, gave up in disgust.
> I've yet to meet an anti-TDD zealot who has actually spent a month developing the art of TDD. The detractors tend to be people who write really brittle tests that are a pain to maintain.
I guess I loosely fall into the category of "anti-TDD". I don't generally go around badmouthing it, but I'm not really sold on the idea that it would save me significant time or effort. I certainly encounter things that TDD would likely catch earlier, but those are typically easy fixes and caught in QA or beta testing. The majority of my bugfixing hours are eaten up by things for which I don't even know what it would mean to write tests, such as:
- reproducing rare hardware-involved errors (often race conditions, often based on the variability of random-ish real-world events) in a way that allows getting relevant information logged/dumped
- clarifying product definition/spec, usually to resolve some inconsistency where the behavior isn't invalid per se but implicitly conflicts with some other requirement (i.e. we basically decide which behavior to call a bug and change that one)
- reverse-engineering undocumented design decisions of some third-party product that our product is expected to work with, because we foolishly believed that there was a standard (overlaps somewhat with #1)
So while I don't claim that TDD has no value, I admit to rolling my eyes a bit at the more fervent evangelists.
The Nagappan study is one of the few that at least tried to do a decent job, but there ought to be some sort of magic auto-responder every time someone cites it that points out that none of the teams they studied was actually doing "pure TDD" as originally advocated by those who coined the term, so the baseline for all those comparative figures isn't entirely clear.
It's also pertinent that it always seems to be that same one report from nearly a decade ago, studying just a few development groups, that gets cited whenever anyone asks for hard data about the effectiveness of TDD. If TDD is as necessary as its more fervent advocates claim, then surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples.
> none of the teams they studied was actually doing "pure TDD"
And still showed dramatic improvements in defect prevention.
> surely after all this time we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples.
The problem is funding. Nobody wants to stump up their project for it.
Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out.
And still showed dramatic improvements in defect prevention.
Right, but because neither what was being tested as close to TDD nor the alternative were fully controlled, we can't tell from the cases studied whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts.
Oh, except at Pivotal, I guess. We've only done a few hundred projects with TDD over several decades. I guess it'll never pan out.
Good for you. Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too. Unfortunately, none of those anecdotes in isolation tell us anything very useful about whether things would have panned out better following some other development process.
> whether it was the full TDD process that would make a difference, or just writing more unit tests, writing those tests first, writing them at all, or various other changes in process that accompanied the TDD-like shifts.
While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect.
Here's what we know: under varying conditions, in multiple teams, with multiple levels of experience, in various problem domains, in multiple languages, in multiple companies in multiple locations, even a half-baked TDD had a powerful effect.
Small sample sizes suck. But when you have a small sample and a very large effect, then the null hypothesis is under serious pressure.
> Thousands of other organisations with millions of developers working for them have done thousands of times as many projects as you have without TDD over those same decades, and plenty of them panned out too.
The point is not "did they pan out?" (modulo the usually utterly abysmal "success" rate in our industry). The point is "could they have panned out better?"
> Unfortunately, none of those anecdotes in isolation tell us anything very useful
I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences.
I did so because you said "we should have a vast body of evidence where real world development teams have tried it and found it clearly superior to what they were doing before, not just a few isolated examples."
Then you shifted the goalposts.
Pivotal is probably the most doctrinaire TDD company in the world. If you can find another company that has done TDD for as long, and as consistently, on as many projects, with as many different engineers, but showed a consistent failure rate, I'd love to hear about it.
Your argument is "if it's so good, why hasn't everyone switched?"
Because people are people. TDD is hard and an unfamiliar practice to almost everyone. It requires someone to teach, which makes it frustrating to the autodidacts who fill the industry.
You might as well ask why we haven't all switched to Dvorak keyboards, why VHS beat Betacam, why we were stuck at 1080p screens for a decade, why we use enriched-uranium high pressure water reactors, why x86 beat every other ISA for decades and on and on and on. That a practice or technology is superior on the metrics we are "supposed" to care about is no guarantor of success. Contrariwise, an argument from popularity proves or disproves nothing about the actual empirical properties of a practice.
I guess if your criterion for "good software" is "mostly works" then that's fine. Mine are better.
But I have seen bank code. It's terrible. I've seen credit card handling code. It's terrible too. Phones used to be reliable devices, but the average uptime of my phones has fallen to something like 3 days. Perhaps 3x/day my phone tells me that some app has stopped and that it would like to send a bug report. The code that runs cars is terrible too: http://www.safetyresearch.net/blog/articles/toyota-unintended-acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code http://www.safetyresearch.net/blog/articles/toyota-unintende...
Most companies I visit have bug databases with hundreds or thousands of open problems. And that's the just ones they know about, the ones where people take the time to report the bug. Things are terrible and people are just used to it. In the aerospace industry, it's called "normalization of deviance", and it's what destroy the Challenger.
None of this is necessary. It's been at least 15 years that Kent Beck has been talking about the various quality practices that drop bug rates dramatically, including test-driven development, pair programming, and continuous integration. It has been at least 10 years that Martin Fowler has been reporting on teams that have bug rates below 1 per team member per month.
I'm certainly willing to be charitable about the people. A lot of organizations making terrible software are staffed with perfectly nice people who mean well. But I decline to be charitable about the software. These are commercial products, not ash-trays made by second-graders at day camp. The software should stand and fail on its own, with no charity needed to soften the blows.
While it is useful to tease out the contributory causes for why adopting even a half-baked TDD had such a powerful effect, in the meantime, the fact is that even a half-baked TDD had a powerful effect.
You've invented this term "half-baked TDD", but that seems a little unfair. My point was that the groups in the Nagappan study were doing significantly more than just TDD. For example, they also had varying levels of dedicated design activities in addition to anything test-driven they were doing. It also wasn't clear to what extent some of them were doing similar kinds of unit testing in the alternative scenarios from the study. So you have neither a clear baseline nor a clear change.
For example, one alternative possibility that still seems reasonably consistent with the evidence in that study is that unit testing is good at improving quality, that TDD promotes writing more tests, and that any design weaknesses that a TDD style might encourage in isolation were mitigated by the separate design activities those groups were doing beyond the basic fail-pass-refactor cycle required for TDD.
The point is "could they have panned out better?"
Erm... Yes. I just made exactly that point, in my very last post, the one you were replying to.
I present Pivotal as a falsification of the argument that "true" TDD hasn't been tried for a long time in commercial settings, with real clients, with real projects, with real consequences.
Who was making that argument? I certainly wasn't.
Your argument is "if it's so good, why hasn't everyone switched?"
No, it isn't. Let's be clear about this.
Prominent TDD advocates, Bob Martin among them, claim quite unambiguously that TDD is essential to writing good software, even using patronising and insulting language like "unprofessional" to describe anyone who doesn't do it.
If that were actually true, if TDD is inherently superior to any other development process and anyone not following it is actively doing programming wrong, then the results achieved by organisations using TDD should be clearly and consistently superior to the results achieved by those not using TDD, other factors being reasonably equivalent.
I claim that this is not the case, and I cite as evidence the simple fact that after nearly two decades TDD still represents a tiny part of the industry practice. This is an industry that over the same time period has brought numerous minor programming languages to prominence and mainstream acceptance; brought many ideas from fields like functional programming and distributed systems from niche applications or academic studies into the mainstream along the way; shifted a large proportion of mainstream development from the previous desktop/server model to web apps, cloud hosting, mobile development, and much more sophisticated embedded systems; made dramatic shifts in development processes such as the migration to a DevOps style of integration and deployment in many cases; and seen the rise of Open Source and more generally of collaborative development from a fun pastime for geeks to a strong influence behind much of the software we run every day.
Given that, I'm sorry but I find it patently absurd to argue that the only reasons hardly anyone is doing TDD, even though it is so inherently superior in both quality of results and cost effectiveness, are that it is hard or unfamiliar. Many of those other changes I mentioned above have been adopted across large parts of the industry within much less time, even though the ideas were completely new and/or required completely different mindsets and understanding.
If TDD really were essential to get good results and so clearly superior to other development processes as the evangelists frequently claim, then as I said before, by now we should see a vast pile of evidence, not just the same study with a sample size of four being cited nearly a decade later, and not just the occasional business -- particularly one that by your own admission is "probably the most doctrinaire TDD company in the world" -- that has used TDD and not failed. Many of us worked on software projects that have not failed. Not failing is table stakes for this debate.