5 ms·

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the

by ekidd 9y ago

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart: 1. John writes some good code, with good integration tests and good unit tests. He understands the code base. When he deploys his code, he finds a couple of bugs and adds regression tests. 2. Six months later, John needs to work on his code again to replace a low-level module. He's forgotten a lot of details. He makes some changes, but he's forgotten some corner cases. The tests fail, showing him what needs to be fixed. 3. A year later, John is busy on another project, and Jane needs to take over John's code and make significant changes. Jane's an awesome developer, but she just got dropped into 20,000 lines of unfamiliar code. The tests will help ensure she doesn't break too much. Also, unit tests (and specifically TDD) can offer two additional advantages: 1. They encourage you to design your APIs before implementing them, making APIs a bit more pleasant and easier to use in isolation. 2. The "red-green-refactor-repeat" loop is almost like the "reward loop" in a video game. By offering small goals and frequent victories, it makes it easier to keep productivity high for hours at a time. Sometimes you can get away without tests: smaller projects, smaller teams, statically-typed languages, and minimal maintenance can all help. But when things involve multiple good developers working for years, tests can really help.

sloppycee 9y ago
Great points, especially: > Jane's an awesome developer, but she just got dropped into 20,000 lines of unfamiliar code. The tests will help ensure she doesn't break too much. In my experience, a well tested codebase lowers the risk and decreases the time to productivity of new hires.
mattmanser 9y ago
I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF. I also have no problems designing APIs, experience is what you need and the experience in asking the right questions. No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements. Other than that API design is (fairly) trivial. I spend a tiny amount of my overall programming time on it. I will map out entire classes and components for large chunks of functionality without filling in any of the code in less than an hour or two and without really thinking about it. Then the hard part of writing the code begins which might take a month or two, and those data structures and API designs will barely change. I see my colleagues doing similar check-ins, un-fleshed components with the broad strokes mapped out. I'd say it's similar to how we never talk about SQL normal forms any more. 15 years ago the boards would discuss it at length. Today no-one seems to. Why? Because we understand it and all the junior programmers just copy what we do (without realizing it took us a decade to get it right). API design was hard, but today it's not, we know what works and everyone uses it. We've learnt from Java Date and the vagaries in .net 1.0 or PHP's crazy param ordering and all the other difficult API designs from the past and now just copy what everyone else does.
- ekidd 9y ago
  > I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF. I've gone back through most of the PDF, and I can't figure out what section you're referring to. There's a bunch of discussion of perverse incentives (mostly involved incompetent managers or painfully sloppy developers, who will fail with any methodology), but I don't see where the author addresses what I'm talking about. Specifically, I'm talking about the hour-to-hour process of implementing complex features, and how tests can "lead" you through the implementation process. It's possible to ride that red-green-refactor loop for hours, deep in the zone. If a colleague interrupts me, no problem, I have a red test case waiting on my monitor as soon as I look back. This "loop" is hard to teach, and it requires both skill and judgment. I've had mixed results teaching it to junior developers—some of them suddenly become massively productive, and others get lost writing reams of lousy tests. It certainly won't turn a terrible programmer into a good one. If anything, the biggest drawback of this process is that it can suck me in for endless productive hours, keeping me going long after I should have taken a break. It's too much of a good thing. Sure, I've written some of the most beautiful and best-designed code in my life inside that loop. But I've also let it push me deep into "brain fry". > No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements. I've occasionally been lucky enough to work on projects where all the requirements could be known in advance. These tend to be either very short consulting projects, or things like "implement a compiler for language X." But usually I work with startups and smaller companies. There's a constant process of discovery—nobody knows the right answer up front, because we have to invent it, in cooperation with paying customers. The idea that I can go ask a bunch of people what I need to build in 6 months, then spend weeks designing it, and months implementing it, is totally alien. We'd be out of business if it took us that long to try an idea for our customers. It's possible to build good software under these conditions, with fairly clean code. But it takes skilled programmers with taste, competent management and a good process. Testing (unit, integration, etc.) is a potentially useful part of that process, allowing you to adapt to changing requirements while minimizing regressions.
  - jrumbut 9y ago
    Your comments on this thread have really helped clarify my thinking on this important issue.
falcolas 9y ago
Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the code is supposed to do. What happens is that John and Jane make a change, find the failing unit tests, and they deem the test failures as reasonable given the changes they were asked to make. They then change the tests to make them pass again. Again, the unit tests are providing no actual indication that their changes were the correct changes to make. WRT the advantages: 1. "design your APIs before implementing them" - this only works out if we know our requirements ahead of time. Given our acknowledgement that these requirements are usually absent via Agile methodologies, this benefit typically vanishes with the first requirement change. 2. "makes it easier to keep productivity high for hours at a time" This tells me that we're rewarding the wrong thing: the creation of passing tests, not the creation of correct code. Those dopamine hits are pretty potent, agreed, but not useful.
- AnimalMuppet 9y ago
  > Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look. That's miles better than "I hope this change doesn't break anything". > "design your APIs before implementing them" - this only works out if we know our requirements ahead of time. No. "API" here includes things as small as the public interface to a class, even if the class is never used by anything other than other classes in the module. You have some idea of the requirements for the class at the time when you're writing the class; otherwise, you have no idea what to write! But writing the test makes you think like a user of that class, not like the author of that class. That gives you a chance to see places where the public interface is awkward - places that you wouldn't see as the class author. > This tells me that we're rewarding the wrong thing: the creation of passing tests, not the creation of correct code. We're rewarding the creation of provably working code. I fail to see how that's "the wrong thing".

by ekidd 9y ago

sloppycee 9y ago

Great points, especially: > Jane's an awesome developer, but she just got dropped into 20,000 lines of unfamiliar code. The tests will help ensure she doesn't break too much. In my experience, a well tested codebase lowers the risk and decreases the time to productivity of new hires.

mattmanser 9y ago

I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF. I also have no problems designing APIs, experience is what you need and the experience in asking the right questions. No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements. Other than that API design is (fairly) trivial. I spend a tiny amount of my overall programming time on it. I will map out entire classes and components for large chunks of functionality without filling in any of the code in less than an hour or two and without really thinking about it. Then the hard part of writing the code begins which might take a month or two, and those data structures and API designs will barely change. I see my colleagues doing similar check-ins, un-fleshed components with the broad strokes mapped out. I'd say it's similar to how we never talk about SQL normal forms any more. 15 years ago the boards would discuss it at length. Today no-one seems to. Why? Because we understand it and all the junior programmers just copy what we do (without realizing it took us a decade to get it right). API design was hard, but today it's not, we know what works and everyone uses it. We've learnt from Java Date and the vagaries in .net 1.0 or PHP's crazy param ordering and all the other difficult API designs from the past and now just copy what everyone else does.

ekidd 9y ago
> I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF. I've gone back through most of the PDF, and I can't figure out what section you're referring to. There's a bunch of discussion of perverse incentives (mostly involved incompetent managers or painfully sloppy developers, who will fail with any methodology), but I don't see where the author addresses what I'm talking about. Specifically, I'm talking about the hour-to-hour process of implementing complex features, and how tests can "lead" you through the implementation process. It's possible to ride that red-green-refactor loop for hours, deep in the zone. If a colleague interrupts me, no problem, I have a red test case waiting on my monitor as soon as I look back. This "loop" is hard to teach, and it requires both skill and judgment. I've had mixed results teaching it to junior developers—some of them suddenly become massively productive, and others get lost writing reams of lousy tests. It certainly won't turn a terrible programmer into a good one. If anything, the biggest drawback of this process is that it can suck me in for endless productive hours, keeping me going long after I should have taken a break. It's too much of a good thing. Sure, I've written some of the most beautiful and best-designed code in my life inside that loop. But I've also let it push me deep into "brain fry". > No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements. I've occasionally been lucky enough to work on projects where all the requirements could be known in advance. These tend to be either very short consulting projects, or things like "implement a compiler for language X." But usually I work with startups and smaller companies. There's a constant process of discovery—nobody knows the right answer up front, because we have to invent it, in cooperation with paying customers. The idea that I can go ask a bunch of people what I need to build in 6 months, then spend weeks designing it, and months implementing it, is totally alien. We'd be out of business if it took us that long to try an idea for our customers. It's possible to build good software under these conditions, with fairly clean code. But it takes skilled programmers with taste, competent management and a good process. Testing (unit, integration, etc.) is a potentially useful part of that process, allowing you to adapt to changing requirements while minimizing regressions.
- jrumbut 9y ago
  Your comments on this thread have really helped clarify my thinking on this important issue.

falcolas 9y ago

Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the code is supposed to do. What happens is that John and Jane make a change, find the failing unit tests, and they deem the test failures as reasonable given the changes they were asked to make. They then change the tests to make them pass again. Again, the unit tests are providing no actual indication that their changes were the correct changes to make. WRT the advantages: 1. "design your APIs before implementing them" - this only works out if we know our requirements ahead of time. Given our acknowledgement that these requirements are usually absent via Agile methodologies, this benefit typically vanishes with the first requirement change. 2. "makes it easier to keep productivity high for hours at a time" This tells me that we're rewarding the wrong thing: the creation of passing tests, not the creation of correct code. Those dopamine hits are pretty potent, agreed, but not useful.

AnimalMuppet 9y ago
> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look. That's miles better than "I hope this change doesn't break anything". > "design your APIs before implementing them" - this only works out if we know our requirements ahead of time. No. "API" here includes things as small as the public interface to a class, even if the class is never used by anything other than other classes in the module. You have some idea of the requirements for the class at the time when you're writing the class; otherwise, you have no idea what to write! But writing the test makes you think like a user of that class, not like the author of that class. That gives you a chance to see places where the public interface is awkward - places that you wouldn't see as the class author. > This tells me that we're rewarding the wrong thing: the creation of passing tests, not the creation of correct code. We're rewarding the creation of provably working code. I fail to see how that's "the wrong thing".