3 ms·

How do you test your test if you don't write the test first? If you write the code, and then you write the test, then you haven't tested your test. You have no

How do you test your test if you don't write the test first? If you write the code, and then you write the test, then you haven't tested your test. You have no proof that your test will detect broken code. The only way to prove that your test can detect broken code is to run it against broken code. So you can write the code, then write the test, and then break the code and run the test. Or you could just write the test first, and run it. It should fail. Surprisingly, sometimes it doesn't: either because the test was bad, or the existing code didn't work like you thought, or sometimes the language doesn't behave like you think. I've dealt with hundreds of thousands of lines of tests where I could go into the code and just start deleting functionality en masse, and no tests failed. If you don't test your tests, they're just bloat. They're the kind of pointless bloat people are complaining about in these unit testing threads.

Verdex_3 9y ago
Although on the other hand, writing your test first with no code and having it fail isn't really telling you much about the test. It just shows that the test handles the least interesting case (the case where there exists no code). You still don't know that your test is correct when you get it passing. You just know that it is correct when no code exists. For example, TestSort(ISorter sorter) { var input = { 1, 2, 3, 4 }; var output = sorter( input ); Assert.IsSorted( output ); } This test will fail if you run it before the ISorter is implemented. But then it will pass if you implement an empty ISorter because the test still has problems (ie it starts with sorted input). Cue the ever familiar: You still have to use your brain when using TDD. If TDD needs brain usage to function, maybe the real thing that is useful is the brain usage and TDD is vestigial.
- lowbloodsugar 9y ago
  If someone gave me that test, I'd implement the code as "return input;" Forget that you, a human, thinks you know what an ISorter should do. What do the tests demand that you do? Doing TDD, the goal, when coding, is to write the minimal amount of broken code that makes the test pass. If you passed me {4,3,2,1}, then my code would be "return {1,2,3,4};". If you wrote a second test that passed me {6,5,4,3}, my code would be "return (input[0]==4)?{1,2,3,4}:{3,4,5,6};". You've got to write a test that makes me actually code what you want. I use this adversarial approach even when I'm writing both the test and the code. Usually the first test I write if I'm starting with a blank slate, is to pass null and check that I get a NullPointerException. That gets the class created. After that, its got to be randomly generated data.
  - Verdex_3 9y ago
    And that works fine ... as long as you know all of the cases that will cause the minimal amount of broken code to fail in a test case. What happens if you're implementing something like a unification algorithm, but you don't know about the occurs check. You'll progressively add test cases that break your code and eventually stop. Until an end user creates a query that contains itself and your algorithm fails to terminate. The solution to get TDD to work is to know what all of the weird edge cases of your algorithm are. However, I assert that the thing that makes TDD ever work is that the practitioners who are "doing it right" already know all of the weird edge cases of their algorithms. And I assert that if you know the weird edge cases you can drop the test first part of TDD and still get working algorithms. Random data is interesting because it's not something I've seen anyone mention when they talk about how they do TDD before now. I suppose if you knew nothing about your problem and had some sort of tool progressively feed you intelligently generated random data, that might work for algorithms that have few edge cases or relatively simple edge cases. However, it seems like you would want something more like quick check for cases where you have really esoteric edge cases. And I don't see how you avoid implementing bubble sort (or its equivalent in your domain). I don't think TDD can't work, but so far nobody has described TDD to me where I have any reason to assume that TDD is actually doing anything worthwhile. It's always sounded like TDD was irrelevant and the real key to success was thinking about the problem carefully. Additionally, it sounds like there are problem domains where anything that TDD might bring to the table would be nullified (ie certain problems that exhibit a certain level of complexity).
    - lowbloodsugar 9y ago
      I can only suggest that you commit to it for some considerable time, such as six months. I, too, thought it was, at first, a fucking stupid idea, and then, when people I respected continued to advocate it, grudgingly tried it, and thought it was a pain in the ass - this was in 2006. However, the number of times where I've written some code that I thought worked, and then commented it out and rewrote it using TDD, and found bugs, is too high. Its rare, but its too high. So either I'm not that great a programmer, or TDD is useful. Another way to think of TDD is rubber ducking, perhaps. So you're in a local maximum, and to get to the next, higher, maximum, you're going to have to go through a trough.
eric_h 9y ago
When I write a test after altering/adding code, I will generally stash the altered application code, and run the test I wrote on unaltered app code to confirm that it fails. As long as one maintains decent version control discipline, you can be pretty confident that the test is valid with this technique. (It does get a bit more difficult when there are schema changes, but then it also encourages good discipline and vigilance when you're making schema changes ;)
- lowbloodsugar 9y ago
  That's good. I suggest that you're doing unnecessary work (having to stash and unstash) when you had "broken" code at first, but at least you are testing your tests, so well done. In my experience coaching teams on TDD, its quite easy, when writing code first, to add more code than is needed to solve the problem. And then only the problem is tested for. Meaning that there is code that is not tested. Maybe an if statement where a branch is never taken. Strict TDD means you don't write any code without a failing test, and you write the minimal code needed to make it pass. Now quite often, a programmer has no idea how to tackle a problem, so I advise that they just write something that works. Just have a go and get something working. Most programmers (if they test at all) will then just go and write a test for it, usually for happy path. And that usually results in code with bugs. I know this because many times the programmer has done as I suggested and commented out their code, and reimplemented from scratch using TDD, and doing TDD finds a bug in the original that nobody saw.
  - eric_h 9y ago
    Oh absolutely. I've done my fair share of "pure" TDD, but my style has evolved over the years to not always favor a pure approach (usually in the interest of time). It really comes down to the nature of the problem - if I foresee a chance that the final architecture is not immediately obvious, I'll start breaking down the problem into small pieces and TDD the units, then the integrations. I guess I'm of the opinion that not everything needs TDD, but everything does need diligence - my code after doing years of TDD has improved drastically, whether or not it was strictly test driven. Merely considering whether it will be easy to reliably test the code improves it, IME.
gvdenbro 9y ago
you could test your tests using mutation testing [0]. it's not bulletproof but quite close. [0] https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing

Verdex_3 9y ago

Although on the other hand, writing your test first with no code and having it fail isn't really telling you much about the test. It just shows that the test handles the least interesting case (the case where there exists no code). You still don't know that your test is correct when you get it passing. You just know that it is correct when no code exists. For example, TestSort(ISorter sorter) { var input = { 1, 2, 3, 4 }; var output = sorter( input ); Assert.IsSorted( output ); } This test will fail if you run it before the ISorter is implemented. But then it will pass if you implement an empty ISorter because the test still has problems (ie it starts with sorted input). Cue the ever familiar: You still have to use your brain when using TDD. If TDD needs brain usage to function, maybe the real thing that is useful is the brain usage and TDD is vestigial.

lowbloodsugar 9y ago
If someone gave me that test, I'd implement the code as "return input;" Forget that you, a human, thinks you know what an ISorter should do. What do the tests demand that you do? Doing TDD, the goal, when coding, is to write the minimal amount of broken code that makes the test pass. If you passed me {4,3,2,1}, then my code would be "return {1,2,3,4};". If you wrote a second test that passed me {6,5,4,3}, my code would be "return (input[0]==4)?{1,2,3,4}:{3,4,5,6};". You've got to write a test that makes me actually code what you want. I use this adversarial approach even when I'm writing both the test and the code. Usually the first test I write if I'm starting with a blank slate, is to pass null and check that I get a NullPointerException. That gets the class created. After that, its got to be randomly generated data.
- Verdex_3 9y ago
  And that works fine ... as long as you know all of the cases that will cause the minimal amount of broken code to fail in a test case. What happens if you're implementing something like a unification algorithm, but you don't know about the occurs check. You'll progressively add test cases that break your code and eventually stop. Until an end user creates a query that contains itself and your algorithm fails to terminate. The solution to get TDD to work is to know what all of the weird edge cases of your algorithm are. However, I assert that the thing that makes TDD ever work is that the practitioners who are "doing it right" already know all of the weird edge cases of their algorithms. And I assert that if you know the weird edge cases you can drop the test first part of TDD and still get working algorithms. Random data is interesting because it's not something I've seen anyone mention when they talk about how they do TDD before now. I suppose if you knew nothing about your problem and had some sort of tool progressively feed you intelligently generated random data, that might work for algorithms that have few edge cases or relatively simple edge cases. However, it seems like you would want something more like quick check for cases where you have really esoteric edge cases. And I don't see how you avoid implementing bubble sort (or its equivalent in your domain). I don't think TDD can't work, but so far nobody has described TDD to me where I have any reason to assume that TDD is actually doing anything worthwhile. It's always sounded like TDD was irrelevant and the real key to success was thinking about the problem carefully. Additionally, it sounds like there are problem domains where anything that TDD might bring to the table would be nullified (ie certain problems that exhibit a certain level of complexity).
  - lowbloodsugar 9y ago
    I can only suggest that you commit to it for some considerable time, such as six months. I, too, thought it was, at first, a fucking stupid idea, and then, when people I respected continued to advocate it, grudgingly tried it, and thought it was a pain in the ass - this was in 2006. However, the number of times where I've written some code that I thought worked, and then commented it out and rewrote it using TDD, and found bugs, is too high. Its rare, but its too high. So either I'm not that great a programmer, or TDD is useful. Another way to think of TDD is rubber ducking, perhaps. So you're in a local maximum, and to get to the next, higher, maximum, you're going to have to go through a trough.

eric_h 9y ago

When I write a test after altering/adding code, I will generally stash the altered application code, and run the test I wrote on unaltered app code to confirm that it fails. As long as one maintains decent version control discipline, you can be pretty confident that the test is valid with this technique. (It does get a bit more difficult when there are schema changes, but then it also encourages good discipline and vigilance when you're making schema changes ;)

lowbloodsugar 9y ago
That's good. I suggest that you're doing unnecessary work (having to stash and unstash) when you had "broken" code at first, but at least you are testing your tests, so well done. In my experience coaching teams on TDD, its quite easy, when writing code first, to add more code than is needed to solve the problem. And then only the problem is tested for. Meaning that there is code that is not tested. Maybe an if statement where a branch is never taken. Strict TDD means you don't write any code without a failing test, and you write the minimal code needed to make it pass. Now quite often, a programmer has no idea how to tackle a problem, so I advise that they just write something that works. Just have a go and get something working. Most programmers (if they test at all) will then just go and write a test for it, usually for happy path. And that usually results in code with bugs. I know this because many times the programmer has done as I suggested and commented out their code, and reimplemented from scratch using TDD, and doing TDD finds a bug in the original that nobody saw.
- eric_h 9y ago
  Oh absolutely. I've done my fair share of "pure" TDD, but my style has evolved over the years to not always favor a pure approach (usually in the interest of time). It really comes down to the nature of the problem - if I foresee a chance that the final architecture is not immediately obvious, I'll start breaking down the problem into small pieces and TDD the units, then the integrations. I guess I'm of the opinion that not everything needs TDD, but everything does need diligence - my code after doing years of TDD has improved drastically, whether or not it was strictly test driven. Merely considering whether it will be easy to reliably test the code improves it, IME.

gvdenbro 9y ago

you could test your tests using mutation testing [0]. it's not bulletproof but quite close. [0] https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing