4 ms·

If you find yourself completely unable to articulate a test for something, you probably don't really know what it is you're trying to build. I don’t buy this a

If you find yourself completely unable to articulate a test for something, you probably don't really know what it is you're trying to build. I don’t buy this argument. How would you write tests to drive the development of a graphics demo, say rendering a Mandelbrot set? Or a tool to convert audio data from one format to another? Or any other kind of software where the output doesn’t consist of readily verifiable, discrete data points?

couchand 13y ago
Are you asking about unit tests or acceptance tests? The problems you describe are very high level, but we could design an acceptance testing scheme for them. For the Mandelbrot set it might involve comparison to a reference rendering, for the audio tool a reference recording. In both cases you'd allow a delta relevant to the application, and probably also benchmark for acceptable performance. But my point was more aimed at unit testing. When you set out to write a function you should know something about that function before starting. If you know enough to write the function signature, you can first write a failing test. If you can write a bit of code in that function, you can write a bit expecting the behavior of that code.
- Chris_Newton 13y ago
  Are you asking about unit tests or acceptance tests? I suppose what I’m really asking is how you would go from not having software to having software that does those things, using TDD. I think in practice its fail-pass-refactor cycle is normally applied at the level of unit tests, but in any case, how would using TDD help to drive a good design, to ensure testability, or otherwise, in that kind of situation? (I’m asking this rhetorically. I don’t think TDD is a very helpful process in this context. I’m just trying to demonstrate this with practical examples rather than bluntly stating it without any supporting argument.)
  - colomon 13y ago
    I think I mostly agree with your larger point, but I'm not in love with your examples. The Mandelbrot set does consist of readily verifiable discrete data points, after all. I don't have any problem imagining myself developing a Mandelbrot set program using TDD. A great example for your point (which might have been what you were getting at with the audio thing) is a test for creating files in a lossy audio format. The acid test is if it sounds right to a human being with good ears, I've got no clue how you would write a pure computer test for that. In my own work, a great example is finding a NURBS approximation of the intersection of two surfaces. There are an infinite number of correct answers for a given pair of surfaces, and testing that the curve you've generated fits the surfaces is a distressingly hard problem.
    - Chris_Newton 13y ago
      The Mandelbrot set does consist of readily verifiable discrete data points, after all. Indeed it does, but in order to verify them I see only two options. One is that you have to choose test cases where the answer is trivially determined. However, with this strategy, it seems you must ultimately rely on the refactoring step to magically convert your implementation to support the general case, so the hard part isn’t really test-driven at all. The other is that you verify non-trivial results. However, to do so you must inevitably reimplement the relevant mathematics one way or another to support your tests. Of course, if you could do that reliably, then you wouldn’t need the tests in the first place. This isn’t to say that writing multiple independent implementations in parallel is a bad thing for testing purposes. If you do that and then run a statistical test comparing their outputs for a large sample of inputs, a perfect match will provide some confidence that you did at least implement the same thing each time, so it is likely that all versions correctly implement the spec. (Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.) However, again for a simple example like rendering a fractal, you could do that by reimplementing the entire algorithm as a whole, without any of the overheads that TDD might impose. I don't have any problem imagining myself developing a Mandelbrot set program using TDD. I’m genuinely curious about how you’d see that going.
      - colomon 13y ago

couchand 13y ago

Are you asking about unit tests or acceptance tests? The problems you describe are very high level, but we could design an acceptance testing scheme for them. For the Mandelbrot set it might involve comparison to a reference rendering, for the audio tool a reference recording. In both cases you'd allow a delta relevant to the application, and probably also benchmark for acceptable performance. But my point was more aimed at unit testing. When you set out to write a function you should know something about that function before starting. If you know enough to write the function signature, you can first write a failing test. If you can write a bit of code in that function, you can write a bit expecting the behavior of that code.

Chris_Newton 13y ago
Are you asking about unit tests or acceptance tests? I suppose what I’m really asking is how you would go from not having software to having software that does those things, using TDD. I think in practice its fail-pass-refactor cycle is normally applied at the level of unit tests, but in any case, how would using TDD help to drive a good design, to ensure testability, or otherwise, in that kind of situation? (I’m asking this rhetorically. I don’t think TDD is a very helpful process in this context. I’m just trying to demonstrate this with practical examples rather than bluntly stating it without any supporting argument.)
- colomon 13y ago
  I think I mostly agree with your larger point, but I'm not in love with your examples. The Mandelbrot set does consist of readily verifiable discrete data points, after all. I don't have any problem imagining myself developing a Mandelbrot set program using TDD. A great example for your point (which might have been what you were getting at with the audio thing) is a test for creating files in a lossy audio format. The acid test is if it sounds right to a human being with good ears, I've got no clue how you would write a pure computer test for that. In my own work, a great example is finding a NURBS approximation of the intersection of two surfaces. There are an infinite number of correct answers for a given pair of surfaces, and testing that the curve you've generated fits the surfaces is a distressingly hard problem.
  - Chris_Newton 13y ago
    The Mandelbrot set does consist of readily verifiable discrete data points, after all. Indeed it does, but in order to verify them I see only two options. One is that you have to choose test cases where the answer is trivially determined. However, with this strategy, it seems you must ultimately rely on the refactoring step to magically convert your implementation to support the general case, so the hard part isn’t really test-driven at all. The other is that you verify non-trivial results. However, to do so you must inevitably reimplement the relevant mathematics one way or another to support your tests. Of course, if you could do that reliably, then you wouldn’t need the tests in the first place. This isn’t to say that writing multiple independent implementations in parallel is a bad thing for testing purposes. If you do that and then run a statistical test comparing their outputs for a large sample of inputs, a perfect match will provide some confidence that you did at least implement the same thing each time, so it is likely that all versions correctly implement the spec. (Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.) However, again for a simple example like rendering a fractal, you could do that by reimplementing the entire algorithm as a whole, without any of the overheads that TDD might impose. I don't have any problem imagining myself developing a Mandelbrot set program using TDD. I’m genuinely curious about how you’d see that going.
    - colomon 13y ago