7 ms·

The real value comes from thorough, automated testing suites, whatever their origin is. Automated testing is useful, for sure. However, TDD purists tend to foc

The real value comes from thorough, automated testing suites, whatever their origin is. Automated testing is useful, for sure. However, TDD purists tend to focus on one very specific type of testing: automated unit testing, in the small, where you already know the expected output for given input and can easily specify that output using assertions in code. By its nature, TDD emphasises being testable in that specific sense above all else. I don’t think that is necessarily a good thing, partly because that type of universal unit testing may not be a good strategy for every software system, and partly because other useful properties of the code might be diminished because of the changes needed to make it “testable” in the TDD sense.

alanfranz 5y ago
> where you already know the expected output for given input and can easily specify that output using assertions in code Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. If you execute your code and copypaste its output blindly, you risk testing the wrong thing, don't you. And if you can't specify your code's output, how do you test? BTW, I agree with the fact that TDD is overly pedantic at times.
- Chris_Newton 5y ago
  Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. Not all code is written knowing the answer in advance. Sometimes you’re implementing a mathematical model to predict next week’s weather, or the effects of a change in economic policy, or the performance of different hull shapes for a ship under different conditions at sea. My second standard challenge to TDD evangelists is to write a program to draw a classic multicoloured Mandelbrot image, showing an arbitrary rectangular region of the complex plane, using only the mathematical specification without any reference to existing code or output data. This can be done with a short program that is simple enough to verify by eye that it matches the required mathematical specification. However, the problem has an infinite input space, its fractal nature means predicting output values is impossible in the most interesting parts of that input space, and you have no idea where those interesting parts are anyway.
  - iudqnolq 5y ago
    What's the issue with using the formula and a calculator to compute sample points?
    - Chris_Newton 5y ago
      What you are advocating is duplicating the calculations the program does manually for some set of specific cases, which can then be used for tests. Let’s consider a few practical questions about how that might work. 1. How do you choose which specific cases to calculate manually? Remember, you have no prior knowledge of what the correct answer looks like or where the interesting parts of the input space are. You have no way to determine whether any given set of manual calculations is representative of the work your program will be doing or covers the areas with the greatest risk of error. 2. How practical is it to make all those manual calculations? Even in this very simple case, calculating the correct answer for a single input point manually might require hundreds of complex arithmetic operations to be performed. That’s going to be slow and error-prone. After all, isn’t that why we’re writing a program to do this for us? Now, how does this idea scale? What if our program isn’t computing a nice analytical solution to a simple arithmetic problem, but instead running a complicated numerical method to process many thousands of data points in each input? It quickly becomes impractical to rely on this strategy for testing. 3. How will having a set of known outputs you can test against drive an implementation from scratch? I mentioned before that Mandebrot is my second standard challenge to TDD evangelists. The first is to write add(x,y) driven by tests. After a bit of back and forth, this invariably ends up with an implementation that was generalised from however many specific cases were given to the general case. Invariably, that generalisation is the step that actually creates a useful solution to the original problem, and invariably it uses an insight that was not driven by the tests. Our Mandelbrot scenario is the same situation, just a slightly more complicated example. No matter how many individual tests you create by choosing sample points in the input space and manually calculating the expected output, you won’t have a systematic way to work back from those answers to derive a correct general implementation of the Mandelbrot calculation. Your specific cases might be useful for verifying an existing implementation, but they give you no insight into how to write a good implementation from scratch. (If I’m dealing with a particularly strident advocate of TDD, this is the point where I mention the word “sudoku”.) And again, we have to ask how this process scales. What if we had a more challenging problem, say extracting an audio track from a video file and running a speech recognition process on it to generate subtitles? It might actually be easier in that scenario to identify individual test cases: just take a known video file as input and write down the expected output for it, and now you have an end-to-end test. However, it’s still true that no amount of end-to-end tests will necessarily offer any insight into how to structure a good implementation of the required functionality in detail, nor will it tell us how to implement any specific part or generate useful unit tests cases for those parts. End-to-end test cases might help us to verify an existing implementation, but in general they won’t reliably drive a correct one from scratch.

alanfranz 5y ago

> where you already know the expected output for given input and can easily specify that output using assertions in code Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. If you execute your code and copypaste its output blindly, you risk testing the wrong thing, don't you. And if you can't specify your code's output, how do you test? BTW, I agree with the fact that TDD is overly pedantic at times.

Chris_Newton 5y ago
Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. Not all code is written knowing the answer in advance. Sometimes you’re implementing a mathematical model to predict next week’s weather, or the effects of a change in economic policy, or the performance of different hull shapes for a ship under different conditions at sea. My second standard challenge to TDD evangelists is to write a program to draw a classic multicoloured Mandelbrot image, showing an arbitrary rectangular region of the complex plane, using only the mathematical specification without any reference to existing code or output data. This can be done with a short program that is simple enough to verify by eye that it matches the required mathematical specification. However, the problem has an infinite input space, its fractal nature means predicting output values is impossible in the most interesting parts of that input space, and you have no idea where those interesting parts are anyway.
- iudqnolq 5y ago
  What's the issue with using the formula and a calculator to compute sample points?
  - Chris_Newton 5y ago
    What you are advocating is duplicating the calculations the program does manually for some set of specific cases, which can then be used for tests. Let’s consider a few practical questions about how that might work. 1. How do you choose which specific cases to calculate manually? Remember, you have no prior knowledge of what the correct answer looks like or where the interesting parts of the input space are. You have no way to determine whether any given set of manual calculations is representative of the work your program will be doing or covers the areas with the greatest risk of error. 2. How practical is it to make all those manual calculations? Even in this very simple case, calculating the correct answer for a single input point manually might require hundreds of complex arithmetic operations to be performed. That’s going to be slow and error-prone. After all, isn’t that why we’re writing a program to do this for us? Now, how does this idea scale? What if our program isn’t computing a nice analytical solution to a simple arithmetic problem, but instead running a complicated numerical method to process many thousands of data points in each input? It quickly becomes impractical to rely on this strategy for testing. 3. How will having a set of known outputs you can test against drive an implementation from scratch? I mentioned before that Mandebrot is my second standard challenge to TDD evangelists. The first is to write add(x,y) driven by tests. After a bit of back and forth, this invariably ends up with an implementation that was generalised from however many specific cases were given to the general case. Invariably, that generalisation is the step that actually creates a useful solution to the original problem, and invariably it uses an insight that was not driven by the tests. Our Mandelbrot scenario is the same situation, just a slightly more complicated example. No matter how many individual tests you create by choosing sample points in the input space and manually calculating the expected output, you won’t have a systematic way to work back from those answers to derive a correct general implementation of the Mandelbrot calculation. Your specific cases might be useful for verifying an existing implementation, but they give you no insight into how to write a good implementation from scratch. (If I’m dealing with a particularly strident advocate of TDD, this is the point where I mention the word “sudoku”.) And again, we have to ask how this process scales. What if we had a more challenging problem, say extracting an audio track from a video file and running a speech recognition process on it to generate subtitles? It might actually be easier in that scenario to identify individual test cases: just take a known video file as input and write down the expected output for it, and now you have an end-to-end test. However, it’s still true that no amount of end-to-end tests will necessarily offer any insight into how to structure a good implementation of the required functionality in detail, nor will it tell us how to implement any specific part or generate useful unit tests cases for those parts. End-to-end test cases might help us to verify an existing implementation, but in general they won’t reliably drive a correct one from scratch.