5 ms·

> where you already know the expected output for given input and can easily specify that output using assertions in code Well, "knowing" (or manually generatin

> where you already know the expected output for given input and can easily specify that output using assertions in code Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. If you execute your code and copypaste its output blindly, you risk testing the wrong thing, don't you. And if you can't specify your code's output, how do you test? BTW, I agree with the fact that TDD is overly pedantic at times.

Chris_Newton 5y ago
Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. Not all code is written knowing the answer in advance. Sometimes you’re implementing a mathematical model to predict next week’s weather, or the effects of a change in economic policy, or the performance of different hull shapes for a ship under different conditions at sea. My second standard challenge to TDD evangelists is to write a program to draw a classic multicoloured Mandelbrot image, showing an arbitrary rectangular region of the complex plane, using only the mathematical specification without any reference to existing code or output data. This can be done with a short program that is simple enough to verify by eye that it matches the required mathematical specification. However, the problem has an infinite input space, its fractal nature means predicting output values is impossible in the most interesting parts of that input space, and you have no idea where those interesting parts are anyway.
- iudqnolq 5y ago
  What's the issue with using the formula and a calculator to compute sample points?
  - Chris_Newton 5y ago
    What you are advocating is duplicating the calculations the program does manually for some set of specific cases, which can then be used for tests. Let’s consider a few practical questions about how that might work. 1. How do you choose which specific cases to calculate manually? Remember, you have no prior knowledge of what the correct answer looks like or where the interesting parts of the input space are. You have no way to determine whether any given set of manual calculations is representative of the work your program will be doing or covers the areas with the greatest risk of error. 2. How practical is it to make all those manual calculations? Even in this very simple case, calculating the correct answer for a single input point manually might require hundreds of complex arithmetic operations to be performed. That’s going to be slow and error-prone. After all, isn’t that why we’re writing a program to do this for us? Now, how does this idea scale? What if our program isn’t computing a nice analytical solution to a simple arithmetic problem, but instead running a complicated numerical method to process many thousands of data points in each input? It quickly becomes impractical to rely on this strategy for testing. 3. How will having a set of known outputs you can test against drive an implementation from scratch? I mentioned before that Mandebrot is my second standard challenge to TDD evangelists. The first is to write add(x,y) driven by tests. After a bit of back and forth, this invariably ends up with an implementation that was generalised from however many specific cases were given to the general case. Invariably, that generalisation is the step that actually creates a useful solution to the original problem, and invariably it uses an insight that was not driven by the tests. Our Mandelbrot scenario is the same situation, just a slightly more complicated example. No matter how many individual tests you create by choosing sample points in the input space and manually calculating the expected output, you won’t have a systematic way to work back from those answers to derive a correct general implementation of the Mandelbrot calculation. Your specific cases might be useful for verifying an existing implementation, but they give you no insight into how to write a good implementation from scratch. (If I’m dealing with a particularly strident advocate of TDD, this is the point where I mention the word “sudoku”.) And again, we have to ask how this process scales. What if we had a more challenging problem, say extracting an audio track from a video file and running a speech recognition process on it to generate subtitles? It might actually be easier in that scenario to identify individual test cases: just take a known video file as input and write down the expected output for it, and now you have an end-to-end test. However, it’s still true that no amount of end-to-end tests will necessarily offer any insight into how to structure a good implementation of the required functionality in detail, nor will it tell us how to implement any specific part or generate useful unit tests cases for those parts. End-to-end test cases might help us to verify an existing implementation, but in general they won’t reliably drive a correct one from scratch.
    - snovv_crash 5y ago
      It gives you something that can be verified for when someone does some optimization, or ports to a different platform. It isn't for today, it is for tomorrow.
      - Chris_Newton 5y ago
        I agree that having an automated test suite might be helpful in those situations, but TDD comes with a lot more baggage.
        snovv_crash 5y ago
        For sure, it's a tool that some people take as a dogma. But I can say definitively that untested codebases beyond toy-scale are harder to work with, and for all its flaws TDD makes sure that doesn't happen.
    - iudqnolq 5y ago
      You're right that if you play the sort of game where you write the test "assertEq(2, add(1, 1))" and then write "fun add(_, _) = 2" you won't get very far with this sort of problem. I personally (without much experience) prefer the kind of TDD where you write say "assertEq(0, add(0, 0)), assertEq(43, add(42, 1))", fully implement the add function, and then move on. In the mandlebrot case I'd maybe compute what a few pixels should be, write all the code, and see if those pixels are right. Not perfect, but I find it better than nothing.
      - Chris_Newton 5y ago
        The point of the example is that the fundamental premise of TDD is flawed here. You can’t reach a program that solves the general case by iteratively writing a failing test for a specific case, making the smallest change required to make that test pass, refactoring, and then moving on to the next test. The important step — what you called “fully implementing the add function” — is taken by implementing some other insight that is not driven by the tests. That step isn’t a refactoring either, because it very much does change the behaviour of the program. To design and implement a good program, sometimes you just have to know what you’re doing. There is no substitute for understanding the problem you are trying to solve and how you intend to solve it. And if you have that understanding and you design and implement your program accordingly, what is the value of following the red-green-refactor process compared to simply writing any unit tests you find helpful for verifying your implementation?
- sterlind 5y ago
  that actually sounds like a fun thing to do in a formally verified language like Dafny. you'd write a "ghost function" for the Mandelbrot equation, then write your code, then prove a theorem that the epsilon between the equation and the code was below a threshold at every pixel in your rectangle. I guess that'd still be TDD, but with the first T standing for Theorem :p
- snovv_crash 5y ago
  Yes, of course there are cases where it doesn't work. But what percentage of code is that, really?
  - Chris_Newton 5y ago
    I don’t know what percentage of all code that gets written would be better tested in other ways. I doubt anyone else does either. For me, unit testing (in the typical xunit style associated with TDD) is most useful for basic data processing code with predictable outputs. That might include a wide range of code, from little utility functions on strings to the business rules for whole CRUD applications. On the other hand, anything with input or output data in an awkward format, anything communicating with any external equipment or remote API, anything involving nondeterminism or heuristics or where the purpose of the code is to perform some calculation where you don’t know the correct answer in advance, these kinds of code don’t tend to fit well with a TDD approach and that style of unit testing as the main test strategy IMHO. Those cover a pretty wide range of code as well.
    - snovv_crash 5y ago
      Then it's really a matter of the granularity of the tests, right? Maybe they should target a higher level of abstraction, where the nasty details (which can hopefully be simplified in the future) aren't an issue. If you don't know about what the code is meant to do (non-determinism, heuristics, etc), to me that you're targeting the wrong level of abstraction. At some level you know what it's meant to do, unless you're working on an abstract art project. TDD done dogmatically is a mess, sure, but then so is anything.
      - Chris_Newton 5y ago
        I agree that it may be more useful to test a larger part of the code together instead of trying to do everything at the finest levels of detail. However, my point here is about more than just the level(s) where you perform the tests. It’s also about what kinds of testing you are doing. Here are some of the most common possibilities: • xUnit-style tests for specific cases • Property-based testing • Snapshot-based testing • Manual testing • Formal verification • Peer code review All of these can be useful under the right circumstances. If your code produces output that is best checked by human inspection but shouldn’t then change (or change very much) then snapshots may be a good choice. If your code won’t deterministically produce the same correct output every run but whatever output it does produce should always satisfy certain conditions, maybe property-based testing is a good way to go. If your code involves communication with external equipment that requires operator interaction to do anything interesting, fully automated testing might simply not be possible. In that case, manual integration testing with someone physically operating the equipment might be appropriate. Formal verification covers a wide range of possibilities from the likes of basic static type checking all the way up to the use of automated theorem provers in specialised programming languages. It almost always has some extra cost in terms of annotating the code but it can sometimes produce far more powerful evidence of correctness in the general case than any test suite checking individual cases ever could. Code reviews are pretty much universally good, as long as you’ve got enough people available with the relevant knowledge to do them. Sometimes, these testing techniques are complementary and using more than one of them together might be beneficial. At other times, a coding style or process that favours one might make another more difficult. This HN discussion is mainly about the style of “testable” code that TDD tends to produce, with many small units and lots of dependency injection, which is of course very friendly to small-scale unit testing. However, it might also be more difficult to review because of all the configurability and indirection. If it relies on doubles to stand in for external resources, it can end up testing the accuracy of the simulation more than anything else, so adding very little (justified) confidence that the real system is operating correctly. And as a rule of thumb, individual testing of specific cases may be less effective than testing large numbers of generated cases with property testing, which in turn may be less effective than proving that all cases work via rigorous analysis. And this brings me back to where I came in, which was that prioritising “testable” code in the TDD sense isn’t necessarily a good thing. The real meaning of “testable” depends greatly on what type of code you’re writing and which testing strategies are most helpful.

Chris_Newton 5y ago

Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. Not all code is written knowing the answer in advance. Sometimes you’re implementing a mathematical model to predict next week’s weather, or the effects of a change in economic policy, or the performance of different hull shapes for a ship under different conditions at sea. My second standard challenge to TDD evangelists is to write a program to draw a classic multicoloured Mandelbrot image, showing an arbitrary rectangular region of the complex plane, using only the mathematical specification without any reference to existing code or output data. This can be done with a short program that is simple enough to verify by eye that it matches the required mathematical specification. However, the problem has an infinite input space, its fractal nature means predicting output values is impossible in the most interesting parts of that input space, and you have no idea where those interesting parts are anyway.

iudqnolq 5y ago
What's the issue with using the formula and a calculator to compute sample points?
- Chris_Newton 5y ago
  What you are advocating is duplicating the calculations the program does manually for some set of specific cases, which can then be used for tests. Let’s consider a few practical questions about how that might work. 1. How do you choose which specific cases to calculate manually? Remember, you have no prior knowledge of what the correct answer looks like or where the interesting parts of the input space are. You have no way to determine whether any given set of manual calculations is representative of the work your program will be doing or covers the areas with the greatest risk of error. 2. How practical is it to make all those manual calculations? Even in this very simple case, calculating the correct answer for a single input point manually might require hundreds of complex arithmetic operations to be performed. That’s going to be slow and error-prone. After all, isn’t that why we’re writing a program to do this for us? Now, how does this idea scale? What if our program isn’t computing a nice analytical solution to a simple arithmetic problem, but instead running a complicated numerical method to process many thousands of data points in each input? It quickly becomes impractical to rely on this strategy for testing. 3. How will having a set of known outputs you can test against drive an implementation from scratch? I mentioned before that Mandebrot is my second standard challenge to TDD evangelists. The first is to write add(x,y) driven by tests. After a bit of back and forth, this invariably ends up with an implementation that was generalised from however many specific cases were given to the general case. Invariably, that generalisation is the step that actually creates a useful solution to the original problem, and invariably it uses an insight that was not driven by the tests. Our Mandelbrot scenario is the same situation, just a slightly more complicated example. No matter how many individual tests you create by choosing sample points in the input space and manually calculating the expected output, you won’t have a systematic way to work back from those answers to derive a correct general implementation of the Mandelbrot calculation. Your specific cases might be useful for verifying an existing implementation, but they give you no insight into how to write a good implementation from scratch. (If I’m dealing with a particularly strident advocate of TDD, this is the point where I mention the word “sudoku”.) And again, we have to ask how this process scales. What if we had a more challenging problem, say extracting an audio track from a video file and running a speech recognition process on it to generate subtitles? It might actually be easier in that scenario to identify individual test cases: just take a known video file as input and write down the expected output for it, and now you have an end-to-end test. However, it’s still true that no amount of end-to-end tests will necessarily offer any insight into how to structure a good implementation of the required functionality in detail, nor will it tell us how to implement any specific part or generate useful unit tests cases for those parts. End-to-end test cases might help us to verify an existing implementation, but in general they won’t reliably drive a correct one from scratch.
  - snovv_crash 5y ago
    It gives you something that can be verified for when someone does some optimization, or ports to a different platform. It isn't for today, it is for tomorrow.
    - Chris_Newton 5y ago
      I agree that having an automated test suite might be helpful in those situations, but TDD comes with a lot more baggage.
      - snovv_crash 5y ago
        For sure, it's a tool that some people take as a dogma. But I can say definitively that untested codebases beyond toy-scale are harder to work with, and for all its flaws TDD makes sure that doesn't happen.
  - iudqnolq 5y ago
    You're right that if you play the sort of game where you write the test "assertEq(2, add(1, 1))" and then write "fun add(_, _) = 2" you won't get very far with this sort of problem. I personally (without much experience) prefer the kind of TDD where you write say "assertEq(0, add(0, 0)), assertEq(43, add(42, 1))", fully implement the add function, and then move on. In the mandlebrot case I'd maybe compute what a few pixels should be, write all the code, and see if those pixels are right. Not perfect, but I find it better than nothing.
    - Chris_Newton 5y ago
      The point of the example is that the fundamental premise of TDD is flawed here. You can’t reach a program that solves the general case by iteratively writing a failing test for a specific case, making the smallest change required to make that test pass, refactoring, and then moving on to the next test. The important step — what you called “fully implementing the add function” — is taken by implementing some other insight that is not driven by the tests. That step isn’t a refactoring either, because it very much does change the behaviour of the program. To design and implement a good program, sometimes you just have to know what you’re doing. There is no substitute for understanding the problem you are trying to solve and how you intend to solve it. And if you have that understanding and you design and implement your program accordingly, what is the value of following the red-green-refactor process compared to simply writing any unit tests you find helpful for verifying your implementation?
sterlind 5y ago
that actually sounds like a fun thing to do in a formally verified language like Dafny. you'd write a "ghost function" for the Mandelbrot equation, then write your code, then prove a theorem that the epsilon between the equation and the code was below a threshold at every pixel in your rectangle. I guess that'd still be TDD, but with the first T standing for Theorem :p
snovv_crash 5y ago
Yes, of course there are cases where it doesn't work. But what percentage of code is that, really?
- Chris_Newton 5y ago
  I don’t know what percentage of all code that gets written would be better tested in other ways. I doubt anyone else does either. For me, unit testing (in the typical xunit style associated with TDD) is most useful for basic data processing code with predictable outputs. That might include a wide range of code, from little utility functions on strings to the business rules for whole CRUD applications. On the other hand, anything with input or output data in an awkward format, anything communicating with any external equipment or remote API, anything involving nondeterminism or heuristics or where the purpose of the code is to perform some calculation where you don’t know the correct answer in advance, these kinds of code don’t tend to fit well with a TDD approach and that style of unit testing as the main test strategy IMHO. Those cover a pretty wide range of code as well.
  - snovv_crash 5y ago
    Then it's really a matter of the granularity of the tests, right? Maybe they should target a higher level of abstraction, where the nasty details (which can hopefully be simplified in the future) aren't an issue. If you don't know about what the code is meant to do (non-determinism, heuristics, etc), to me that you're targeting the wrong level of abstraction. At some level you know what it's meant to do, unless you're working on an abstract art project. TDD done dogmatically is a mess, sure, but then so is anything.
    - Chris_Newton 5y ago
      I agree that it may be more useful to test a larger part of the code together instead of trying to do everything at the finest levels of detail. However, my point here is about more than just the level(s) where you perform the tests. It’s also about what kinds of testing you are doing. Here are some of the most common possibilities: • xUnit-style tests for specific cases • Property-based testing • Snapshot-based testing • Manual testing • Formal verification • Peer code review All of these can be useful under the right circumstances. If your code produces output that is best checked by human inspection but shouldn’t then change (or change very much) then snapshots may be a good choice. If your code won’t deterministically produce the same correct output every run but whatever output it does produce should always satisfy certain conditions, maybe property-based testing is a good way to go. If your code involves communication with external equipment that requires operator interaction to do anything interesting, fully automated testing might simply not be possible. In that case, manual integration testing with someone physically operating the equipment might be appropriate. Formal verification covers a wide range of possibilities from the likes of basic static type checking all the way up to the use of automated theorem provers in specialised programming languages. It almost always has some extra cost in terms of annotating the code but it can sometimes produce far more powerful evidence of correctness in the general case than any test suite checking individual cases ever could. Code reviews are pretty much universally good, as long as you’ve got enough people available with the relevant knowledge to do them. Sometimes, these testing techniques are complementary and using more than one of them together might be beneficial. At other times, a coding style or process that favours one might make another more difficult. This HN discussion is mainly about the style of “testable” code that TDD tends to produce, with many small units and lots of dependency injection, which is of course very friendly to small-scale unit testing. However, it might also be more difficult to review because of all the configurability and indirection. If it relies on doubles to stand in for external resources, it can end up testing the accuracy of the simulation more than anything else, so adding very little (justified) confidence that the real system is operating correctly. And as a rule of thumb, individual testing of specific cases may be less effective than testing large numbers of generated cases with property testing, which in turn may be less effective than proving that all cases work via rigorous analysis. And this brings me back to where I came in, which was that prioritising “testable” code in the TDD sense isn’t necessarily a good thing. The real meaning of “testable” depends greatly on what type of code you’re writing and which testing strategies are most helpful.