5 ms·

> Bugs are correlated with lines of code and TDD forces writing more code so how can it reduce bug counts? If the test code has no bugs then just write the rest

> Bugs are correlated with lines of code and TDD forces writing more code so how can it reduce bug counts? If the test code has no bugs then just write the rest of the code in the same style Nobody has pointed out the fault in this reasoning yet, so I will. The linear relationship lines-of-code vs. total-bug-count is based on independence of bug introduction in different parts of the code. That is, introducing a bug in line 10000 is more-or-less independent of adding line 20000. For product code that is arguably the case, but adding test code into the mix, this basic assumption doesn't just not hold, but it's turned upside down. Test code is not customer-visible, but only dev-visible, with the sole purpose of finding bugs. Thus, adding test code to a code base decreases the average probability of a bug for lines of product code. More formally, if you have N lines of product code and thus c * N bugs for some fraction c, then adding M lines of test code does not increase the number of customer-visible bugs to c * (N + M). Instead, it reduces that number to c' * N with c'<c and the difference being caused by your test coverage. (A 100% test coverage, i.e., an exhaustive test without bugs or, equivalently, a formal verification, would bring c' to 0.) Sure, the M lines of test code may well have bugs on their own, but that only increases c' slightly while keeping it below c, and more importantly, those test bugs are not customer-visible. They only annoy developers. I agree with the rest of the post though.

sombremesa 8y ago
Tests don't exist for solving bugs as much as they exist to prevent regression. Writing exhaustive tests is not sufficient to bring c' to zero, because some of those bugs exist because of the developer's own presumptions about how code should behave.
- chii 8y ago
  But testing exhaustively means the full range of inputs are used, and therefore, if a bug arises as inputs not considered by the dev initially, it would show up as a failure. What you're actually saying is that reaching full exhaustive testing is near impossible.
  - icebraining 8y ago
    I think what they're saying is that both the code and test will contain the bug, since the bug is in the programmer's understanding rather than just in their coding. If I understood func(5, 6) should return 9, but the client actually wants 10, no amount of tests I write will reveal the error.
    - neo2006 8y ago
      I don't think that any form of formal specification will solve that issue. The only way to minimize the impact of it is to work on small iterations (agile) and be close to the client. One of the weakness in agile methodologies today is that the client representation (Product owner) is not aware of the client needs or too far from the client point of view.
      - sombremesa 8y ago
        This is roughly what I meant - there are ways to end up with zero bugs, but writing tests is usually insufficient for achieving such a goal.
  - mikekchar 8y ago
    Testing for regressions is a a valuable thing to do, but your point is also valid. We can prove that it is impossible to test every scenario in general (for example, does this program end ;-) ). In my mind, unit tests exist for documenting what the code is doing now -- not to determine if it is correct (acceptance tests are different). Your goal for a unit test is to have it fail when the behaviour of the production code fails. This way you can verify that the changes you are making are having the effect you expected. Similarly, you can be warned when changes you make affect part of the code you didn't expect to be affected. Of course, you will never get full coverage, but tests are a programmer's tool. It allows you to reduce the amount of time you spend inspecting parts of the code you aren't working on. For example, when I'm writing code with good test coverage, I can find out what happens if I change an attribute in a structure simply by doing it and watching the tests fail. Each failure tells me where in the code I'm likely to have to make a change. If you make a small viewpoint change that tests are not to find bugs, but rather are a tool to help you understand the code base, they make a lot more sense. (IMHO)
gilbetron 8y ago
Do you have an actual source that corroborates that with actual data? It seems true, but I haven't found anything that actually tests the premise. I've found studies that show that code with lots of unit tests tends to be more bug free, but not anything about whether unit tests increases (or decreases) development time or whether test code actually has less bugs. I think it would be interesting if there is also a measure of code complexity, and if less complex test code correlates with less bugs in the test code.
- Aeolun 8y ago
  Arguing from the point of logic and formalisms the author is talking about, this would seem to be true.
  - gilbetron 8y ago
    > Instead, it reduces that number to c' * N with c'<c and the difference being caused by your test coverage. That is unproven conjecture. It feels right, but just because it is written as a formula doesn't automatically mean it is correct.
    - peteradio 8y ago
      It is an axiom. If your testing is such that this isn't true then you are doing testing wrong, in a self-evident sort of way.
      - sidlls 8y ago
        That's about as useful as observing that if your code is such that it doesn't produce the correct result then you've written the code wrong, in a self-evident sort of way.
        peteradio 8y ago
        > observing that if your code is such that it doesn't produce the correct result Hmm. Sounds like you are doing some sort of testing.
paulddraper 8y ago
Yeah, it's like saying "CAP says partitions decreases availability/consistency, so putting your app servers in a scale group will make things less available/consistent." That's really superficial analysis. Yes there is some fundamental tradeoffs. But it is possible to change the properties of a system or combine them in intelligent ways and move the curve. Or it's like saying that speedometers make your GPS less accurate due the Heisenberg principal.
p1necone 8y ago
The other fault in that reasoning is that usually test code is much much simpler than product code. It's a lot harder to write bugs when the structure of the code is just 1. call single function with some input data 2. verify that output data/side effects meet expectations 3. repeat for more sets of inputs.
- neo2006 8y ago
  One of the mistake I see a lot around this is when people start to write intelligent mocks. Usually this is where test bugs reside.
  - p1necone 8y ago
    The best is when someone writes an entirely new implementation of something just for tests, and only calls that from their tests instead of the actual product code! Then you just get blank stares when you try to explain why what they've just done is completely pointless.
    - loup-vaillant 8y ago
      The way they do it may be pointless, but the intent is not. The crux is making sure your two implementation are independent. There are two ways to achieve this. The first is clean room. Just have independent persons or teams implement the same thing. With different tools and programming languages if you can. The second is use an entirely different approach for both. The test code for instance can be constructed around the tests, and for instance rely on the fact that everything happens at the same place at the same time (when you run the test). --- I'm currently writing on a secure handshakes generator. https://github.com/LoupVaillant/Monokex https://github.com/LoupVaillant/Monokex It generates specifications and source code from a Noise pattern. A bit like Noise Explorer, only with less features. Previous versions of the code were hand written, and the generated code is doing its best to look like it was hand written. Here's the result: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/monokex.c https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... Now how do I ensure the code match the specs? (The correctness of the specs itself is currently checked by hand.) I could write another implementation, but I'm only me, and I can't go erase my own memory to make a clean room implementation and compare the two. Instead, I took the specs, and wrote code that takes all the inputs, and spits out all the intermediate buffers and outputs, paying no heed to stuff like order of execution, or who does what. I only concentrated on generating the test vectors: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/vectors.c#L81 https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... The structure of the vector generating code and the actual production code are very different. This is how I decorrelate mistakes, and make sure that if the two "implementations" agree, I'm very likely to have something that works. --- Of course, this is all a lot of effort, so I wouldn't do this for non-critical code.
foobarchu 8y ago
I think this holds for well written tests, but if one is not talented in writing them then more tests will simply bolster the bugs in the main code. For example, if a developer generate test data by running the program and copying it's output (something I see far more often than I'm comfortable with), they can tell themselves that section was tested thoroughly and must be bug free. Thus when someones notices an end result someplace is wrong, the first place checked may not be that "well tested" function. Tests are only as good as the domain knowledge of the person writing them.
mikekchar 8y ago
Bugs in tests exist (and are just as prevalent as bugs in production code in my experience). However, writing production code introduces a kind of constraint on the system. If you run a function with certain program state, then it returns a certain result. No matter what you do, this will happen because that's how you wrote the code. A test introduces the same constraint. Instead of implementing the function, it runs the function. No matter what you do, the test will run certain code and get a certain result. Tests pass when the constraints introduced by the production code match the constraints introduced by the tests. Because of this, if there is a software error in the production code, usually you need to have a corresponding error in the tests in order for the test to pass. This definitely happens from time to time, but the odds of it happening are much lower. Similarly, if you have a software error in the tests, you need to have a corresponding software error in the production code in order for the test to pass -- assuming that the test is actually exercising the code (something you can't always assume, unfortunately).
ericathegreat 8y ago
Agree. Consider the situation of data entry professionals, those people who transcribe audio recordings or enter data into databases. If you have one person entering data into a computer, then the odds of them introducing an error and failing to spot it are fairly high. If you have twice as many people entering twice as much data data, then the odds of an error getting introduced are roughly doubled. However, if you have those two people entering the same data, then their mistakes cancel each other out. If person A and person B both entered the same thing, it's extremely unlikely that it's incorrect. If they differ though, the a problem has been identified, and can now be fixed. The odds of both of those people entering the same piece of data incorrectly is tiny. Likewise, accidentally introducing a bug into both the production code and the test is pretty unlikely. That said though, if those two theoretical data entry people above are given the wrong data to enter, then they the system cannot protect them. They will both correctly enter incorrect data. "Garbage in, garbage out". Likewise, if the requirements of a piece of software are poorly understood, then it is quite likely that both the test and the production code will implement the same "bug". Writing tests won't fix a failure to understand the problem you're trying to solve. And they're not supposed to.
- gdfasfklshg4 8y ago
  > The odds of both of those people entering the same piece of data incorrectly is tiny. This is just not correct. There may be a systematic reason why they are making a mistake (e.g. a miss-pronounced word) in which case increasing the confidence intervals does not increase the accuracy. Check out the concepts of accuracy, precision etc from physical sciences.

sombremesa 8y ago

Tests don't exist for solving bugs as much as they exist to prevent regression. Writing exhaustive tests is not sufficient to bring c' to zero, because some of those bugs exist because of the developer's own presumptions about how code should behave.

chii 8y ago
But testing exhaustively means the full range of inputs are used, and therefore, if a bug arises as inputs not considered by the dev initially, it would show up as a failure. What you're actually saying is that reaching full exhaustive testing is near impossible.
- icebraining 8y ago
  I think what they're saying is that both the code and test will contain the bug, since the bug is in the programmer's understanding rather than just in their coding. If I understood func(5, 6) should return 9, but the client actually wants 10, no amount of tests I write will reveal the error.
  - neo2006 8y ago
    I don't think that any form of formal specification will solve that issue. The only way to minimize the impact of it is to work on small iterations (agile) and be close to the client. One of the weakness in agile methodologies today is that the client representation (Product owner) is not aware of the client needs or too far from the client point of view.
    - sombremesa 8y ago
      This is roughly what I meant - there are ways to end up with zero bugs, but writing tests is usually insufficient for achieving such a goal.
- mikekchar 8y ago
  Testing for regressions is a a valuable thing to do, but your point is also valid. We can prove that it is impossible to test every scenario in general (for example, does this program end ;-) ). In my mind, unit tests exist for documenting what the code is doing now -- not to determine if it is correct (acceptance tests are different). Your goal for a unit test is to have it fail when the behaviour of the production code fails. This way you can verify that the changes you are making are having the effect you expected. Similarly, you can be warned when changes you make affect part of the code you didn't expect to be affected. Of course, you will never get full coverage, but tests are a programmer's tool. It allows you to reduce the amount of time you spend inspecting parts of the code you aren't working on. For example, when I'm writing code with good test coverage, I can find out what happens if I change an attribute in a structure simply by doing it and watching the tests fail. Each failure tells me where in the code I'm likely to have to make a change. If you make a small viewpoint change that tests are not to find bugs, but rather are a tool to help you understand the code base, they make a lot more sense. (IMHO)

gilbetron 8y ago

Do you have an actual source that corroborates that with actual data? It seems true, but I haven't found anything that actually tests the premise. I've found studies that show that code with lots of unit tests tends to be more bug free, but not anything about whether unit tests increases (or decreases) development time or whether test code actually has less bugs. I think it would be interesting if there is also a measure of code complexity, and if less complex test code correlates with less bugs in the test code.

Aeolun 8y ago
Arguing from the point of logic and formalisms the author is talking about, this would seem to be true.
- gilbetron 8y ago
  > Instead, it reduces that number to c' * N with c'<c and the difference being caused by your test coverage. That is unproven conjecture. It feels right, but just because it is written as a formula doesn't automatically mean it is correct.
  - peteradio 8y ago
    It is an axiom. If your testing is such that this isn't true then you are doing testing wrong, in a self-evident sort of way.
    - sidlls 8y ago
      That's about as useful as observing that if your code is such that it doesn't produce the correct result then you've written the code wrong, in a self-evident sort of way.
      - peteradio 8y ago
        > observing that if your code is such that it doesn't produce the correct result Hmm. Sounds like you are doing some sort of testing.

paulddraper 8y ago

Yeah, it's like saying "CAP says partitions decreases availability/consistency, so putting your app servers in a scale group will make things less available/consistent." That's really superficial analysis. Yes there is some fundamental tradeoffs. But it is possible to change the properties of a system or combine them in intelligent ways and move the curve. Or it's like saying that speedometers make your GPS less accurate due the Heisenberg principal.

p1necone 8y ago

The other fault in that reasoning is that usually test code is much much simpler than product code. It's a lot harder to write bugs when the structure of the code is just 1. call single function with some input data 2. verify that output data/side effects meet expectations 3. repeat for more sets of inputs.

neo2006 8y ago
One of the mistake I see a lot around this is when people start to write intelligent mocks. Usually this is where test bugs reside.
- p1necone 8y ago
  The best is when someone writes an entirely new implementation of something just for tests, and only calls that from their tests instead of the actual product code! Then you just get blank stares when you try to explain why what they've just done is completely pointless.
  - loup-vaillant 8y ago
    The way they do it may be pointless, but the intent is not. The crux is making sure your two implementation are independent. There are two ways to achieve this. The first is clean room. Just have independent persons or teams implement the same thing. With different tools and programming languages if you can. The second is use an entirely different approach for both. The test code for instance can be constructed around the tests, and for instance rely on the fact that everything happens at the same place at the same time (when you run the test). --- I'm currently writing on a secure handshakes generator. https://github.com/LoupVaillant/Monokex https://github.com/LoupVaillant/Monokex It generates specifications and source code from a Noise pattern. A bit like Noise Explorer, only with less features. Previous versions of the code were hand written, and the generated code is doing its best to look like it was hand written. Here's the result: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/monokex.c https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... Now how do I ensure the code match the specs? (The correctness of the specs itself is currently checked by hand.) I could write another implementation, but I'm only me, and I can't go erase my own memory to make a clean room implementation and compare the two. Instead, I took the specs, and wrote code that takes all the inputs, and spits out all the intermediate buffers and outputs, paying no heed to stuff like order of execution, or who does what. I only concentrated on generating the test vectors: https://github.com/LoupVaillant/Monocypher-Handshake/blob/master/vectors.c#L81 https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma... The structure of the vector generating code and the actual production code are very different. This is how I decorrelate mistakes, and make sure that if the two "implementations" agree, I'm very likely to have something that works. --- Of course, this is all a lot of effort, so I wouldn't do this for non-critical code.

foobarchu 8y ago

I think this holds for well written tests, but if one is not talented in writing them then more tests will simply bolster the bugs in the main code. For example, if a developer generate test data by running the program and copying it's output (something I see far more often than I'm comfortable with), they can tell themselves that section was tested thoroughly and must be bug free. Thus when someones notices an end result someplace is wrong, the first place checked may not be that "well tested" function. Tests are only as good as the domain knowledge of the person writing them.

mikekchar 8y ago

Bugs in tests exist (and are just as prevalent as bugs in production code in my experience). However, writing production code introduces a kind of constraint on the system. If you run a function with certain program state, then it returns a certain result. No matter what you do, this will happen because that's how you wrote the code. A test introduces the same constraint. Instead of implementing the function, it runs the function. No matter what you do, the test will run certain code and get a certain result. Tests pass when the constraints introduced by the production code match the constraints introduced by the tests. Because of this, if there is a software error in the production code, usually you need to have a corresponding error in the tests in order for the test to pass. This definitely happens from time to time, but the odds of it happening are much lower. Similarly, if you have a software error in the tests, you need to have a corresponding software error in the production code in order for the test to pass -- assuming that the test is actually exercising the code (something you can't always assume, unfortunately).

ericathegreat 8y ago

Agree. Consider the situation of data entry professionals, those people who transcribe audio recordings or enter data into databases. If you have one person entering data into a computer, then the odds of them introducing an error and failing to spot it are fairly high. If you have twice as many people entering twice as much data data, then the odds of an error getting introduced are roughly doubled. However, if you have those two people entering the same data, then their mistakes cancel each other out. If person A and person B both entered the same thing, it's extremely unlikely that it's incorrect. If they differ though, the a problem has been identified, and can now be fixed. The odds of both of those people entering the same piece of data incorrectly is tiny. Likewise, accidentally introducing a bug into both the production code and the test is pretty unlikely. That said though, if those two theoretical data entry people above are given the wrong data to enter, then they the system cannot protect them. They will both correctly enter incorrect data. "Garbage in, garbage out". Likewise, if the requirements of a piece of software are poorly understood, then it is quite likely that both the test and the production code will implement the same "bug". Writing tests won't fix a failure to understand the problem you're trying to solve. And they're not supposed to.

gdfasfklshg4 8y ago
> The odds of both of those people entering the same piece of data incorrectly is tiny. This is just not correct. There may be a systematic reason why they are making a mistake (e.g. a miss-pronounced word) in which case increasing the confidence intervals does not increase the accuracy. Check out the concepts of accuracy, precision etc from physical sciences.