5 ms·

>Bugs are correlated with lines of code and TDD forces writing more code so how can it reduce bug counts? If the test code has no bugs then just write the rest

by jasode 8y ago

>Bugs are correlated with lines of code and TDD forces writing more code so how can it reduce bug counts? If the test code has no bugs then just write the rest of the code in the same style I'm not advocating for TDD (the programmer methodology in the IDE) but the author's explanation about "test code" isn't correct. Code written for explicit purposes of a test to exercise other code has been shown to increase correctness. E.g. SQLite database has 711x more test code than the core engine code.[1] (I made a previous comment why this is possible: https://news.ycombinator.com/item?id=15593121 https://news.ycombinator.com/item?id=15593121) Low-level infrastructure code like database engines, string manipulation libraries, crypto libraries, math libraries, network protocol routines, etc can benefit from a suite of regression tests. It's the high-level stuff like GUIs in webpages being tested with Selenium or LoadRunner that has conflicting business value because altering one pixel can have a cascading effect of breaking a bunch of fragile UI test scripts. [1] https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

herge 8y ago
Maybe the spry takeaway is that you should write tests for things you want to make sure they will not break. Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test. However, your business logic, or network protocol routine, those should not break even if you heavily refactor or add new features (especially business logic where a broken behaviour might seem correct), so those need to be heavily tested. If it is hard to test the juicy parts like business logic without also dragging in the UI, different OS/platform/db parts, etc, then you should look at how your application is structured and if it is really optimized for writing good tests.
- jasode 8y ago
  >Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test. For brevity in the previous comment, I didn't fully flesh out the background on why fragile UI tests get created. It happens accidentally. What sometimes happens is the the UI tester uses a "macro recorder" to record mouse movements and clicks. But then, a programmer shifts the position of a zipcode field by one pixel which throws the script off because it expected UI elements in a different spot. Fixing the broken UI tests is time consuming and can leave a bad impression that tests create a lot of effort for very little payback. The return-on-investment of UI tests depends on the business circumstances. I'm guessing Boeing and Airbus have automated UI tests that sometimes break when programmers change things which causes rework. However, the pain of fixing the UI tests and keeping it sync'd with the UI code is worth it for avionics software.
  - strken 8y ago
    Visual diffing in UI tests is something that actually can be solved with more code, see https://github.com/americanexpress/jest-image-snapshot https://github.com/americanexpress/jest-image-snapshot, https://github.com/gemini-testing/gemini https://github.com/gemini-testing/gemini, or https://github.com/BBC-News/wraith https://github.com/BBC-News/wraith.
- gav 8y ago
  > you should write tests for things you want to make sure they will not break You should also write tests for things that are already broken before you work on the fix so that you can be sure it's actually fixed. Basically the red/green/refactor cycle[1] from TDD. [1] http://www.brendanconnolly.net/test-driven-testing-red-green-refactor/ http://www.brendanconnolly.net/test-driven-testing-red-green...
greenyouse 8y ago
You can tune the pixel difference in Selenium testing so it doesn't fail when under the acceptable pixel change limit. If it's off by one or two no biggie but off by 30 or 50 then maybe there was a structural change to the page, so it should fail. Something like "misMatchTolerance" from wdio-visual-regression-service[0] would allow for tweaking this. I'm sure there are similar tools for other languages. If the UI tests catch bugs during development or help the team during a data migration, they're probably still worth having. [0] https://github.com/zinserjan/wdio-visual-regression-service https://github.com/zinserjan/wdio-visual-regression-service
userbinator 8y ago
The question that always comes to mind whenever testing is discussed is "how do you know the test code is itself free of bugs?" I distinctively remember once posing that question in a meeting about testing, and a manager replying --- seriously --- with "then perhaps the test code should itself have tests." Someone else must've come up with that before too, because (at a different job) I've also worked on a codebase where a surprising number of tests were basically testing the function of another test.
- gjm11 8y ago
  You don't know that the test code is free of bugs. You don't need to. Case 0: No bugs in the test code. All is well. Case 1: Bug in the test code that causes some bugs in the real code not to get caught. That's bad, but you're no worse off than if you didn't have the test at all. Case 2: Bug in the test code that causes correct real code to look buggy. Result: the test fails, you look for problems, most likely you find that the problem is in the test code and fix it. Going forward, you have a working test. Case 3: Bug in the test code that makes something else break. This can happen and is genuinely bad, but (1) it only affects testing, not your actual product, and (2) most bugs don't behave that way. The test code is a net win if the bugs it catches in your real code are worth the effort of writing and debugging the test code. That's no less true on account of the possibility of bugs in the test code. It just means that when you estimate the benefit you have to be aware that sometimes the tests might be less effective because of bugs, and when you estimate the cost you have to be aware that you have to debug the code, not just write it the first time. And then you just ... decide what's the best tradeoff, just like everything else in engineering. (And no, you don't need tests for your test code. The test code is a test for itself as well as for the code it's testing.)
  - pintxo 8y ago
    I see 5 cases: Code | Test | Result -------------------------------------------------- fine | fine | a) We have a regression test, yay! fine | buggy | b) Someone breaks the code to make the test work. Oops. c) Someone fixes the test, we have a regression test, yay! buggy | fine | d) We'll fix the code, and we have a regression test, yay! buggy | buggy | e) Bug remains in code. Oops. If the probability of introducing an error is p, Code | Test | Probabilities | p = .01 (one in ten) -------------------------------------------------- fine | fine | a) (1-p) * (1-p) | .9801 fine | buggy | b) (1-p) * p * .5 | .00495 c) (1-p) * p * .5 | .00495 buggy | fine | d) p * (1-p) | .0099 buggy | buggy | e) p * p | .0001 So we see, probability for: no harm done .9801 (a) bugs found + fixed .01485 (c,d) bugs introduced / not found .00505 (b,e) The above completely ignores the fact, that depending on the code base there will be significantly more test code than production code. But then test code is quite often highly redundant, and might actually have a lower defect rate itself. Also the probability on introducing an error in the production code and the test code, might actually not be statistically independent, which I assumes here. So take with a grain of salt. [Edit] Actually d) could also end negatively. Guess a working model would have to take into account that on failing test cases, a sensible developer should take a step back and reason about why this happened. So the negative outcomes would be (hopefully) less likely than the positive ones here.
danmaz74 8y ago
What the author didn't consider is that test code adds redundancy to the code base - if you introduce a bug only in the test code, while the tested code is correct, will make the test fail, and you will fix this. Where instead the amount of test code can become a problem, in my experience, is with maintenance. Efficiency is very important.

by jasode 8y ago

herge 8y ago

Maybe the spry takeaway is that you should write tests for things you want to make sure they will not break. Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test. However, your business logic, or network protocol routine, those should not break even if you heavily refactor or add new features (especially business logic where a broken behaviour might seem correct), so those need to be heavily tested. If it is hard to test the juicy parts like business logic without also dragging in the UI, different OS/platform/db parts, etc, then you should look at how your application is structured and if it is really optimized for writing good tests.

jasode 8y ago
>Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test. For brevity in the previous comment, I didn't fully flesh out the background on why fragile UI tests get created. It happens accidentally. What sometimes happens is the the UI tester uses a "macro recorder" to record mouse movements and clicks. But then, a programmer shifts the position of a zipcode field by one pixel which throws the script off because it expected UI elements in a different spot. Fixing the broken UI tests is time consuming and can leave a bad impression that tests create a lot of effort for very little payback. The return-on-investment of UI tests depends on the business circumstances. I'm guessing Boeing and Airbus have automated UI tests that sometimes break when programmers change things which causes rework. However, the pain of fixing the UI tests and keeping it sync'd with the UI code is worth it for avionics software.
- strken 8y ago
  Visual diffing in UI tests is something that actually can be solved with more code, see https://github.com/americanexpress/jest-image-snapshot https://github.com/americanexpress/jest-image-snapshot, https://github.com/gemini-testing/gemini https://github.com/gemini-testing/gemini, or https://github.com/BBC-News/wraith https://github.com/BBC-News/wraith.
gav 8y ago
> you should write tests for things you want to make sure they will not break You should also write tests for things that are already broken before you work on the fix so that you can be sure it's actually fixed. Basically the red/green/refactor cycle[1] from TDD. [1] http://www.brendanconnolly.net/test-driven-testing-red-green-refactor/ http://www.brendanconnolly.net/test-driven-testing-red-green...

greenyouse 8y ago

You can tune the pixel difference in Selenium testing so it doesn't fail when under the acceptable pixel change limit. If it's off by one or two no biggie but off by 30 or 50 then maybe there was a structural change to the page, so it should fail. Something like "misMatchTolerance" from wdio-visual-regression-service[0] would allow for tweaking this. I'm sure there are similar tools for other languages. If the UI tests catch bugs during development or help the team during a data migration, they're probably still worth having. [0] https://github.com/zinserjan/wdio-visual-regression-service https://github.com/zinserjan/wdio-visual-regression-service

userbinator 8y ago

The question that always comes to mind whenever testing is discussed is "how do you know the test code is itself free of bugs?" I distinctively remember once posing that question in a meeting about testing, and a manager replying --- seriously --- with "then perhaps the test code should itself have tests." Someone else must've come up with that before too, because (at a different job) I've also worked on a codebase where a surprising number of tests were basically testing the function of another test.

gjm11 8y ago
You don't know that the test code is free of bugs. You don't need to. Case 0: No bugs in the test code. All is well. Case 1: Bug in the test code that causes some bugs in the real code not to get caught. That's bad, but you're no worse off than if you didn't have the test at all. Case 2: Bug in the test code that causes correct real code to look buggy. Result: the test fails, you look for problems, most likely you find that the problem is in the test code and fix it. Going forward, you have a working test. Case 3: Bug in the test code that makes something else break. This can happen and is genuinely bad, but (1) it only affects testing, not your actual product, and (2) most bugs don't behave that way. The test code is a net win if the bugs it catches in your real code are worth the effort of writing and debugging the test code. That's no less true on account of the possibility of bugs in the test code. It just means that when you estimate the benefit you have to be aware that sometimes the tests might be less effective because of bugs, and when you estimate the cost you have to be aware that you have to debug the code, not just write it the first time. And then you just ... decide what's the best tradeoff, just like everything else in engineering. (And no, you don't need tests for your test code. The test code is a test for itself as well as for the code it's testing.)
- pintxo 8y ago
  I see 5 cases: Code | Test | Result -------------------------------------------------- fine | fine | a) We have a regression test, yay! fine | buggy | b) Someone breaks the code to make the test work. Oops. c) Someone fixes the test, we have a regression test, yay! buggy | fine | d) We'll fix the code, and we have a regression test, yay! buggy | buggy | e) Bug remains in code. Oops. If the probability of introducing an error is p, Code | Test | Probabilities | p = .01 (one in ten) -------------------------------------------------- fine | fine | a) (1-p) * (1-p) | .9801 fine | buggy | b) (1-p) * p * .5 | .00495 c) (1-p) * p * .5 | .00495 buggy | fine | d) p * (1-p) | .0099 buggy | buggy | e) p * p | .0001 So we see, probability for: no harm done .9801 (a) bugs found + fixed .01485 (c,d) bugs introduced / not found .00505 (b,e) The above completely ignores the fact, that depending on the code base there will be significantly more test code than production code. But then test code is quite often highly redundant, and might actually have a lower defect rate itself. Also the probability on introducing an error in the production code and the test code, might actually not be statistically independent, which I assumes here. So take with a grain of salt. [Edit] Actually d) could also end negatively. Guess a working model would have to take into account that on failing test cases, a sensible developer should take a step back and reason about why this happened. So the negative outcomes would be (hopefully) less likely than the positive ones here.

danmaz74 8y ago

What the author didn't consider is that test code adds redundancy to the code base - if you introduce a bug only in the test code, while the tested code is correct, will make the test fail, and you will fix this. Where instead the amount of test code can become a problem, in my experience, is with maintenance. Efficiency is very important.