3 ms·

Agreed I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not wo

by smoe 6y ago

Agreed I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not working. They are a fairly cheap way to tell me during development that this ain't going to work no matter what. And if everything compiles and the tests and reviews pass, I think this has a good shot at working out in production, but there still might well be cases and circumstances I haven't or coulnd't have considered. No setup will cover 100%, but with not that much work you can already get quite far. The tricky thing in my opinions is more on deciding which kind of tests, processes and automations are actually worth implementing and maintaining at a given stage of a project.

table-delete 6y ago
Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors." The preface to this is even more enlightening: "When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors. Therefore, don’t test a program to show that it works; rather, you should start with the assumption that the program contains errors (a valid assumption for almost any program) and then test the program to find as many of the errors as possible."
- faizshah 6y ago
  I like this quote from Kent Beck: “ It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test? You should test things that might break. If code is so simple that it can't possibly break, and you measure that the code in question doesn't actually break in practice, then you shouldn't write a test for it... Testing is a bet. The bet pays off when your expectations are violated [when a test that you expect to pass fails, or when a test that you expect to fail passes]... So, if you could, you would only write those tests that pay off. Since you can't know which tests would pay off (if you did, then you would already know and you wouldn't be learning anything), you write tests that might pay off. As you test, you reflect on which kinds of tests tend to pay off and which don't, and you write more of the ones that do pay off, and fewer of the ones that don't.” Source: https://softwareengineering.stackexchange.com/a/244709 https://softwareengineering.stackexchange.com/a/244709
  - josephg 6y ago
    When I’ve gone full TDD on projects, I’m always surprised to find there’s a power law distribution or something on failing tests. Most of the tests I write never catch a bug in the life of the software. Something like 90% of the value of a test suite could be achieved with only about 10% of the tests. Of course, the trick is figuring out which tests are going to repeatedly fail ahead of time. But there’s something interesting down this rabbit hole. I wonder if it would be worth collecting per-test stats through the life of a project to explore this.
    - lmm 6y ago
      You'd get a lot of the value by just being willing to delete tests once it's clear they're not serving any purpose. Sadly a lot of people would rather see a high test coverage percentage than have an effective test suite.
      - nitrogen 6y ago
        What is the value of deleting a test once it's written, though?
        lmm 6y ago
        Tests impose a maintenance burden like any other lines of code. Just making it harder to navigate to relevant code (including useful tests) is a significant cost.
        ornornor 6y ago
        You’ve already paid the cost for writing the test, might as well keep it. If a test fails when modifying the codebase or adding a feature or updating dependencies, it still serves a purpose in my opinion even if fixing it takes some time. Having the test no matter how trivial also helps boost confidence that a new change didn’t break anything no matter how trivial. But tests should be refactored once in a while, and if a thing is complicated to setup for testing, either the thing to test should be refactored before shipping and/or the developer working on that thing should also ship a factory and helper functions for testing that thing.

by smoe 6y ago

table-delete 6y ago

Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors." The preface to this is even more enlightening: "When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors. Therefore, don’t test a program to show that it works; rather, you should start with the assumption that the program contains errors (a valid assumption for almost any program) and then test the program to find as many of the errors as possible."

faizshah 6y ago
I like this quote from Kent Beck: “ It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test? You should test things that might break. If code is so simple that it can't possibly break, and you measure that the code in question doesn't actually break in practice, then you shouldn't write a test for it... Testing is a bet. The bet pays off when your expectations are violated [when a test that you expect to pass fails, or when a test that you expect to fail passes]... So, if you could, you would only write those tests that pay off. Since you can't know which tests would pay off (if you did, then you would already know and you wouldn't be learning anything), you write tests that might pay off. As you test, you reflect on which kinds of tests tend to pay off and which don't, and you write more of the ones that do pay off, and fewer of the ones that don't.” Source: https://softwareengineering.stackexchange.com/a/244709 https://softwareengineering.stackexchange.com/a/244709
- josephg 6y ago
  When I’ve gone full TDD on projects, I’m always surprised to find there’s a power law distribution or something on failing tests. Most of the tests I write never catch a bug in the life of the software. Something like 90% of the value of a test suite could be achieved with only about 10% of the tests. Of course, the trick is figuring out which tests are going to repeatedly fail ahead of time. But there’s something interesting down this rabbit hole. I wonder if it would be worth collecting per-test stats through the life of a project to explore this.
  - lmm 6y ago
    You'd get a lot of the value by just being willing to delete tests once it's clear they're not serving any purpose. Sadly a lot of people would rather see a high test coverage percentage than have an effective test suite.
    - nitrogen 6y ago
      What is the value of deleting a test once it's written, though?
      - lmm 6y ago
        Tests impose a maintenance burden like any other lines of code. Just making it harder to navigate to relevant code (including useful tests) is a significant cost.
        ornornor 6y ago
        You’ve already paid the cost for writing the test, might as well keep it. If a test fails when modifying the codebase or adding a feature or updating dependencies, it still serves a purpose in my opinion even if fixing it takes some time. Having the test no matter how trivial also helps boost confidence that a new change didn’t break anything no matter how trivial. But tests should be refactored once in a while, and if a thing is complicated to setup for testing, either the thing to test should be refactored before shipping and/or the developer working on that thing should also ship a factory and helper functions for testing that thing.