4 ms·

The problem is that most tests break frequently not because of bugs, but because of some environment dependency that changed. The tests break and we have to fi

by vfc1 8y ago

The problem is that most tests break frequently not because of bugs, but because of some environment dependency that changed. The tests break and we have to fix them not because a bug was found, but because something else changed that affected the test. Most of the time spent with tests is not to catch bugs. When was the last time that you caught a significant bug with a test?

sethammons 8y ago
For us, a failing unit test is almost always a bug. Lots of people write terrible unit tests. Too much mocking, asserting methods called, etc. Brittle. Test the edge of your public interface. Supporting methods only need specific unit tests if they are sufficiently complex. If your unit's interface changes or it's behavior changes, it should fail. That is a good thing.
- vfc1 8y ago
  think there is a lot of religious-like thinking when it comes down to tests: many people swear by them even against all the day to day proof to the contrary that they are really not that useful at the application level. I think if we are shipping a framework to thousands of users its great that it has unit tests, but if its an in-house application built by a team of 10 developers, I doubt that the tests are doing anything useful. But what happens is that these best practices that are used by developers of libraries are then used blindly at the level of application code, without pausing to think if those are indeed best practices in that particular context. Developing software at the leaves of the dependency tree (the application) is very different than developing libraries. Some things are the same while many others are not. The end result in practice is a lot of wasted time and effort spent at the application level writing bad tests that yield no results, all in the name of the testing religion. Because that is what it really is, its an unprovable belief system with a set of rituals that you absolutely must follow blindly, or else.
  - vonholstein 8y ago
    Do you extend that opinion to both unit and integration tests? I agree with you that unit tests at the application level have low ROI, but I've found integration tests to be invaluable for applications/services, especially when performing large scale refactoring. Where I work currently there is a fairly large service, full of unit tests, but which the team is wary of changing because it lacks integration tests.
  - sethammons 8y ago
    Usually "unit tests" should be for verifying error code paths. These are valuable at the library/package level and maybe at some level of internal integration. 100% code coverage is likely useless most of the time. At the application level, testing user inputs to user expectations is highly valuable (perhaps more so than unit tests in a lot of contexts). It is this integration/acceptance testing that any application with users should have under automation. Any codebase lacking that risks breaking or regression at the cost of user experience (which, depending on your app might be a fair trade off).
Eleopteryx 8y ago
>When was the last time that you caught a significant bug with a test? Tests only really catch bugs when you're refactoring the implementation of something while trying to keep its behavior the same. You're right, in many ways, tests exist to be broken. But they still serve their purpose. They either break because you unintentionally changed behavior and need to fix your code, or because you intentionally changed behavior and need to change your tests.
koala_man 8y ago
>When was the last time that you caught a significant bug with a test? This happens at least once a week, and that's just for me on a ~15 person team. I work on a compiler and most of our tests are in terms of user visible effects from running snippets. If any of them break, it's either a rare test harness failure or a real bug. Whether or not any particular spec violation is significant is another matter of course, but they're definitely genuine bugs.
nimblerabit 8y ago
> When was the last time that you caught a significant bug with a test? Constantly. This is an extremely surprising question to me, our unit tests catch bugs all the time.

by vfc1 8y ago

sethammons 8y ago

For us, a failing unit test is almost always a bug. Lots of people write terrible unit tests. Too much mocking, asserting methods called, etc. Brittle. Test the edge of your public interface. Supporting methods only need specific unit tests if they are sufficiently complex. If your unit's interface changes or it's behavior changes, it should fail. That is a good thing.

vfc1 8y ago
think there is a lot of religious-like thinking when it comes down to tests: many people swear by them even against all the day to day proof to the contrary that they are really not that useful at the application level. I think if we are shipping a framework to thousands of users its great that it has unit tests, but if its an in-house application built by a team of 10 developers, I doubt that the tests are doing anything useful. But what happens is that these best practices that are used by developers of libraries are then used blindly at the level of application code, without pausing to think if those are indeed best practices in that particular context. Developing software at the leaves of the dependency tree (the application) is very different than developing libraries. Some things are the same while many others are not. The end result in practice is a lot of wasted time and effort spent at the application level writing bad tests that yield no results, all in the name of the testing religion. Because that is what it really is, its an unprovable belief system with a set of rituals that you absolutely must follow blindly, or else.
- vonholstein 8y ago
  Do you extend that opinion to both unit and integration tests? I agree with you that unit tests at the application level have low ROI, but I've found integration tests to be invaluable for applications/services, especially when performing large scale refactoring. Where I work currently there is a fairly large service, full of unit tests, but which the team is wary of changing because it lacks integration tests.
- sethammons 8y ago
  Usually "unit tests" should be for verifying error code paths. These are valuable at the library/package level and maybe at some level of internal integration. 100% code coverage is likely useless most of the time. At the application level, testing user inputs to user expectations is highly valuable (perhaps more so than unit tests in a lot of contexts). It is this integration/acceptance testing that any application with users should have under automation. Any codebase lacking that risks breaking or regression at the cost of user experience (which, depending on your app might be a fair trade off).

Eleopteryx 8y ago

>When was the last time that you caught a significant bug with a test? Tests only really catch bugs when you're refactoring the implementation of something while trying to keep its behavior the same. You're right, in many ways, tests exist to be broken. But they still serve their purpose. They either break because you unintentionally changed behavior and need to fix your code, or because you intentionally changed behavior and need to change your tests.

koala_man 8y ago

>When was the last time that you caught a significant bug with a test? This happens at least once a week, and that's just for me on a ~15 person team. I work on a compiler and most of our tests are in terms of user visible effects from running snippets. If any of them break, it's either a rare test harness failure or a real bug. Whether or not any particular spec violation is significant is another matter of course, but they're definitely genuine bugs.

nimblerabit 8y ago

> When was the last time that you caught a significant bug with a test? Constantly. This is an extremely surprising question to me, our unit tests catch bugs all the time.