3 ms·

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the t

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like people pick a stance and then stick with it forever when I couldn’t care less how I’ve been doing something for 10 years if today you showed me a better way.

smrq 2y ago
This is the way. My work codebase has probably 5% unit test coverage -- it's frontend and a lot of it isn't sensible to unit test -- but I'm quite happy to have the tests we do. If it's nontrivial logic, just test it. If it isn't (it's trivial, it's aesthetic, whatever your reason)... just don't.
wesselbindt 2y ago
>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, and in those I've rarely had to write a commit message like "fix tests", and all that without losing coverage.
- kbolino 2y ago
  It feels like there are two levels of test writing proficiency. The first is writing the tests that have high benefit and low cost: e.g. pure functions with comprehensive tabular tests, simple method chains that have well defined sequential behavior and few dependencies, high value regression tests against detailed bug reports, etc. IMO it's harder to argue against writing these tests than to argue for writing them. Then there's the second level of proficiency, related to what you're discussing with "test-implementation detail coupling". This is the domain of high test coverage, repeatable end-to-end tests, automated QA, etc. I've always struggled with this next level and I've yet to work in any environment where it was done effectively (if at all). It's also harder to argue for this kind of testing because the tests often end up brittle and false negatives drown out the benefits. Moreover, most of the discourse centers around the first level of proficiency only and it's much harder to find digestible advice for achieving the second.
- bluefirebrand 2y ago
  > This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details Depending on how high coverage you are aiming for, I find it hard to imagine a way to achieve it without inevitably tying the tests to implementation details
  - wesselbindt 2y ago
    Decoupling tests from what they test does require a concerted effort, and is a skill that requires practice, but in general not as much as you'd think. Most devs are quite comfortable getting rid of coupling between two non-test components (functions, classes, services, whatever) of their system. The main mental hurdle seems to be treating your automated test code as any other code, capable and worthy of being decoupled. There will always be some coupling between test and testee (for example, if I change my `double` function from doing `x -> 2 * x` to `x -> (x, x)`, my tests for `double` better fail), but there is a lot of coupling which is unnecessary, and this can often be removed. Like decoupling any two pieces of code, there is no one size fits all solution. There are some common sources of unnecessary coupling, and some rules of thumb to avoid them. For example, let's say we have a (public) sorting routine `sort` which consists of two (private) phases: `prep` and `finish`. The implementation of `sort` would look like function sort(xs): prepped_xs = prep(xs) return finish(prepped_xs) Let's look at two ways to write a test suite for `sort`. The first is quite simple, just chuck in a bunch of unsorted arrays of varying sizes and degrees of unsortedness, and assert that what comes out is sorted: assert sort([]) == [] assert sort([3, 2, 1]) == [1, 2, 3] etc The other way is to make the tests more specific by testing `sort`, `prep`, and `finish` separately. For example, we might mock `prep` and do x = [1, 2] sort(x) mocked_prep.assert_called_with(x) // and something similar for finish and have individual tests for `prep` and `finish`: assert is_prepped(prep([1, 2])) etc assert is_sorted(finish(prepped_xs)) etc Now suppose you decide that the code would be a lot more readable if some functionality of `prep` would move to `finish`. Nothing changes in the functionality of `sort` as this is purely a refactoring. When you make this change, the first test suite will pass, but the second will fail and require changes (i.e., it is coupled to implementation details of `sort`), because you've changed what `prep` and `finish` do. So here, by increasing the scope of your test suite from `prep` and `finish` to `sort`, you've achieved looser coupling between the test suite and what is being tested. This can be applied much more generally: by testing at the boundary of your system (at a higher scope), you achieve looser coupling between your tests and your code. That is, you'll have fewer false failures.
- armchairhacker 2y ago
  Even if the tests aren’t coupled to implementation details, in most projects the specification itself goes through many changes. Furthermore, as the implementation is being changed, it stops depending on some lower-level helper code and requires new code with a different purpose; the tests in the old code turn out to be largely (albeit not entirely) a waste of effort. Changing specifications and code which turns out to be unnecessary aren’t ideal. but I believe they’re inevitable to some extent (unless the project is a narrow re-implementation of something that already exists). There are questions like “how will people use this product?” and “what will they like/dislike about it?” that are crucial to the specification yet can’t be answered or even predicted very well until there’s already a MVP. And you can’t know exactly what helper classes and functions you will use to implement something until you have the working implementation. Of course, that doesn’t mean all tests are wasted effort; development will be slower if the developers have to spend more time debugging, due to not knowing where bugs originate from, due to not having tests. There’s a middle ground, where you have tests to catch probable and/or tricky bugs, and tests for code unlikely to be made redundant, but don’t spend too long on unnecessary tests for unnecessary code.
int_19h 2y ago
Overtesting usually comes from TDD cargo culting where literally every function in the code, no matter how small, is a "unit" that needs to be tested.
mabster 2y ago
All the places I've worked for had some balance here, but it would definitely be on the very few tests end. We would write tests to catch a bug in a low level system, and keep the test after. We had lots of Design by Contract, including Invariants that were enabled in debug mode. But the reality was that we couldn't test gameplay code very well. That changed so dramatically over the course of a project that if we did test we would just end up commenting tests by the end of a project. And as an optimisation guy, I would often have to change the "feel" of gameplay code to get performance out of code, which is checked by a Quality Assurance team, because it's subjective. That kind of stuff would make gameplay tests very brittle. The pace of game Dev was incredibly fast. We were struggling to get all our stuff in, never mind adding any scaffolding that would slow us down.

smrq 2y ago

This is the way. My work codebase has probably 5% unit test coverage -- it's frontend and a lot of it isn't sensible to unit test -- but I'm quite happy to have the tests we do. If it's nontrivial logic, just test it. If it isn't (it's trivial, it's aesthetic, whatever your reason)... just don't.

wesselbindt 2y ago

>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, and in those I've rarely had to write a commit message like "fix tests", and all that without losing coverage.

kbolino 2y ago
It feels like there are two levels of test writing proficiency. The first is writing the tests that have high benefit and low cost: e.g. pure functions with comprehensive tabular tests, simple method chains that have well defined sequential behavior and few dependencies, high value regression tests against detailed bug reports, etc. IMO it's harder to argue against writing these tests than to argue for writing them. Then there's the second level of proficiency, related to what you're discussing with "test-implementation detail coupling". This is the domain of high test coverage, repeatable end-to-end tests, automated QA, etc. I've always struggled with this next level and I've yet to work in any environment where it was done effectively (if at all). It's also harder to argue for this kind of testing because the tests often end up brittle and false negatives drown out the benefits. Moreover, most of the discourse centers around the first level of proficiency only and it's much harder to find digestible advice for achieving the second.
bluefirebrand 2y ago
> This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details Depending on how high coverage you are aiming for, I find it hard to imagine a way to achieve it without inevitably tying the tests to implementation details
- wesselbindt 2y ago
  Decoupling tests from what they test does require a concerted effort, and is a skill that requires practice, but in general not as much as you'd think. Most devs are quite comfortable getting rid of coupling between two non-test components (functions, classes, services, whatever) of their system. The main mental hurdle seems to be treating your automated test code as any other code, capable and worthy of being decoupled. There will always be some coupling between test and testee (for example, if I change my `double` function from doing `x -> 2 * x` to `x -> (x, x)`, my tests for `double` better fail), but there is a lot of coupling which is unnecessary, and this can often be removed. Like decoupling any two pieces of code, there is no one size fits all solution. There are some common sources of unnecessary coupling, and some rules of thumb to avoid them. For example, let's say we have a (public) sorting routine `sort` which consists of two (private) phases: `prep` and `finish`. The implementation of `sort` would look like function sort(xs): prepped_xs = prep(xs) return finish(prepped_xs) Let's look at two ways to write a test suite for `sort`. The first is quite simple, just chuck in a bunch of unsorted arrays of varying sizes and degrees of unsortedness, and assert that what comes out is sorted: assert sort([]) == [] assert sort([3, 2, 1]) == [1, 2, 3] etc The other way is to make the tests more specific by testing `sort`, `prep`, and `finish` separately. For example, we might mock `prep` and do x = [1, 2] sort(x) mocked_prep.assert_called_with(x) // and something similar for finish and have individual tests for `prep` and `finish`: assert is_prepped(prep([1, 2])) etc assert is_sorted(finish(prepped_xs)) etc Now suppose you decide that the code would be a lot more readable if some functionality of `prep` would move to `finish`. Nothing changes in the functionality of `sort` as this is purely a refactoring. When you make this change, the first test suite will pass, but the second will fail and require changes (i.e., it is coupled to implementation details of `sort`), because you've changed what `prep` and `finish` do. So here, by increasing the scope of your test suite from `prep` and `finish` to `sort`, you've achieved looser coupling between the test suite and what is being tested. This can be applied much more generally: by testing at the boundary of your system (at a higher scope), you achieve looser coupling between your tests and your code. That is, you'll have fewer false failures.
armchairhacker 2y ago
Even if the tests aren’t coupled to implementation details, in most projects the specification itself goes through many changes. Furthermore, as the implementation is being changed, it stops depending on some lower-level helper code and requires new code with a different purpose; the tests in the old code turn out to be largely (albeit not entirely) a waste of effort. Changing specifications and code which turns out to be unnecessary aren’t ideal. but I believe they’re inevitable to some extent (unless the project is a narrow re-implementation of something that already exists). There are questions like “how will people use this product?” and “what will they like/dislike about it?” that are crucial to the specification yet can’t be answered or even predicted very well until there’s already a MVP. And you can’t know exactly what helper classes and functions you will use to implement something until you have the working implementation. Of course, that doesn’t mean all tests are wasted effort; development will be slower if the developers have to spend more time debugging, due to not knowing where bugs originate from, due to not having tests. There’s a middle ground, where you have tests to catch probable and/or tricky bugs, and tests for code unlikely to be made redundant, but don’t spend too long on unnecessary tests for unnecessary code.

int_19h 2y ago

Overtesting usually comes from TDD cargo culting where literally every function in the code, no matter how small, is a "unit" that needs to be tested.

mabster 2y ago

All the places I've worked for had some balance here, but it would definitely be on the very few tests end. We would write tests to catch a bug in a low level system, and keep the test after. We had lots of Design by Contract, including Invariants that were enabled in debug mode. But the reality was that we couldn't test gameplay code very well. That changed so dramatically over the course of a project that if we did test we would just end up commenting tests by the end of a project. And as an optimisation guy, I would often have to change the "feel" of gameplay code to get performance out of code, which is checked by a Quality Assurance team, because it's subjective. That kind of stuff would make gameplay tests very brittle. The pace of game Dev was incredibly fast. We were struggling to get all our stuff in, never mind adding any scaffolding that would slow us down.