9 ms·

I genuinely don't get how you can rely primarily on unit tests to prevent regressions in your code. They are at such a fine granularity that most significant ch

I genuinely don't get how you can rely primarily on unit tests to prevent regressions in your code. They are at such a fine granularity that most significant changes require restructuring the "units" in a way that invalidates the tests. Then we're left with these aenemic E2E tests (because fast!) that only tell you if something basic is broken. These CD advocates seem to be saying that you can take human out of the testing equation and rely solely on automation. I just don't buy it. I haven't seen true continuous delivery in the wild and I'm starting to think this is just the empty promises of a club of consultants who peddle their wares to unsuspecting VPs. Tell me what I'm missing.

dimes 5y ago
I rewrote the backend on a team I used to work on. The service had a ton of unit tests. Given that this was a full rewrite, those unit test were useless. I spent the first few days writing a comprehensive suite of integration tests I could run against the existing service. These tests directly mimicked client calls, so the same tests should be just as valid for the rewritten service. Using these tests, I was able to catch 90%+ of potential issues before cutting over to the new service. Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice.
- mumblemumble 5y ago
  I find that, in this life, you usually get what you pay for, and, compared to other options, unit tests' primary virtue is that they're inexpensive.
  - megameter 5y ago
    Unit tests help verify individual components of a system - which makes them top-of-mind for library code. I think the issue with them lies in that most developers aren't shipping libraries, they're shipping integrated systems, so there's no component worth testing. (you can always invent one, but that's just overcomplicating the code). At the same time, it's also genuinely hard to write good, principled tests of integrated systems, harder than it is to code up a thing that kinda-works and then manually debugging it enough to ship. You have to have the system set up to be tested, and feature complexity actively resists this - you fight a losing battle against "YOLO code" that gets the effect at the expense of going around the test paradigm.
- thatswrong0 5y ago
  I've arrived at this exact same conclusion for frontend work as well. I always go for integration tests first, and only rely on unit tests if hitting some edge case is hard via integration test.
  - thatswrong0 5y ago
    And to clarify, if any individual function reaches some arbitrary level of irreducible complexity, then I'll absolutely unit test that. It's kind of a "you know it when you see it" kind of thing.
- jniedrauer 5y ago
  How does this scale though? If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. As your project expands, the tests will take longer and longer to run. Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. The feedback loop becomes so slow that it's not helpful during actual coding. At best, it's a semi-useful release gate. Is there another way?
  - abystander 5y ago
    I've seen this more often than not. The pyramid is a pyramid for a reason, and I'm somewhat skeptical that we just have to throw out unit tests or they suck or something. You don't have to be a TDD acolyte to find unit tests useful or essential.
  - overscore 5y ago
    > If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. That is a very normal setup. > Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. This is called flakiness and is generally a symptom not to be ignored, as it is almost always indicative of bigger issues. It's rare that flakiness is limited to test environments. Instead it's much more likely that whatever your smoke tests are experiencing is a something end-users are also intermittently hitting. > The feedback loop becomes so slow that it's not helpful during actual coding. Devs can write their own unit tests when working on their assigned tasks. Smoke tests are designed to run when you're trying to integrate those changes into the existing codebase. At that point, you have the calculus all wrong. Smoke tests slow down devs enough that they don't merge broken code into production. That is a useful release gate unto itself. If unit tests pass but smoke tests fail, then often (the vast majority of the time in my experience) the issue is that either the dev didn't understand the task or, more often, didn't understand the system they were integrating into.
- abystander 5y ago
  > Personally, I find unit tests to be mostly useless. Meanwhile I've found myself in the position where people eschewed unit testing entirely under this "just do integration tests" thinking and now I'm stuck fumbling around with flaky and hard to run integration tests that take forever to run. Unit tests are good because they test whether the code you've written work as you expect. They run instantly. They don't require a full setup with other integrations. This is indispensable - while it's nice to see a service integrating well and behaving as expected, it's also very useful to know practically instantaneously that the logic you're calling from a controller or something isn't broken. And if it's really tough you can easily put in breakpoints and step through, etc.
- everyone 5y ago
  I'm a game dev and over time I've settled on using two groups of tests for my projects. Both at opposite ends of the spectrum. . 1. Unit tests. But I only write them for stuff that needs them.. Eg. Some complex math functions that translate between coordinate systems; the point of the unit tests is to confirm that the functions are doing exactly what I think they are doing. With mathsy stuff it can be very easy to look at the output of some function and think, that looks fine, but in reality its actually slightly off, and not exactly what it should be. The unit tests are to confirm that its really doing what I think its doing. . 2.Acceptance tests by a human. Theres a spreadsheet of everything you can do in the game and what should happen. Eg. press this button -> door should open.. As we add features we add more stuff to this list. At regular intervals and before any release several humans try every test on various hardware. This is to catch big / complex bugs and regressions. Its super tedious but it has to be done imo. Automating this would be an insane amount of work and also pointless as we are also testing the hardware, you get weird problems with certain GPUs, gamepads, weird smartphones etc. . I find those two types of tests to be essential, the bare minimum. But also anything in between, like some kind of automated integration testing is just a shittone of work and will only be useful for a relatively brief period of development, changes will quickly render those sort of tests useless.
  - dimes 5y ago
    Yes, totally agree. Any code that has complicated logic with few / no dependencies benefits from unit testing.
- tablespoon 5y ago
  > Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice. I think they're mostly useless when refactoring, but they're useful when writing new code and and making relatively small to medium sized changes. For new code, it's helpful to me at least to express my intentions in a more concrete form and it gives me more confidence that I didn't miss something. For making relatively small changes, they help catch fined-gained regressions. Even if I meant to make a change, a failing test forces me to think about handling a particular case correctly that I might have forgotten. The kind of unit test I do hate are the ones that are so mock-heavy that they're pretty much only testing the structure of your codebase (did you call all the methods the right order and nothing more?). I was once on a team where that was pretty much all they wrote, and they were very resistant to any level of integration unit testing because (I think) they read in a opinionated book somewhere that low level tests were good enough (they weren't).
  - 0xabe 5y ago
    When refactoring, unit tests confirm that you did it right (or wrong).
    - h0l0cube 5y ago
      Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. Integration tests and E2E tests are far more valuable because they're attacking it at the business logic side, which is far less volatile, and particularly in a refactor, a useful invariant.
      - tablespoon 5y ago
        > Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. That's not a bad thing, though. DRY might be fine for your main implementation, but redundancy is a time-tested way of catching errors (a.k.a. "double checking").
- marcosdumay 5y ago
  If you have some code that if its callers changed, they would stop using that code or use it on a different place, it's a unit and it's a good idea to unit test it. If you have some code that if its callers changed you would want to change it too, then it's on the same unit as the calling code, and it's bad to divide it away.
colordrops 5y ago
Maybe I'm missing something but when did continuous delivery imply no human testing? My last two jobs have had what I understand to be continuous delivery, to great success, but perhaps it was something other than CD.
- ehutch79 5y ago
  What's being sold as TDD/CD/ETC has a lot of hyperbole involved.
sparker72678 5y ago
We deploy to production multiple times per day, with no human intervention, successfully. Our test suite includes some unit and full end to end tests, and yea sometimes a breaking change slips through, we write a test to catch it next time, and move on. (Though our deploy process involves spinning up new instances to run our code, and our load balance won’t switch over to new boxes if they return 500s, so fully broken deploys don’t actually get hit by customers.) Different projects have different complexities and differing levels of CD viability, but there are definitely many people truly doing automated continuous deployment all day every day.
alisonkisk 5y ago
what counts as "C" in CD? FANG push changes to each of many components daily, with feature flags. of course each change goes through stages: dev, preprod, prod over a few days.
fennecfoxen 5y ago
> I genuinely don't get how you can rely primarily on unit tests to prevent regressions in your code. They are at such a fine granularity that most significant changes require restructuring the "units" in a way that invalidates the tests. Besides less-anemic E2E tests, this is why type systems exist. Define a well-typed interface, code to the interface, test the units — and then it's hard to make terrible mistakes wiring things up. To realize the full benefit does, of course, require substantially better data structures than string-to-Any hashes, and it requires some proper old-school software engineering (i.e. injecting a client as an argument and not just mocking out the someOtherClient.client() method.) And there is some overhead, though I'd say that a proper null-checking pass alone would be worth that much. (Sorry, Java, your types are terrible.)
deleted 5y ago
[deleted]
throwaway894345 5y ago
I’ve done CD in production. If you have “enough” test automation, you can have enough confidence that significant regressions become rare, and when they do crop up you can find and fix them in minutes-to-hours from finding the bug to releasing a fix to production. It works very well if you can tolerate that some of your users will have (typically minor) issues once in a while. Coupled with blue/green or canary deployments you can move pretty quickly while minimizing the impact of bugs.
ledauphin 5y ago
I don't agree that you can take humans out of the equation, particularly for new code. but functional programming makes unit tests the natural way to regression-test the top 80 to 90 percent of your codebase, and those don't have to target tiny pieces - they can target large units.

dimes 5y ago

I rewrote the backend on a team I used to work on. The service had a ton of unit tests. Given that this was a full rewrite, those unit test were useless. I spent the first few days writing a comprehensive suite of integration tests I could run against the existing service. These tests directly mimicked client calls, so the same tests should be just as valid for the rewritten service. Using these tests, I was able to catch 90%+ of potential issues before cutting over to the new service. Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice.

mumblemumble 5y ago
I find that, in this life, you usually get what you pay for, and, compared to other options, unit tests' primary virtue is that they're inexpensive.
- megameter 5y ago
  Unit tests help verify individual components of a system - which makes them top-of-mind for library code. I think the issue with them lies in that most developers aren't shipping libraries, they're shipping integrated systems, so there's no component worth testing. (you can always invent one, but that's just overcomplicating the code). At the same time, it's also genuinely hard to write good, principled tests of integrated systems, harder than it is to code up a thing that kinda-works and then manually debugging it enough to ship. You have to have the system set up to be tested, and feature complexity actively resists this - you fight a losing battle against "YOLO code" that gets the effect at the expense of going around the test paradigm.
thatswrong0 5y ago
I've arrived at this exact same conclusion for frontend work as well. I always go for integration tests first, and only rely on unit tests if hitting some edge case is hard via integration test.
- thatswrong0 5y ago
  And to clarify, if any individual function reaches some arbitrary level of irreducible complexity, then I'll absolutely unit test that. It's kind of a "you know it when you see it" kind of thing.
jniedrauer 5y ago
How does this scale though? If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. As your project expands, the tests will take longer and longer to run. Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. The feedback loop becomes so slow that it's not helpful during actual coding. At best, it's a semi-useful release gate. Is there another way?
- abystander 5y ago
  I've seen this more often than not. The pyramid is a pyramid for a reason, and I'm somewhat skeptical that we just have to throw out unit tests or they suck or something. You don't have to be a TDD acolyte to find unit tests useful or essential.
- overscore 5y ago
  > If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. That is a very normal setup. > Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. This is called flakiness and is generally a symptom not to be ignored, as it is almost always indicative of bigger issues. It's rare that flakiness is limited to test environments. Instead it's much more likely that whatever your smoke tests are experiencing is a something end-users are also intermittently hitting. > The feedback loop becomes so slow that it's not helpful during actual coding. Devs can write their own unit tests when working on their assigned tasks. Smoke tests are designed to run when you're trying to integrate those changes into the existing codebase. At that point, you have the calculus all wrong. Smoke tests slow down devs enough that they don't merge broken code into production. That is a useful release gate unto itself. If unit tests pass but smoke tests fail, then often (the vast majority of the time in my experience) the issue is that either the dev didn't understand the task or, more often, didn't understand the system they were integrating into.
abystander 5y ago
> Personally, I find unit tests to be mostly useless. Meanwhile I've found myself in the position where people eschewed unit testing entirely under this "just do integration tests" thinking and now I'm stuck fumbling around with flaky and hard to run integration tests that take forever to run. Unit tests are good because they test whether the code you've written work as you expect. They run instantly. They don't require a full setup with other integrations. This is indispensable - while it's nice to see a service integrating well and behaving as expected, it's also very useful to know practically instantaneously that the logic you're calling from a controller or something isn't broken. And if it's really tough you can easily put in breakpoints and step through, etc.
everyone 5y ago
I'm a game dev and over time I've settled on using two groups of tests for my projects. Both at opposite ends of the spectrum. . 1. Unit tests. But I only write them for stuff that needs them.. Eg. Some complex math functions that translate between coordinate systems; the point of the unit tests is to confirm that the functions are doing exactly what I think they are doing. With mathsy stuff it can be very easy to look at the output of some function and think, that looks fine, but in reality its actually slightly off, and not exactly what it should be. The unit tests are to confirm that its really doing what I think its doing. . 2.Acceptance tests by a human. Theres a spreadsheet of everything you can do in the game and what should happen. Eg. press this button -> door should open.. As we add features we add more stuff to this list. At regular intervals and before any release several humans try every test on various hardware. This is to catch big / complex bugs and regressions. Its super tedious but it has to be done imo. Automating this would be an insane amount of work and also pointless as we are also testing the hardware, you get weird problems with certain GPUs, gamepads, weird smartphones etc. . I find those two types of tests to be essential, the bare minimum. But also anything in between, like some kind of automated integration testing is just a shittone of work and will only be useful for a relatively brief period of development, changes will quickly render those sort of tests useless.
- dimes 5y ago
  Yes, totally agree. Any code that has complicated logic with few / no dependencies benefits from unit testing.
tablespoon 5y ago
> Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice. I think they're mostly useless when refactoring, but they're useful when writing new code and and making relatively small to medium sized changes. For new code, it's helpful to me at least to express my intentions in a more concrete form and it gives me more confidence that I didn't miss something. For making relatively small changes, they help catch fined-gained regressions. Even if I meant to make a change, a failing test forces me to think about handling a particular case correctly that I might have forgotten. The kind of unit test I do hate are the ones that are so mock-heavy that they're pretty much only testing the structure of your codebase (did you call all the methods the right order and nothing more?). I was once on a team where that was pretty much all they wrote, and they were very resistant to any level of integration unit testing because (I think) they read in a opinionated book somewhere that low level tests were good enough (they weren't).
- 0xabe 5y ago
  When refactoring, unit tests confirm that you did it right (or wrong).
  - h0l0cube 5y ago
    Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. Integration tests and E2E tests are far more valuable because they're attacking it at the business logic side, which is far less volatile, and particularly in a refactor, a useful invariant.
    - tablespoon 5y ago
      > Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. That's not a bad thing, though. DRY might be fine for your main implementation, but redundancy is a time-tested way of catching errors (a.k.a. "double checking").
marcosdumay 5y ago
If you have some code that if its callers changed, they would stop using that code or use it on a different place, it's a unit and it's a good idea to unit test it. If you have some code that if its callers changed you would want to change it too, then it's on the same unit as the calling code, and it's bad to divide it away.

colordrops 5y ago

Maybe I'm missing something but when did continuous delivery imply no human testing? My last two jobs have had what I understand to be continuous delivery, to great success, but perhaps it was something other than CD.

ehutch79 5y ago
What's being sold as TDD/CD/ETC has a lot of hyperbole involved.

sparker72678 5y ago

We deploy to production multiple times per day, with no human intervention, successfully. Our test suite includes some unit and full end to end tests, and yea sometimes a breaking change slips through, we write a test to catch it next time, and move on. (Though our deploy process involves spinning up new instances to run our code, and our load balance won’t switch over to new boxes if they return 500s, so fully broken deploys don’t actually get hit by customers.) Different projects have different complexities and differing levels of CD viability, but there are definitely many people truly doing automated continuous deployment all day every day.

alisonkisk 5y ago

what counts as "C" in CD? FANG push changes to each of many components daily, with feature flags. of course each change goes through stages: dev, preprod, prod over a few days.

fennecfoxen 5y ago

> I genuinely don't get how you can rely primarily on unit tests to prevent regressions in your code. They are at such a fine granularity that most significant changes require restructuring the "units" in a way that invalidates the tests. Besides less-anemic E2E tests, this is why type systems exist. Define a well-typed interface, code to the interface, test the units — and then it's hard to make terrible mistakes wiring things up. To realize the full benefit does, of course, require substantially better data structures than string-to-Any hashes, and it requires some proper old-school software engineering (i.e. injecting a client as an argument and not just mocking out the someOtherClient.client() method.) And there is some overhead, though I'd say that a proper null-checking pass alone would be worth that much. (Sorry, Java, your types are terrible.)

deleted 5y ago

[deleted]

throwaway894345 5y ago

I’ve done CD in production. If you have “enough” test automation, you can have enough confidence that significant regressions become rare, and when they do crop up you can find and fix them in minutes-to-hours from finding the bug to releasing a fix to production. It works very well if you can tolerate that some of your users will have (typically minor) issues once in a while. Coupled with blue/green or canary deployments you can move pretty quickly while minimizing the impact of bugs.

ledauphin 5y ago

I don't agree that you can take humans out of the equation, particularly for new code. but functional programming makes unit tests the natural way to regression-test the top 80 to 90 percent of your codebase, and those don't have to target tiny pieces - they can target large units.