5 ms·

I rewrote the backend on a team I used to work on. The service had a ton of unit tests. Given that this was a full rewrite, those unit test were useless. I spen

by dimes 5y ago

I rewrote the backend on a team I used to work on. The service had a ton of unit tests. Given that this was a full rewrite, those unit test were useless. I spent the first few days writing a comprehensive suite of integration tests I could run against the existing service. These tests directly mimicked client calls, so the same tests should be just as valid for the rewritten service. Using these tests, I was able to catch 90%+ of potential issues before cutting over to the new service. Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice.

mumblemumble 5y ago
I find that, in this life, you usually get what you pay for, and, compared to other options, unit tests' primary virtue is that they're inexpensive.
- megameter 5y ago
  Unit tests help verify individual components of a system - which makes them top-of-mind for library code. I think the issue with them lies in that most developers aren't shipping libraries, they're shipping integrated systems, so there's no component worth testing. (you can always invent one, but that's just overcomplicating the code). At the same time, it's also genuinely hard to write good, principled tests of integrated systems, harder than it is to code up a thing that kinda-works and then manually debugging it enough to ship. You have to have the system set up to be tested, and feature complexity actively resists this - you fight a losing battle against "YOLO code" that gets the effect at the expense of going around the test paradigm.
thatswrong0 5y ago
I've arrived at this exact same conclusion for frontend work as well. I always go for integration tests first, and only rely on unit tests if hitting some edge case is hard via integration test.
- thatswrong0 5y ago
  And to clarify, if any individual function reaches some arbitrary level of irreducible complexity, then I'll absolutely unit test that. It's kind of a "you know it when you see it" kind of thing.
jniedrauer 5y ago
How does this scale though? If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. As your project expands, the tests will take longer and longer to run. Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. The feedback loop becomes so slow that it's not helpful during actual coding. At best, it's a semi-useful release gate. Is there another way?
- abystander 5y ago
  I've seen this more often than not. The pyramid is a pyramid for a reason, and I'm somewhat skeptical that we just have to throw out unit tests or they suck or something. You don't have to be a TDD acolyte to find unit tests useful or essential.
- overscore 5y ago
  > If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. That is a very normal setup. > Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. This is called flakiness and is generally a symptom not to be ignored, as it is almost always indicative of bigger issues. It's rare that flakiness is limited to test environments. Instead it's much more likely that whatever your smoke tests are experiencing is a something end-users are also intermittently hitting. > The feedback loop becomes so slow that it's not helpful during actual coding. Devs can write their own unit tests when working on their assigned tasks. Smoke tests are designed to run when you're trying to integrate those changes into the existing codebase. At that point, you have the calculus all wrong. Smoke tests slow down devs enough that they don't merge broken code into production. That is a useful release gate unto itself. If unit tests pass but smoke tests fail, then often (the vast majority of the time in my experience) the issue is that either the dev didn't understand the task or, more often, didn't understand the system they were integrating into.
abystander 5y ago
> Personally, I find unit tests to be mostly useless. Meanwhile I've found myself in the position where people eschewed unit testing entirely under this "just do integration tests" thinking and now I'm stuck fumbling around with flaky and hard to run integration tests that take forever to run. Unit tests are good because they test whether the code you've written work as you expect. They run instantly. They don't require a full setup with other integrations. This is indispensable - while it's nice to see a service integrating well and behaving as expected, it's also very useful to know practically instantaneously that the logic you're calling from a controller or something isn't broken. And if it's really tough you can easily put in breakpoints and step through, etc.
everyone 5y ago
I'm a game dev and over time I've settled on using two groups of tests for my projects. Both at opposite ends of the spectrum. . 1. Unit tests. But I only write them for stuff that needs them.. Eg. Some complex math functions that translate between coordinate systems; the point of the unit tests is to confirm that the functions are doing exactly what I think they are doing. With mathsy stuff it can be very easy to look at the output of some function and think, that looks fine, but in reality its actually slightly off, and not exactly what it should be. The unit tests are to confirm that its really doing what I think its doing. . 2.Acceptance tests by a human. Theres a spreadsheet of everything you can do in the game and what should happen. Eg. press this button -> door should open.. As we add features we add more stuff to this list. At regular intervals and before any release several humans try every test on various hardware. This is to catch big / complex bugs and regressions. Its super tedious but it has to be done imo. Automating this would be an insane amount of work and also pointless as we are also testing the hardware, you get weird problems with certain GPUs, gamepads, weird smartphones etc. . I find those two types of tests to be essential, the bare minimum. But also anything in between, like some kind of automated integration testing is just a shittone of work and will only be useful for a relatively brief period of development, changes will quickly render those sort of tests useless.
- dimes 5y ago
  Yes, totally agree. Any code that has complicated logic with few / no dependencies benefits from unit testing.
tablespoon 5y ago
> Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice. I think they're mostly useless when refactoring, but they're useful when writing new code and and making relatively small to medium sized changes. For new code, it's helpful to me at least to express my intentions in a more concrete form and it gives me more confidence that I didn't miss something. For making relatively small changes, they help catch fined-gained regressions. Even if I meant to make a change, a failing test forces me to think about handling a particular case correctly that I might have forgotten. The kind of unit test I do hate are the ones that are so mock-heavy that they're pretty much only testing the structure of your codebase (did you call all the methods the right order and nothing more?). I was once on a team where that was pretty much all they wrote, and they were very resistant to any level of integration unit testing because (I think) they read in a opinionated book somewhere that low level tests were good enough (they weren't).
- 0xabe 5y ago
  When refactoring, unit tests confirm that you did it right (or wrong).
  - h0l0cube 5y ago
    Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. Integration tests and E2E tests are far more valuable because they're attacking it at the business logic side, which is far less volatile, and particularly in a refactor, a useful invariant.
    - tablespoon 5y ago
      > Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. That's not a bad thing, though. DRY might be fine for your main implementation, but redundancy is a time-tested way of catching errors (a.k.a. "double checking").
marcosdumay 5y ago
If you have some code that if its callers changed, they would stop using that code or use it on a different place, it's a unit and it's a good idea to unit test it. If you have some code that if its callers changed you would want to change it too, then it's on the same unit as the calling code, and it's bad to divide it away.

by dimes 5y ago

mumblemumble 5y ago

I find that, in this life, you usually get what you pay for, and, compared to other options, unit tests' primary virtue is that they're inexpensive.

megameter 5y ago
Unit tests help verify individual components of a system - which makes them top-of-mind for library code. I think the issue with them lies in that most developers aren't shipping libraries, they're shipping integrated systems, so there's no component worth testing. (you can always invent one, but that's just overcomplicating the code). At the same time, it's also genuinely hard to write good, principled tests of integrated systems, harder than it is to code up a thing that kinda-works and then manually debugging it enough to ship. You have to have the system set up to be tested, and feature complexity actively resists this - you fight a losing battle against "YOLO code" that gets the effect at the expense of going around the test paradigm.

thatswrong0 5y ago

I've arrived at this exact same conclusion for frontend work as well. I always go for integration tests first, and only rely on unit tests if hitting some edge case is hard via integration test.

thatswrong0 5y ago
And to clarify, if any individual function reaches some arbitrary level of irreducible complexity, then I'll absolutely unit test that. It's kind of a "you know it when you see it" kind of thing.

jniedrauer 5y ago

How does this scale though? If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. As your project expands, the tests will take longer and longer to run. Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. The feedback loop becomes so slow that it's not helpful during actual coding. At best, it's a semi-useful release gate. Is there another way?

abystander 5y ago
I've seen this more often than not. The pyramid is a pyramid for a reason, and I'm somewhat skeptical that we just have to throw out unit tests or they suck or something. You don't have to be a TDD acolyte to find unit tests useful or essential.
overscore 5y ago
> If you've got integration tests that include state, now you've got to either run your tests serially or set up and tear down multiple copies of the state to prevent tests from clobbering each other. That is a very normal setup. > Worse, they'll start to become unreliable due to the number of operations being performed. So you'll end up with a test suite that takes potentially multiple hours to run, and may periodically fail just because. This is called flakiness and is generally a symptom not to be ignored, as it is almost always indicative of bigger issues. It's rare that flakiness is limited to test environments. Instead it's much more likely that whatever your smoke tests are experiencing is a something end-users are also intermittently hitting. > The feedback loop becomes so slow that it's not helpful during actual coding. Devs can write their own unit tests when working on their assigned tasks. Smoke tests are designed to run when you're trying to integrate those changes into the existing codebase. At that point, you have the calculus all wrong. Smoke tests slow down devs enough that they don't merge broken code into production. That is a useful release gate unto itself. If unit tests pass but smoke tests fail, then often (the vast majority of the time in my experience) the issue is that either the dev didn't understand the task or, more often, didn't understand the system they were integrating into.

abystander 5y ago

> Personally, I find unit tests to be mostly useless. Meanwhile I've found myself in the position where people eschewed unit testing entirely under this "just do integration tests" thinking and now I'm stuck fumbling around with flaky and hard to run integration tests that take forever to run. Unit tests are good because they test whether the code you've written work as you expect. They run instantly. They don't require a full setup with other integrations. This is indispensable - while it's nice to see a service integrating well and behaving as expected, it's also very useful to know practically instantaneously that the logic you're calling from a controller or something isn't broken. And if it's really tough you can easily put in breakpoints and step through, etc.

everyone 5y ago

I'm a game dev and over time I've settled on using two groups of tests for my projects. Both at opposite ends of the spectrum. . 1. Unit tests. But I only write them for stuff that needs them.. Eg. Some complex math functions that translate between coordinate systems; the point of the unit tests is to confirm that the functions are doing exactly what I think they are doing. With mathsy stuff it can be very easy to look at the output of some function and think, that looks fine, but in reality its actually slightly off, and not exactly what it should be. The unit tests are to confirm that its really doing what I think its doing. . 2.Acceptance tests by a human. Theres a spreadsheet of everything you can do in the game and what should happen. Eg. press this button -> door should open.. As we add features we add more stuff to this list. At regular intervals and before any release several humans try every test on various hardware. This is to catch big / complex bugs and regressions. Its super tedious but it has to be done imo. Automating this would be an insane amount of work and also pointless as we are also testing the hardware, you get weird problems with certain GPUs, gamepads, weird smartphones etc. . I find those two types of tests to be essential, the bare minimum. But also anything in between, like some kind of automated integration testing is just a shittone of work and will only be useful for a relatively brief period of development, changes will quickly render those sort of tests useless.

dimes 5y ago
Yes, totally agree. Any code that has complicated logic with few / no dependencies benefits from unit testing.

tablespoon 5y ago

> Personally, I find unit tests to be mostly useless. Every time I touch code with a unit test, I also need to change the unit test. Rather than testing, it feels like writing the same code twice. I think they're mostly useless when refactoring, but they're useful when writing new code and and making relatively small to medium sized changes. For new code, it's helpful to me at least to express my intentions in a more concrete form and it gives me more confidence that I didn't miss something. For making relatively small changes, they help catch fined-gained regressions. Even if I meant to make a change, a failing test forces me to think about handling a particular case correctly that I might have forgotten. The kind of unit test I do hate are the ones that are so mock-heavy that they're pretty much only testing the structure of your codebase (did you call all the methods the right order and nothing more?). I was once on a team where that was pretty much all they wrote, and they were very resistant to any level of integration unit testing because (I think) they read in a opinionated book somewhere that low level tests were good enough (they weren't).

0xabe 5y ago
When refactoring, unit tests confirm that you did it right (or wrong).
- h0l0cube 5y ago
  Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. Integration tests and E2E tests are far more valuable because they're attacking it at the business logic side, which is far less volatile, and particularly in a refactor, a useful invariant.
  - tablespoon 5y ago
    > Except you often need to rewrite them, so now you've got two places (per 'unit') where you could have introduced a bug. That's not a bad thing, though. DRY might be fine for your main implementation, but redundancy is a time-tested way of catching errors (a.k.a. "double checking").

marcosdumay 5y ago

If you have some code that if its callers changed, they would stop using that code or use it on a different place, it's a unit and it's a good idea to unit test it. If you have some code that if its callers changed you would want to change it too, then it's on the same unit as the calling code, and it's bad to divide it away.