4 ms·

Why does everyone rethink a working strategy. Write lots of unit tests that are fast. Write a good amount of integration tests that are relatively fast. Write f

Why does everyone rethink a working strategy. Write lots of unit tests that are fast. Write a good amount of integration tests that are relatively fast. Write fewer system integration tests that are slower. The testing pyramid works. He even talks about it in this post, and then ignores the point of it. You write lots of unit tests because you can run them inline pre-commit or in a component build. If you integration tests are numerous and interesting enough, that won't work. They are better suited to gating feature branch merges. System integration (the actual name for "end to end") take longer and usually gate progressively more stable branches upstream, or nightlies depending on where you are.

a-saleh 9y ago
Because for certain kind of project the strategy stops working. I work as QE on a fairly large, ~7 years old project. Micorservice architecture has been attempted. We always merge to master, which means that everything more-or-less is a feature-branch merge. We have too many repositories to count. And what we learned is, that most of the components we have are just too thin to allow for useful unit-test coverage. Almost everything is [Gui]--request->[Middleware]--request-->[Proxy]--request-->[Backend]->[Database]. In reality, [Middleware] and [Backend] probably should have been a single component, but devs wanted to do microservices, and be scalable, but they didn't really understand the bounded contexts of their services. All of this leads us to a place, where unit-tests don't tell us much. On the other hand, we managed to spawn [Middleware]->[Backend]->[Database], and we can run a useful integration tests-suite in ~2 minutes. So, on one hand, if we desined this better, the good-old pyramid might be a working strategy. On the other hand, if I can get actual services running in minute, and test them end-to-end, I don't think I will bother with true unit-tests on my next projects. I.e. why mock the database, if I can spawn it seconds :-)
- miskin 9y ago
  So, if I understand it correctly, Middleware and Backend should have been single component since it's one bounded context and splitting it makes one of those feature envy? Is there some benefit keeping these separate or is the cost of change too high at this point? If it's not about features, but more about API, have you tried Consumer-driven contract testing approach?
  - a-saleh 9y ago
    The reason was, you can have more instances of backend for a single middleware and that should have helped with scalability. If we had the resources to do the refactoring, we would probably end up with two-three different backends for various contexts, and without the middle-man between the gui and the backends. On the other hand, the cost of change is probably too high, and most probably this version of our product will be kept on minimum-resource life support. We are looking for doing consumer-driven testing for our new set of services we are working on.
- throwaway5752 9y ago
  The units tell you tons. They just don't tell you the whole story. Trust me, try complex distributed systems testing in an environment the underlying services themselves are bug-prone and issuing stack traces all over because of poor fencing/bounds checking/et al. You may think that way now regarding mocking the database, but where you will find yourself down the line is trying to devise a functional system integration test case for a slightly esoteric condition (deadlocks, timeouts). It's nice to have the scaffolding of a unit testing framework with robust mocks for those situations. Edit: also, you should consider a gitflow workflow (dev -> integration -> master/stable) and make feature branch off of dev to insulate your master.
crdoconnor 9y ago
>Why does everyone rethink a working strategy. Write lots of unit tests that are fast. * Because I want to avoid writing more code than necessary. * Because I want to avoid writing tightly coupled code. * Because I'd rather have a test that takes 2x longer and catches 5% more bugs. Premature optimization and all that.
- chewbacha 9y ago
  I would recomnend not optimizing for less code. Optimize for reading less code. Unit tests actually tend to favor highly uncoupled code while integration seem to favor more coupling with e2e favoring the most coupling. I believe this is because the higher the level of testing the fewer public interfaces are thought about at lower levels. As for percentages about speed and coverage, that seems like a bad trade off of 5% gain for 100% slow down. Especially because test time compounds.
  - crdoconnor 9y ago
    >I would recomnend not optimizing for less code. That is a terrible recommendation. Unless writing less code comes at the expense of readability or coupling you should always aim to write less code instead of more. >Unit tests actually tend to favor highly uncoupled code while integration seem to favor more coupling with e2e favoring the most coupling. It's the exact opposite. End to end tests do not even necessarily couple to a language, let alone specific modules. They can be used to refactor virtually the entire code base without rewriting any test code. That isn't to say that you should only use E2E tests. IMHO wherever there is a naturally loose coupling and a clean, relatively unchanging interface - that is a good place to cover with integration tests. The worst thing to surround with tests is a module whose API you know you will be changing (which will break the test when you do). >As for percentages about speed and coverage, that seems like a bad trade off of 5% gain for 100% slow down. Especially because test time compounds. No, it's an excellent trade off. CPU time is dirt cheap and bugs are very expensive. Moreover, you can run regression test suites while you eat, sleep and visit the water cooler so the absolute time does not really matter provided it catches bugs before release.
- jononor 9y ago
saas_co_de 9y ago
the point is that blind 100% coverage cargo-cultism is not a working strategy.
- therealdrag0 9y ago
  Does anyone even believe in 100% coverage?
  - throwaway5752 9y ago
    I never said anything about it, personally. It's nice to shoot for if you're greenfield. Line coverage != path coverage and blind adherence to line coverage metrics isn't going to guarantee anything.
sheepmullet 9y ago
> The testing pyramid works. The testing pyramid is built around a lot of assumptions which are often not true. For example I run our ~10,000 integration tests in under two minutes on our large enterprise codebase. In recent years it has become possible to have fast integration tests. I've worked on other apps that take 5+ minutes just to start up and integration tests can take hours. Applying the same testing strategy to both does not make sense.
- throwaway5752 9y ago
  The pyramid isn't a law, it's just a heuristic that says to have more low level tests that are faster than high level tests that are slower. The unit/integration/system integration division tends to be correct, but isn't always. It just reminds us that there is time/compute scarcity, and to maximize those resources for optimal roi. And yes, a mobile app != a PaaS platform != system software and adapt the principles sensibly to the situation. Seriously, though, I salute you on those integration test numbers. I assume containers are involved?

a-saleh 9y ago

Because for certain kind of project the strategy stops working. I work as QE on a fairly large, ~7 years old project. Micorservice architecture has been attempted. We always merge to master, which means that everything more-or-less is a feature-branch merge. We have too many repositories to count. And what we learned is, that most of the components we have are just too thin to allow for useful unit-test coverage. Almost everything is [Gui]--request->[Middleware]--request-->[Proxy]--request-->[Backend]->[Database]. In reality, [Middleware] and [Backend] probably should have been a single component, but devs wanted to do microservices, and be scalable, but they didn't really understand the bounded contexts of their services. All of this leads us to a place, where unit-tests don't tell us much. On the other hand, we managed to spawn [Middleware]->[Backend]->[Database], and we can run a useful integration tests-suite in ~2 minutes. So, on one hand, if we desined this better, the good-old pyramid might be a working strategy. On the other hand, if I can get actual services running in minute, and test them end-to-end, I don't think I will bother with true unit-tests on my next projects. I.e. why mock the database, if I can spawn it seconds :-)

miskin 9y ago
So, if I understand it correctly, Middleware and Backend should have been single component since it's one bounded context and splitting it makes one of those feature envy? Is there some benefit keeping these separate or is the cost of change too high at this point? If it's not about features, but more about API, have you tried Consumer-driven contract testing approach?
- a-saleh 9y ago
  The reason was, you can have more instances of backend for a single middleware and that should have helped with scalability. If we had the resources to do the refactoring, we would probably end up with two-three different backends for various contexts, and without the middle-man between the gui and the backends. On the other hand, the cost of change is probably too high, and most probably this version of our product will be kept on minimum-resource life support. We are looking for doing consumer-driven testing for our new set of services we are working on.
throwaway5752 9y ago
The units tell you tons. They just don't tell you the whole story. Trust me, try complex distributed systems testing in an environment the underlying services themselves are bug-prone and issuing stack traces all over because of poor fencing/bounds checking/et al. You may think that way now regarding mocking the database, but where you will find yourself down the line is trying to devise a functional system integration test case for a slightly esoteric condition (deadlocks, timeouts). It's nice to have the scaffolding of a unit testing framework with robust mocks for those situations. Edit: also, you should consider a gitflow workflow (dev -> integration -> master/stable) and make feature branch off of dev to insulate your master.

crdoconnor 9y ago

>Why does everyone rethink a working strategy. Write lots of unit tests that are fast. * Because I want to avoid writing more code than necessary. * Because I want to avoid writing tightly coupled code. * Because I'd rather have a test that takes 2x longer and catches 5% more bugs. Premature optimization and all that.

chewbacha 9y ago
I would recomnend not optimizing for less code. Optimize for reading less code. Unit tests actually tend to favor highly uncoupled code while integration seem to favor more coupling with e2e favoring the most coupling. I believe this is because the higher the level of testing the fewer public interfaces are thought about at lower levels. As for percentages about speed and coverage, that seems like a bad trade off of 5% gain for 100% slow down. Especially because test time compounds.
- crdoconnor 9y ago
  >I would recomnend not optimizing for less code. That is a terrible recommendation. Unless writing less code comes at the expense of readability or coupling you should always aim to write less code instead of more. >Unit tests actually tend to favor highly uncoupled code while integration seem to favor more coupling with e2e favoring the most coupling. It's the exact opposite. End to end tests do not even necessarily couple to a language, let alone specific modules. They can be used to refactor virtually the entire code base without rewriting any test code. That isn't to say that you should only use E2E tests. IMHO wherever there is a naturally loose coupling and a clean, relatively unchanging interface - that is a good place to cover with integration tests. The worst thing to surround with tests is a module whose API you know you will be changing (which will break the test when you do). >As for percentages about speed and coverage, that seems like a bad trade off of 5% gain for 100% slow down. Especially because test time compounds. No, it's an excellent trade off. CPU time is dirt cheap and bugs are very expensive. Moreover, you can run regression test suites while you eat, sleep and visit the water cooler so the absolute time does not really matter provided it catches bugs before release.
jononor 9y ago

saas_co_de 9y ago

the point is that blind 100% coverage cargo-cultism is not a working strategy.

therealdrag0 9y ago
Does anyone even believe in 100% coverage?
- throwaway5752 9y ago
  I never said anything about it, personally. It's nice to shoot for if you're greenfield. Line coverage != path coverage and blind adherence to line coverage metrics isn't going to guarantee anything.

sheepmullet 9y ago

> The testing pyramid works. The testing pyramid is built around a lot of assumptions which are often not true. For example I run our ~10,000 integration tests in under two minutes on our large enterprise codebase. In recent years it has become possible to have fast integration tests. I've worked on other apps that take 5+ minutes just to start up and integration tests can take hours. Applying the same testing strategy to both does not make sense.

throwaway5752 9y ago
The pyramid isn't a law, it's just a heuristic that says to have more low level tests that are faster than high level tests that are slower. The unit/integration/system integration division tends to be correct, but isn't always. It just reminds us that there is time/compute scarcity, and to maximize those resources for optimal roi. And yes, a mobile app != a PaaS platform != system software and adapt the principles sensibly to the situation. Seriously, though, I salute you on those integration test numbers. I assume containers are involved?