5 ms·

I love integration tests. You know why? Because I can safely refactor all I want! Unit tests are great, but if you significantly refactor how several classes t

I love integration tests. You know why? Because I can safely refactor all I want! Unit tests are great, but if you significantly refactor how several classes talk to each other, and each of those classes had their own, isolated unit tests that mocked out all of the others, you're suddenly refactoring with no tests. But a black box integration tests? Refactor all your code, replace your databases, do whatever you want, integration test still passes. Unit test speed is a huge win, and they're incredibly useful for quickly testing weird little edge cases that are annoying to write integration tests for, but if I can write an integration test for it, I prefer the integration test.

hughesjj 3y ago
Legit. Probably an unpopular opinion but if I had to chose only one type of test (queue a long discussion with no resolution over defining exact taxonomic boundaries), I'd go with integration over unit. Especially if you're a new contributor to a project. I think it comes down to exercising the flow between... Well, integrations across components. Even better? Take your integration test, put it on a cronjob in your VPN/vpc, use real endpoints and make bespoke auth credentials + namespace, and now you have canaries. Canaries are IMHO God tier for whole system observability. Then take your canary, clean it up, and now you have examples for documentation. Unit tests are for me mostly testing domain+codomain of functions and adherence to business logic, but a good type system along with discipline for actually making schemas/POJOs etc instead of just tossing around maps strings and ints everywhere already accomplishes a lot of that (still absolutely needed though!)
- piokoch 3y ago
  Right. Unit tests are typically a waste of time unless you have some complicated business logic (say, some insurance rates calculation, etc.). This was advocated long time ago in the (great) book "Next Generation Java Testing: TestNG and Advanced Concepts" by Cédric Beust and Hani Suleiman (old people will remember his (in)famous The Bile Blog...).
  - Aeolun 3y ago
    Unit tests are fine for testing any kind of logic. Whether you consider the logic important is a different question.
cyptus 3y ago
how do you handle resetting a sql database after every integration test? Testcontainers may help here by spinning up a new instance for every test but that seems very slow
- tomjakubowski 3y ago
  Do the whole test in a transaction and roll it back at the end.
- simonw 3y ago
  If I'm using Django I let Django's default test harness handle that for me - it runs each test in a transaction and rolls it back at the end of the test, which is pretty fast. https://docs.djangoproject.com/en/5.0/topics/testing/overview/ https://docs.djangoproject.com/en/5.0/topics/testing/overvie... For my other projects I'm generally using SQLite where starting a new in-memory database is so fast it's effectively free.
  - dkdbejwi383 3y ago
    How does that work when the system under test uses transactions itself?
    - simonw 3y ago
      A lot of databases these days support nested transactions using savepoints, which Django's test framework can take advantage of. There's a separate mechanism for writing tests where you need to explicitly test transaction mechanics: https://docs.djangoproject.com/en/5.0/topics/testing/tools/#django.test.TransactionTestCase https://docs.djangoproject.com/en/5.0/topics/testing/tools/#...
- flakes 3y ago
  I do this a lot for Postgres testing. In my setup, I create a single database for the entire test run. Each test creates its own schema in that database and applies the latest table definitions. With this setup, I only eat the container creation once, while allowing every test to operate in isolation from one another, be parallelized, and test against a real database. I do a similar trick for S3 containers by applying a unique guid prefix to the buckets in each test.
rollulus 3y ago
Thanks for saying this out loud. I’m a solo dev and in my project I’m doing exactly this: 90% black box integration tests and 10% unit tests for edge cases I cannot trigger otherwise. It buys me precious time to not adjust tests after refactoring. Yet it made me feel like a heretic: everyone knows the testing pyramid and it comes from Google so I must be very wrong.
- fiznool 3y ago
  You might be interested in the ‘testing trophy’ as an alternative to the traditional pyramid. https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests
  - imiric 3y ago
    This advice is so misguided that I'm concerned for our industry it's getting so much traction. > You really want to avoid testing implementation details because it doesn't give you very much confidence that your application is working and it slows you down when refactoring. You should very rarely have to change tests when you refactor code. Unit tests don't need to test implementation details. You could just as well make that mistake with integration or E2E tests. Black box testing is a good practice at all layers. What unit tests do is confirm that the smallest pieces of the system work as expected in isolation. Yes, you should also test them in combination with each other, but it serves you no good if you get a green integration test, when it's likely only testing a small fraction of the functionality of the units themselves. This whole "unit tests slow you down" mentality is incredibly toxic. You know what genuinely slows me down? A suite with hundreds of integration tests, each taking several seconds to run, and depend on external systems. But hey, testcontainers to the rescue, right? Tests shouldn't be a chore, but an integral part of software development. These days I suppose we can offload some of that work to AI, but even that should be done very carefully to ensure that the code is high quality and actually tests what we need. Test code is as important as application code. It's lazy to think otherwise.
    - ric2b 3y ago
      If by "smallest pieces of the system" you mean something like individual classes then you are definitely testing implementation details. Whenever you change a method's parameters in one of those internal classes you'll have unit tests breaking, even though you're just refactoring code. Unit testing at the smallest piece level calcifies the codebase by making refactors much more costly.
imiric 3y ago
I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. Sure, integration tests "save" you from writing pesky unit tests, and changing them frequently after every refactor. But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. Unit tests give you that ability. If written correctly, they should be the first to fail (which is a good thing!), and if an integration test fails, it should ideally also be accompanied by at least one unit test failure. This way it immediately pinpoints the root cause. The higher up the stack you test, the harder it is to debug. With E2E tests you're essentially debugging the entire system, which is why we don't exclusively write E2E tests, even though they're very useful. To me the traditional test pyramid is still the best way to think about tests. Tests shouldn't be an afterthought or a chore. Maintaining a comprehensive and effective test suite takes as much hard work as, if not more than, maintaining the application itself, and it should test all layers of the system. But if you do have that, it gives you superpowers to safely and reliably work on any part of the system.
- hamandcheese 3y ago
  > But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. As long as the error is reproducible, never in my career have I had a hard time locating the source of the error. Bisection does wonders (as a general concept, not specifically referring to git bisect). That said, I have encountered plenty of non-reproducible test failures. Moral of the story: make things reproducible, especially tests. Easier said than done.
- mk89 3y ago
  > I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. It's very simple: most of the time people are told by management that they MUST achieve a 80-90-95% of code coverage (with unit tests), which leads to a lot of absolutely worthless tests - tests for the sake of it. The irony is that the pieces that really count don't get tested properly, because you unit-test the happy-path and maybe 1 or 2 negative scenarios, and that's it, missing out a bunch of potential regressions. EDIT: This just to say that I don't believe the author of the comment said "don't write unit tests" (I hope not, at least!) but, if I can rephrase it, "well, the integration tests give you a better dopamine effect because they actually help you catch bugs". Which would be partially true also with properly written unit tests (and they would do so in a fraction of the time you need with integration tests).

hughesjj 3y ago

Legit. Probably an unpopular opinion but if I had to chose only one type of test (queue a long discussion with no resolution over defining exact taxonomic boundaries), I'd go with integration over unit. Especially if you're a new contributor to a project. I think it comes down to exercising the flow between... Well, integrations across components. Even better? Take your integration test, put it on a cronjob in your VPN/vpc, use real endpoints and make bespoke auth credentials + namespace, and now you have canaries. Canaries are IMHO God tier for whole system observability. Then take your canary, clean it up, and now you have examples for documentation. Unit tests are for me mostly testing domain+codomain of functions and adherence to business logic, but a good type system along with discipline for actually making schemas/POJOs etc instead of just tossing around maps strings and ints everywhere already accomplishes a lot of that (still absolutely needed though!)

piokoch 3y ago
Right. Unit tests are typically a waste of time unless you have some complicated business logic (say, some insurance rates calculation, etc.). This was advocated long time ago in the (great) book "Next Generation Java Testing: TestNG and Advanced Concepts" by Cédric Beust and Hani Suleiman (old people will remember his (in)famous The Bile Blog...).
- Aeolun 3y ago
  Unit tests are fine for testing any kind of logic. Whether you consider the logic important is a different question.

cyptus 3y ago

how do you handle resetting a sql database after every integration test? Testcontainers may help here by spinning up a new instance for every test but that seems very slow

tomjakubowski 3y ago
Do the whole test in a transaction and roll it back at the end.
simonw 3y ago
If I'm using Django I let Django's default test harness handle that for me - it runs each test in a transaction and rolls it back at the end of the test, which is pretty fast. https://docs.djangoproject.com/en/5.0/topics/testing/overview/ https://docs.djangoproject.com/en/5.0/topics/testing/overvie... For my other projects I'm generally using SQLite where starting a new in-memory database is so fast it's effectively free.
- dkdbejwi383 3y ago
  How does that work when the system under test uses transactions itself?
  - simonw 3y ago
    A lot of databases these days support nested transactions using savepoints, which Django's test framework can take advantage of. There's a separate mechanism for writing tests where you need to explicitly test transaction mechanics: https://docs.djangoproject.com/en/5.0/topics/testing/tools/#django.test.TransactionTestCase https://docs.djangoproject.com/en/5.0/topics/testing/tools/#...
flakes 3y ago
I do this a lot for Postgres testing. In my setup, I create a single database for the entire test run. Each test creates its own schema in that database and applies the latest table definitions. With this setup, I only eat the container creation once, while allowing every test to operate in isolation from one another, be parallelized, and test against a real database. I do a similar trick for S3 containers by applying a unique guid prefix to the buckets in each test.

rollulus 3y ago

Thanks for saying this out loud. I’m a solo dev and in my project I’m doing exactly this: 90% black box integration tests and 10% unit tests for edge cases I cannot trigger otherwise. It buys me precious time to not adjust tests after refactoring. Yet it made me feel like a heretic: everyone knows the testing pyramid and it comes from Google so I must be very wrong.

fiznool 3y ago
You might be interested in the ‘testing trophy’ as an alternative to the traditional pyramid. https://kentcdodds.com/blog/write-tests https://kentcdodds.com/blog/write-tests
- imiric 3y ago
  This advice is so misguided that I'm concerned for our industry it's getting so much traction. > You really want to avoid testing implementation details because it doesn't give you very much confidence that your application is working and it slows you down when refactoring. You should very rarely have to change tests when you refactor code. Unit tests don't need to test implementation details. You could just as well make that mistake with integration or E2E tests. Black box testing is a good practice at all layers. What unit tests do is confirm that the smallest pieces of the system work as expected in isolation. Yes, you should also test them in combination with each other, but it serves you no good if you get a green integration test, when it's likely only testing a small fraction of the functionality of the units themselves. This whole "unit tests slow you down" mentality is incredibly toxic. You know what genuinely slows me down? A suite with hundreds of integration tests, each taking several seconds to run, and depend on external systems. But hey, testcontainers to the rescue, right? Tests shouldn't be a chore, but an integral part of software development. These days I suppose we can offload some of that work to AI, but even that should be done very carefully to ensure that the code is high quality and actually tests what we need. Test code is as important as application code. It's lazy to think otherwise.
  - ric2b 3y ago
    If by "smallest pieces of the system" you mean something like individual classes then you are definitely testing implementation details. Whenever you change a method's parameters in one of those internal classes you'll have unit tests breaking, even though you're just refactoring code. Unit testing at the smallest piece level calcifies the codebase by making refactors much more costly.

imiric 3y ago

I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. Sure, integration tests "save" you from writing pesky unit tests, and changing them frequently after every refactor. But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. Unit tests give you that ability. If written correctly, they should be the first to fail (which is a good thing!), and if an integration test fails, it should ideally also be accompanied by at least one unit test failure. This way it immediately pinpoints the root cause. The higher up the stack you test, the harder it is to debug. With E2E tests you're essentially debugging the entire system, which is why we don't exclusively write E2E tests, even though they're very useful. To me the traditional test pyramid is still the best way to think about tests. Tests shouldn't be an afterthought or a chore. Maintaining a comprehensive and effective test suite takes as much hard work as, if not more than, maintaining the application itself, and it should test all layers of the system. But if you do have that, it gives you superpowers to safely and reliably work on any part of the system.

hamandcheese 3y ago
> But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. As long as the error is reproducible, never in my career have I had a hard time locating the source of the error. Bisection does wonders (as a general concept, not specifically referring to git bisect). That said, I have encountered plenty of non-reproducible test failures. Moral of the story: make things reproducible, especially tests. Easier said than done.
mk89 3y ago
> I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. It's very simple: most of the time people are told by management that they MUST achieve a 80-90-95% of code coverage (with unit tests), which leads to a lot of absolutely worthless tests - tests for the sake of it. The irony is that the pieces that really count don't get tested properly, because you unit-test the happy-path and maybe 1 or 2 negative scenarios, and that's it, missing out a bunch of potential regressions. EDIT: This just to say that I don't believe the author of the comment said "don't write unit tests" (I hope not, at least!) but, if I can rephrase it, "well, the integration tests give you a better dopamine effect because they actually help you catch bugs". Which would be partially true also with properly written unit tests (and they would do so in a fraction of the time you need with integration tests).