4 ms·
Not sure how I hadn't encountered this before, I LOVE this pattern. I find integration tests that exercise actual databases/Elasticsearch/Redis/Varnish etc to
by simonw 3y ago
Not sure how I hadn't encountered this before, I LOVE this pattern.
I find integration tests that exercise actual databases/Elasticsearch/Redis/Varnish etc to be massively more valuable than traditional unit tests. In the past I've gone to pretty deep lengths to do things like spin up a new Elasticsearch index for the duration of a test suite and spin it down again at the end.
It looks like Testcontainers does all of that work for me.
My testing strategy is to have as much of my application's functionality covered by proper end-to-end integration-style tests as possible - think tests that simulate an incoming HTTP request and then run assertions against the response (and increasingly Playwright-powered browser automation tests for anything with heavy JavaScript).
I'll use unit tests sparingly, just for the bits of my code that have very clear input/output pairs that afford unit testing.
I only use mocks for things that I don't have any chance of controlling - calls to external APIs for example, where I can't control if the API provider will be flaky or not.
- trevor-e 3y agoDo you find this results in less overall test code to maintain since you likely have fewer but higher quality/signal tests?
- simonw 3y agoYeah - I find that sticking to tests like this means I don't have hundreds of tiny unit tests that rely on mocks, and it's still very supportive of refactoring - I can make some pretty big changes and be confident that I've not broken anything because a given request continues to return the expected response.
- gorjusborg 3y agoThe choice isn't unit tests vs . end-to-end tests, its between testing things you don't really care about and those you do. You care about real use cases and verifying design constraints are met. You don't care about internal implementation details. The nuance is that there are often things one cares about at multiple levels.
- stingraycharles 3y agoYes, you just focus on a few high level behaviors that you want to validate, instead of the units. It’s more difficult to pull these tests off, as there are more chances for them to become flaky tests, but if they work they provide much more value. I’d prefer a dozen well written integration tests over a hundred unit tests. Having said that, both solve different problems, ideally you have both. But when time-constrained, I always focus on integration tests with actual services underneath.
- DanHulton 3y agoThis is exactly the strategy I have discovered to bring the most value as well. And honestly, something that simplifies the setup of those containers is pretty great.
- briHass 3y agoAnother technique I've found very useful is generative integration tests (kind of like fuzzing), especially for idempotent API endpoints (GETs). For example, assuming you have a test database with realistic data (or scrubbed production data), write tests that are based on generalizable business rules, e.g: the total line of an 'invoice' GET response should be the sum of all the 'sections' endpoint responses tied to that invoice id. Then, just have a process that runs before the tests create a bunch of test cases (invoice IDs to try), randomly selected from all the IDs in the database. Limit the number of cases to something reasonable for total test duration. As one would expect, overly tight assertions can often lead to many false positives, but really tough edge cases hidden in diverse/unexpected data (null refs) can be found that usually escape the artificial or 'happy path' pre-selected cases.
- mzi 3y agoRunning unit tests as integration tests will explode in your face. In any decent complex code base testing time will go through the roof and you will have a hard time getting the genie back in the bottle. Testing that you actually run "sum()" is a unit test.
- CobrastanJorji 3y agoI love integration tests. You know why? Because I can safely refactor all I want! Unit tests are great, but if you significantly refactor how several classes talk to each other, and each of those classes had their own, isolated unit tests that mocked out all of the others, you're suddenly refactoring with no tests. But a black box integration tests? Refactor all your code, replace your databases, do whatever you want, integration test still passes. Unit test speed is a huge win, and they're incredibly useful for quickly testing weird little edge cases that are annoying to write integration tests for, but if I can write an integration test for it, I prefer the integration test.
- hughesjj 3y agoLegit. Probably an unpopular opinion but if I had to chose only one type of test (queue a long discussion with no resolution over defining exact taxonomic boundaries), I'd go with integration over unit. Especially if you're a new contributor to a project. I think it comes down to exercising the flow between... Well, integrations across components. Even better? Take your integration test, put it on a cronjob in your VPN/vpc, use real endpoints and make bespoke auth credentials + namespace, and now you have canaries. Canaries are IMHO God tier for whole system observability. Then take your canary, clean it up, and now you have examples for documentation. Unit tests are for me mostly testing domain+codomain of functions and adherence to business logic, but a good type system along with discipline for actually making schemas/POJOs etc instead of just tossing around maps strings and ints everywhere already accomplishes a lot of that (still absolutely needed though!)
- piokoch 3y agoRight. Unit tests are typically a waste of time unless you have some complicated business logic (say, some insurance rates calculation, etc.). This was advocated long time ago in the (great) book "Next Generation Java Testing: TestNG and Advanced Concepts" by Cédric Beust and Hani Suleiman (old people will remember his (in)famous The Bile Blog...).
- Aeolun 3y ago
- theonething 3y agoI once failed a take home assignment because of this. It was writing a couple of api endpoints and for testing, I focused on integration over unit. I even explained my reasoning in the writeup. There was no indication that the company preferred unit tests, but the feedback was "didn't have enough unit tests". What a dumb company.