4 ms·
I view it differently. I view all my code through end to end tests and use very few mocks. If I need a mocks/fakes for things that aren’t external to the code (
by vlovich123 4y ago
I view it differently. I view all my code through end to end tests and use very few mocks. If I need a mocks/fakes for things that aren’t external to the code (eg I/O to network services), it usually means I haven’t designed my code modular enough and refactor. The reason I view it that way is that all fakes (of which mocks are a subset) means you’re reimplementing your real code with a reimplementation specific to your test suite which means your test suite isn’t providing coverage and now you have two codebase to maintain hidden in one which grows very very difficult and time consuming to do (velocity of any complex refactor massively slows down). The tests themselves I view as a layered thing where each layer of the program has matching tests. So as I add a primitive, I’ll add tests to that piece + how it integrates into the next layer (if there’s observable changes). Sometimes primitives will cross-cut all the way into the end API/UI. In those cases there’s potentially new tests I have to add at every layer to make sure it’s stable all the way down.
This approach has a strong advantage aside from maintaining only one codebase and your tests being much easier to refactor in coordination with code changing along the way in smaller ways. It makes it easier to find and figure out bugs because if it’s not visible at a layer it means either you’re missing tests immediately below that layer OR your integration in that layer is broken. As you move up the layers, it’s often more complicated to debug issues at much lower levels. So if a very high level test fails, you can do a search through lower levels writing reduced test cases that reproduce your observation. That can be a good compliment in addition to reasoning about behavior from reasoning about what the failure might be from symptoms (which often can be faster if you understand your codebase but trickier problems that you can’t figure out and are stuck on can be tackled through a linear search or maybe even bisected search if your clever).
The problem with most tests is that they tend to capture “happy paths” you’ve thought about. Even though negative tests (which are hugely important to make sure to add about how you expect error scenarios / failures to behave) are hugely important, they have a similar characteristic. I like to use property check tests and fuzzing techniques more broadly to improve my coverage when I can (can get expensive in the test suite so it can sometimes be worth balancing testing it on every commit vs post-merge in the background).
Still. It’s a problem. Mutation testing feels like another important tool. I’ve not had success getting Stryker (mutation testing) running on any TypeScript project I maintain which is frustrating (I got it running kind of but it seems to struggle/I haven’t figured out how to generate a proper report because I’m using the workerd version of Miniflare where the code runs out-of-process). But mutation testing as a concept seems like a fantastic way to figure out code coverage and where you might have gaps that traditional testing / fuzzing (property checking is a subset) isn’t catching. We all have blindspots and mutation testing is a far more defensible position that your test coverage is good vs traditional techniques we’re taught in school like branch and line coverage (which in my opinion are extremely poor and misleading ways to measure code coverage).