8 ms·
The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers
by hinkley 5y ago
The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them.
It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth.
Often E2E tests have such sunk costs involved that they materially affect the project roadmap.
- lupire 5y agoGoing the other way, unit tests prevent devs from writing systems that behave correctly. It doesn't help to prove that a module does what you expect, if your expectations are wrong.
- BeetleB 5y agoWhile I agree that writing unit tests are a lot harder, and you develop good skills in attempting to write them, I must say that in the projects I worked on most bugs were caught by integration tests (technically not E2E tests), and not unit tests. I've also had projects with only unit tests, and almost no bugs were found by it, and there were plenty of bugs. Ideally, I would like both. But if I had to have only one, I'd go with tests at a coarser granularity than unit tests.
- josephg 5y agoMy experience is that most bugs are found by randomisers / fuzzers. I’m consistently surprised they aren’t used more often, because they’re insanely good value for the time spent writing them. Eg, a b-tree has a bunch of invariants: Leaves have equal height, data is sorted, nodes have between N/2 and N values, they contain everything that was inserted and not deleted, etc. So write a test which makes random changes to a b-tree in a loop, and makes those same changes to a simple sorted list. Every iteration, verify the invariants hold and values match. Every 1000 iterations, throw out the object and start again with a new seed. If the test ever fails, print out the seed for easy reproducibility. In your unit testing suite, run this fuzzer for about 100ms or something. This catches lots of bugs. And occasionally leave the randomiser running overnight looking for rare bugs. This sort of thing is so humbling, for the sheer volume of “obvious” bugs you find in otherwise working code. It’s hands down the best value testing code I’ve ever written.
- shakezula 5y ago> Often E2E tests have such sunk costs involved that they materially affect the project roadmap. I've seen this first hand. When the e2e tests take hours to run, are flaky on a good day, and are only really understood by one or two people on the whole team, they can be a major roadblock to new features or even just moderate refactors.
- hinkley 5y agoThat definitely happens. The E2E tests tend to make assumptions about how the app works (encode not just the requirements but also the architecture) and some features change the design. In order to add this feature we have to fix dozens of other tests. I’ve seen people on multiple projects team up to fix these, take over a day working together, and still not be done. They always try to tweak the tests but the test assumptions fight them. Meanwhile if we add a feature that invalidates a unit test, you just delete the unit test and start over. Unit tests are cattle, E2E tests are pets.
- hinkley 5y agoI should add as well: after that day, day and a half working together on old tests, those engineers look beaten down. They are not having a good time. It’s miserable work. It must be some sort of Stockholm syndrome that people in this state still defend the tests. Even after they’ve invested more time and energy into fixing them than we ever would just manually testing that part of the code in perpetuity.
- void_mint 5y ago> The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. Either alone is insufficient. Both together aren't necessarily sufficient.
- sidlls 5y ago> The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. I'd view that as a win then. Unit tests are next to worthless for anything but tightly bound domains, like libraries, in most cases. They're actively harmful in things like application-level service code, where they're used to turn perfectly well-written code into a chockablock mess to satisfy "testibility".
- mbrodersen 5y agoUnit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.
- R0b0t1 5y agoYou're both kind of right. You need to E2E your product, and what your product is is what matters.
- hinkley 5y agoYou can’t test all error cases end to end. If you can you have shitty error handling. Clock skew between servers? Drifting clock skew? Disk space exhaustion? Disk space exhaustion at each possible failure point? There are so many of these and you’re going to inject most of them in unit tests. My original point was that if you can’t write good unit tests your e2e tests are also going to be lousy, and you will never get good at either, let alone both, if you fixate on more coverage with E2E tests. They’re also just too damned expensive even if they were qualitatively as good. Which they are not. They are less numerous, sure, but that’s false economy because they are usually 3 orders of magnitude slower.
- mbrodersen 5y agoOf course you can. Part of the end-to-end test is to setup the test scenarios you want to test (including limited HD space etc.)
- hinkley 5y agoYou're going to spin up a vm with the wrong system time, and then advance it between two operations that take 200 ms on a live system? Bullshit.
- mbrodersen 5y agoWhat a BS straw man answer. There are much easier ways to do that kind of testing.