9 ms·

The testing pyramid should look more like a crab

44 points by jerodsanto 6y ago

jjgreen 6y ago
Everyone wants to be a crab https://www.popularmechanics.com/science/a34389129/crab-evolution-carcinization/ https://www.popularmechanics.com/science/a34389129/crab-evol...
mcv 6y ago
> "And when we look at what Cypress allows you to do (which is write many useful tests that have very little flake) then you wanna write more end-to-end tests." Very little flake? That's not my experience with Cypress. The Cypress e2e tests on our build server sometimes fail for no discernible reason. Then you run them locally, and different tests fail, sometimes for understandable reasons (but then why didn't they fail on the server?) or sometimes also for no clear reason. I'm spending a lot of time maintaining and fixing our Cypress tests, and the work they add seems far worse than the help they provide. Most of the time, a failing e2e test is not a problem with the code, but a problem with the test. Honestly, I find myself longing for the days of Protractor. Horrible to set up perhaps, but in my memory of it, it was reliable once you got it working.
- bluntfang 6y ago
  I've found 2 reasons that e2e tests are flakey (whether it be selenium, puppeteer or cypress): 1. poorly developed tests. There are nuances that go into writing browser interaction tests that the less-experienced will have to work through. 2. There's a weird functionality implemented in such a way that makes testing difficult. This could be something like a modal that has a fly-in animation and if you try to click it while it's animated it won't fire the onClick action. If you want e2e tests to be successful, you have to write testable front end code, which is pretty dang hard for a lot of reasons.
- wotamRobin 6y ago
  I’ve seen the same at several different companies. There seems to be just too much that can go wrong during an e2e test like this: network failures, slow CPU causing timeouts, etc. At one company, the entire suite of 700 tests was re-run 3 times nightly, taking 5+ hours, and some tests still failed randomly. I have found that the majority of errors can be caught by unit and integration tests on a single service, and that the extra burden of maintaining Cypress just isn’t worth the small amount of coverage.
- Vinnl 6y ago
  As I remember it Protractor was just a wrapper around WebDriver with some glue code to wait for Angular doing its thing after data changes. If you're using e.g. React or anything else that does not do dirty checking, you should still be able to use WebDriver. (That said, that's certainly still flaky enough.)
  - Dylanlacey 6y ago
    The cool part was that all of the extra JS Protractor used (sometimes for native WebDriver tasks) made things _less_ reliable.
- vosper 6y ago
  > The Cypress e2e tests on our build server sometimes fail for no discernible reason. You'll probably be interested to hear that Cypress are... uhm... addressing this by adding "flaky test detection" and some amount of automatic retrying in v5. We use Cypress at work, we have flaky test problems, but I'm not convinced much of it is actually Cypress's fault. As others have mentioned, writing front-end code to be testable by E2E is not necessarily straightforward, and when you're trying to add tests to a big existing codebase I think you kinda have to accept some flakiness and retrying. What I've been wondering about recently though is: if you construct applications in a certain way do you still need real E2E tests? For example, React is declarative, and with a state-management system like Redux shouldn't you be able to get the same kind of assurance that E2E gets you without needing an actual browser? Maybe things like animations, visibility etc are where that comes unstuck?
raziel2p 6y ago
> Well, the unit tests can find you a few logical errors, which is great. I write unit tests for that all the time. But all possible sources of error are discovered by end-to-end tests. Citation needed, sounds anecdotal. If this really is how your app behaves then you're going to run into problems eventually because end-to-end tests are slow as well as difficult to write+maintain, and you'll need hundreds (if not thousands) of them to cover all the important business logic in your app. This will slow down your developer productivity (PRs take long to merge because people wait for CI to finish) and ability to deploy quickly. If your code is well-structured, your important business logic should be easily testable with unit and/or integration tests, and you don't need an end-to-end test to discover that, for example, a calculation is wrong. The pattern of functional core/imperative shell embodies this really well, I think.
- deleted 6y ago
  [deleted]
- musingsole 6y ago
  > end-to-end tests are slow as well as difficult to write+maintain Citation needed, sounds anecdotal. End-to-end tests should run as fast as any given use case (for most software, that means relatively quick), plus some set-up time as testing overhead. But that testing overhead is often self-inflicted and can be managed. I have anecdotes to prove it. For the speed to write end-to-end tests, shouldn't they be the easiest to capture? I need the system inputs and I need to capture the system outputs. If you imagine usage-driven development, then the last step is to capture your initial state, use your new code, capture the final state and voila, here's your new end-to-end test. > your important business logic Consider the reality that no one cares about your backend. No one. Is it correct enough for the application is the only business question concerned with the backend (and most applications have a HIGH fault tolerance). What does everyone care about? Your frontend. The colors, dropdown box or search field. That column should be over here. Focusing your testing on the parts of your application people could shrug about is not sound reasoning.
  - Tainnor 6y ago
    I'm pretty sure more people care about the VAT being calculated correctly than about whether it's displayed on the left or on the right. You're taking a reasonable point of view - code matters only inasmuch as it has an effect on users - and take it to an absurd extreme.
jjk166 6y ago
Carcinisation truly knows no limits
r0s 6y ago
End to end tests are less precise to triage when they break, and they break more often than unit tests. As the author said, they show all bugs, so they break a lot. When the test is broken, it's not providing coverage until it's fixed. Which is why I advocate for less assertive testing in that case: https://assertless.org/ https://assertless.org/ The priority of fixing end to end tests becomes critical, and so test maintenance is more important. With a large test suite of functional tests, that effort is not sustainable in my experience. I've been there. An image comparison suite of tests has the same issue, except it may defer all assertions, which is good, but it offloads a huge amount of analysis to the engineer consuming all the output. You really do need a human to check those tiny style changes it finds, a human with extensive knowledge of the product and it's ever-changing visual quirks. I also maintain a large visual testing suite like that. Cypress is not different from Selenium in that aspect. (There's not much real difference at all that I can see.) I think end to end tests have more of a value than the traditional testing pyramid would indicate, but there's pitfalls and benefits to each type. A suite of concise unit tests will find obvious bugs with precise granularity incredibly quick on the code they cover, which can never be complete coverage, but it doesn't need to be. This can be used for things like commit hooks that end to end tests just can't.
- marcosdumay 6y ago
  Hum... Depending on exactly what you test, it's not only the coverage that increases with a larger scope, but the tests also break less. If you engineers things for testing¹, e2e tests need much less maintenance than unit tests. Yes, their failure is also more severe, but overall, they are a large gain on that area. But then, you may discover that your current project just can not be adapted to enable e2e tests, like some projects can not be adapted to enable unity or integration tests. The single size preaching is completely flawed. 1 - Not on the ways the unity testing fans uses that phrase obviously, but if you keep your (human and machine) interfaces stable and generic enough to survive some software evolution.
- username90 6y ago
  > and they break more often than unit tests. That isn't my experience, unit tests breaks all the time and has to be rewritten from simple refactoring while larger tests pass as long as you keep behavior the same. This means that the bigger tests are way better for refactoring and safely making changes to your code. If most of your tests are unit tests and you don't have bigger tests covering the same things then refactoring safely is hell.
lxe 6y ago
Don't do 'unit tests' at all. Your sweet spot in testing should be the fastest possible 'semi-integrated' (real clients, faked services) tests that can be done reproducibly and hermetically, testing specific end-goal functionality.
- ozim 6y ago
  I would agree because I see my devs writing unit tests for plumbing code that does not have any logic. I am annoyed by that. So I am 'party pooper' when it comes to unit testing. But I would like to have unit tests for something that has calculations or non trivial logic only. Other thing I would really like 'risk based' approach, like what is the worst thing that can happen with that code. If it can drop half of the records in table, yes do checks and unit testing. If it is going to be something in the interface not displayed or displayed incorrectly then meh.
  - jniedrauer 6y ago
    There is some value in unit tests for plumbing. Something I see a lot is a set of values being transformed across several domains, losing data at each step. A very simple test that inputs values on one end of the pipe, and checks the values on the other end of the pipe, makes it very easy to be sure that you didn't drop something.
    - ozim 6y ago
      If you transform values, then it has logic, so it does not fit in my definition of simple plumbing. If you check that integer value AgencyId is passed to another object as integer value with the same name of AgencyId that is basic plumbing that should not be covered by unit test. Of course someone might forget to wire that up somewhere, but it is not something I would say writing test and then maintaining it costs more than just fixing it when you find it and once it is fixed there is no real reason to keep regression on it.
- jniedrauer 6y ago
  > real clients, faked services I'm curious what you think this looks like in practice. I've had success with these kinds of tests, but only for very simple external services. For example, writing a fake elasticsearch backend for a few known inputs and outputs might be a good fit. But what about an RDBMS? You can't realistically write a mock backend that can talk to a real client.
bmohlenhoff 6y ago
Don't forget that generally e2e tests take longer to run and typically require more test assets. For example, if e2e tests involve simulations running in real time, then by increasing the number of distinct e2e tests you also increase the requirement on test assets and/or overall CI duration.
remote_phone 6y ago
This is why humans need to test end-to-end when there is a UI component. Trying to automate tests on a GUI is the most torturous aspect of software with no long term success unless your GUI is primitive and never changes. It’s easier to just send it off to a low-cost center and have them hand test various scenarios and create a report.
hinkley 6y ago
By moving the 'meat' of the tests up high into the hierarchy, the author has just re-invented the testing ice cream cone with a different flavor. In software, we can make pretty much any process work for 18 months, before it starts to fall apart. If you don't stay at a company for at least a couple of years after they start doing testing in earnest, you won't really see that what you're doing doesn't scale/isn't resilient to changing requirements. I have watched so many people try to rescue deeply coupled integration or E2E tests and it's just painful to watch. It's a deadly cocktail of cargo culting and Sunk Cost Fallacy - we don't know for sure all the corner cases these tests cover, so we aren't going to delete them and lack the confidence to rewrite, so we'll spend all day trying to fix them, and if that doesn't work, we'll pair with someone for day 2 to get it fixed. That's 3 man days, for a handful of tests. I've seen it many times, on different teams, rarely are there enough other people noticing how crazy this is to stage an intervention. It's crazy. A contributory reason to why fixing such tests takes so long is that they're so slow. Slow tasks have poor feedback loops. Testing, at least when done as part of CI/CD, is meant to provide fast feedback, and E2E tests fail at this (most especially at 18 months and beyond, where your E2E test is one of hundreds). There's a physics and a psychology to the tiers in the testing pyramid that I meant to write up publicly but I don't think ever escaped a corporate wiki. Here are the Cliff's Notes, based on my own metrics but corroborated by the handful of people who've inspired my testing journey: 1) Moving tests down a tier reduces the power of the test, so you need more of them (about 5x) 2) Moving tests down a tier makes them much faster. (8x common, 10x best case, depending on framework) 3) The simplest tier of tests will be rewritten or deleted when requirements change. All other tests will be 'rescued', sometimes at great expense in time and energy. Rules 1 & 2 create a pseudo-rule, the 5/8ths Rule. If you move a functional test to units, the same coverage will run about 30% faster in aggregate, and you will not have to provide ongoing support for those tests. That's a huge win. If you pull a test down 2 levels, they'll run 60-75% faster.
- hinkley 6y ago
  This got so long I decided to split it in two: It is important to note you can't pull all of the tests down. I tend to use a plumbing analogy that I probably have stolen from the literature or a mentor - It's likely that QA already tested all of the pipes at the factory before they were installed in your house. But every plumber will turn on the taps and flush the toilet at once before they leave, to make sure the stuff going in one end actually comes out the other (and only the other). Test all the bits in isolation, but always leave at least one happy and one unhappy path test going through the whole stack to make sure good and bad answers end up in the right place. I think here the pyramid analogy is a bit wrong, because in the pyramid there's a geometric reduction at each level, but I have a suspicion that in a best case scenario, you can achieve a log scale reduction (because your E2E tests can potentially reflect the height of the decision tree, rather than the surface area of a module) Tests higher up the pyramid rarely if ever suggest, let alone demand API improvements. If the first person writes elaborate mocks instead of simplifying the design to not need them, they've locked in the implementation in a way that unit tests don't (Sunk Cost). This can turn a masochistic programmer into a sadist. And that is the trap most of us fall into the first few times, and some never outgrow. It's easier to write greenfield E2E tests, but they're hell to maintain, and the create a codependent testing relationship, where the existence of the tests dissuades you from engaging in healthier activities. I almost think that you should ban E2E testing frameworks until the team has gotten comfortable with Unit Tests because of this. With E2E, you can get really far down the wrong road and find yourself in a blind alley that you now lack the imagination or conviction to escape, because everything else feels remedial. That is the main problem with unit tests - they feel like some sort of grade school activity. Doing something so simple feels like doing multiplication tables, which is kids work. I am a smart and functional adult, I shouldn't be doing something so boring. This is making me feel dumb. It takes a special disposition to enjoy a simple task for what it is, and many software developers - myself included - were lured into software by the idea of automating any task that is so simple. Rarely do I let production code be so dominated by idioms with so little meat, so why do I enjoy that in tests? Honestly, I don't have an answer, except "better in the tests than in the production code," which is hardly an answer at all. And importantly, all of these uneasy feelings are used as evidence by anti-testing people that it's too painful and too little gain and we should stop doing it. A form of learned helplessness.
adenta 6y ago
I write a bunch of e2e tests, and not many unit tests. I'm on the fence about if this is a good idea or not. I feel like when you are starting at zero, you need more e2e tests, before writing more unit/service tests as your product is solidified. I wish e2e tests were faster. I feel like this should be achievable, but it's a problem way out of my depths. I feel like you could run a browser within a docker container at 2x or 5x speed with enough virtualization. I've been told this would not work in practice.
- Hokusai 6y ago
  e2e testing is the way to go. But our current tech stack is really bad for testing. Testing in browsers seems an afterthought, browsers first were designed for features, later on debugging was added, quite good nowadays, but testing is out of the scope. Until we get easy to test browsers everything will continue being fragile and bug prone.
MengerSponge 6y ago
Science Twitter: "Everything is crab." HN two weeks later: "What if testing were crab?"
- luhn 6y ago
  For those not on Twitter, this went viral a couple weeks ago: https://en.wikipedia.org/wiki/Carcinisation https://en.wikipedia.org/wiki/Carcinisation
tmcneal 6y ago
I'm co-founder of an end-to-end testing product, but I don't actually agree that better end-to-end testing coverage means you should create less unit tests. The reason why is because there's a class of things you want to test that are much better served at the unit testing level. Unit tests are always going to be faster to execute and by definition have a much smaller scope, which is helpful in reasoning about what you're testing. Our product is really focused on making end-to-end tests much easier to create and maintain. Here's an example video I recorded yesterday of automating Monday.com's sign-up flow using our product: https://www.loom.com/share/1336a5bfa9f54f2190269961991802b5 https://www.loom.com/share/1336a5bfa9f54f2190269961991802b5 - looking at this I don't really see what unit tests this would replace, rather I think it gives coverage for things that you'd often have no automated coverage for - i.e. the interplay between disparate components along with interacting with email - basically the steps that form a complete workflow from the user's perspective.
- dpix 6y ago
  The author talks about how many other sources of bug there might be outside of your code like integration, assumptions, compilers etc that e2e tests will help catch. Well at least a large portion of those bugs might be able to be caught by unit tests! No reason to write less of those, having good unit and integration tests still make you more confident about your application. And because they are faster to run the feedback loop when they break is also faster
bluGill 6y ago
I prefer to think of the vertical dimension of the testing pyramid as time not test type. Nobody can agree on what the different between a unit test and an integration test is, much less end to end. Time is objective though. At the bottom of my pyramid I have the tests that I always run - even if there is no way they could break (I just changed the IP network stack, there is no way the add() function could break - but I'll run that test anyway. Then I have the tests that run pretty fast, but I find it worth some effort to add some dependency analysis to ensure I only run them when it is possible my code broke them. Next there are tests that are slow, so people working in the area it tests run, but everybody else disables them. These tests are just useful enough that the group that runs them doesn't stop running them, but not useful enough for any other group working on a different area of the code. Then there are tests that a very very slow, so I only run them when I am specifically worried I might have broke something that the test covers (typically never because you forget about them when you should, but there are just enough exceptions to get a mention). Last are tests that I only look at after they have failed 3 times in a row on the CI system, and then I'll spend a week wildly/blindly throwing possible fixes at the CI system before I break down and run them myself. Note that last two are for the CI system to run, not humans. In every case I've seen they take long enough that you have to combine the changes of several people before running them just so the CI system can keep up. They are still valuable to have because some important things cannot be tested any other way. However where possible you should find a different way to test that functionality.
CivBase 6y ago
This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a machine works, it doesn't mean the machine itself works. That's where integration tests come in. Turns out it's usually fairly easy to make sure all the parts work together once you've verified all of the parts individually, so you usually need fewer integration tests. Once you've verified all the parts work and that they work together correctly, now all you need to do is make sure you actually made the right thing. Enter functional tests and E2E tests. These are usually especially costly to maintain, but they're also so high-level that you usually need very few of them. Writing tests is easy, but they're only useful if you run them. It's not enough for a test to work when you write it - it has to continue working into the future. Maintenance is a recurring cost and, in my experience, is where automated testing can get really expensive. The testing pyramid helps minimize that cost. In an ideal world, you would maximize testing at every level - but in the real world that's often just too expensive.
- pydry 6y ago
  Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.
  - Tainnor 6y ago
    This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense. However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree structure and I need to transform it into another structure" and stuff like that. It's also worth keeping in mind that "unit testing" doesn't have to mean "class (or method) testing". Sometimes the class is the right level of abstraction but sometimes it can also be a small module of classes working in concert etc. I agree that tests shouldn't simply reflect implementation, but there are functional requirements that can be specified at a lower level.
alkonaut 6y ago
Tests should document behavior and aid in refactoring and maintaining boundaries. E2e tests don’t document modules (functions/types), only the highest interface, and it doesn’t maintain any separation of layers etc. All kinds of tests are needed, but I find the pyramid arises from the fact that there are a lot more low level units than high level behaviors.

44 points by jerodsanto 6y ago

Everyone wants to be a crab https://www.popularmechanics.com/science/a34389129/crab-evolution-carcinization/ https://www.popularmechanics.com/science/a34389129/crab-evol...

mcv 6y ago

> "And when we look at what Cypress allows you to do (which is write many useful tests that have very little flake) then you wanna write more end-to-end tests." Very little flake? That's not my experience with Cypress. The Cypress e2e tests on our build server sometimes fail for no discernible reason. Then you run them locally, and different tests fail, sometimes for understandable reasons (but then why didn't they fail on the server?) or sometimes also for no clear reason. I'm spending a lot of time maintaining and fixing our Cypress tests, and the work they add seems far worse than the help they provide. Most of the time, a failing e2e test is not a problem with the code, but a problem with the test. Honestly, I find myself longing for the days of Protractor. Horrible to set up perhaps, but in my memory of it, it was reliable once you got it working.

bluntfang 6y ago
I've found 2 reasons that e2e tests are flakey (whether it be selenium, puppeteer or cypress): 1. poorly developed tests. There are nuances that go into writing browser interaction tests that the less-experienced will have to work through. 2. There's a weird functionality implemented in such a way that makes testing difficult. This could be something like a modal that has a fly-in animation and if you try to click it while it's animated it won't fire the onClick action. If you want e2e tests to be successful, you have to write testable front end code, which is pretty dang hard for a lot of reasons.
wotamRobin 6y ago
I’ve seen the same at several different companies. There seems to be just too much that can go wrong during an e2e test like this: network failures, slow CPU causing timeouts, etc. At one company, the entire suite of 700 tests was re-run 3 times nightly, taking 5+ hours, and some tests still failed randomly. I have found that the majority of errors can be caught by unit and integration tests on a single service, and that the extra burden of maintaining Cypress just isn’t worth the small amount of coverage.
Vinnl 6y ago
As I remember it Protractor was just a wrapper around WebDriver with some glue code to wait for Angular doing its thing after data changes. If you're using e.g. React or anything else that does not do dirty checking, you should still be able to use WebDriver. (That said, that's certainly still flaky enough.)
- Dylanlacey 6y ago
  The cool part was that all of the extra JS Protractor used (sometimes for native WebDriver tasks) made things _less_ reliable.
vosper 6y ago
> The Cypress e2e tests on our build server sometimes fail for no discernible reason. You'll probably be interested to hear that Cypress are... uhm... addressing this by adding "flaky test detection" and some amount of automatic retrying in v5. We use Cypress at work, we have flaky test problems, but I'm not convinced much of it is actually Cypress's fault. As others have mentioned, writing front-end code to be testable by E2E is not necessarily straightforward, and when you're trying to add tests to a big existing codebase I think you kinda have to accept some flakiness and retrying. What I've been wondering about recently though is: if you construct applications in a certain way do you still need real E2E tests? For example, React is declarative, and with a state-management system like Redux shouldn't you be able to get the same kind of assurance that E2E gets you without needing an actual browser? Maybe things like animations, visibility etc are where that comes unstuck?

raziel2p 6y ago

> Well, the unit tests can find you a few logical errors, which is great. I write unit tests for that all the time. But all possible sources of error are discovered by end-to-end tests. Citation needed, sounds anecdotal. If this really is how your app behaves then you're going to run into problems eventually because end-to-end tests are slow as well as difficult to write+maintain, and you'll need hundreds (if not thousands) of them to cover all the important business logic in your app. This will slow down your developer productivity (PRs take long to merge because people wait for CI to finish) and ability to deploy quickly. If your code is well-structured, your important business logic should be easily testable with unit and/or integration tests, and you don't need an end-to-end test to discover that, for example, a calculation is wrong. The pattern of functional core/imperative shell embodies this really well, I think.

deleted 6y ago
[deleted]
musingsole 6y ago
> end-to-end tests are slow as well as difficult to write+maintain Citation needed, sounds anecdotal. End-to-end tests should run as fast as any given use case (for most software, that means relatively quick), plus some set-up time as testing overhead. But that testing overhead is often self-inflicted and can be managed. I have anecdotes to prove it. For the speed to write end-to-end tests, shouldn't they be the easiest to capture? I need the system inputs and I need to capture the system outputs. If you imagine usage-driven development, then the last step is to capture your initial state, use your new code, capture the final state and voila, here's your new end-to-end test. > your important business logic Consider the reality that no one cares about your backend. No one. Is it correct enough for the application is the only business question concerned with the backend (and most applications have a HIGH fault tolerance). What does everyone care about? Your frontend. The colors, dropdown box or search field. That column should be over here. Focusing your testing on the parts of your application people could shrug about is not sound reasoning.
- Tainnor 6y ago
  I'm pretty sure more people care about the VAT being calculated correctly than about whether it's displayed on the left or on the right. You're taking a reasonable point of view - code matters only inasmuch as it has an effect on users - and take it to an absurd extreme.

jjk166 6y ago

Carcinisation truly knows no limits

r0s 6y ago

End to end tests are less precise to triage when they break, and they break more often than unit tests. As the author said, they show all bugs, so they break a lot. When the test is broken, it's not providing coverage until it's fixed. Which is why I advocate for less assertive testing in that case: https://assertless.org/ https://assertless.org/ The priority of fixing end to end tests becomes critical, and so test maintenance is more important. With a large test suite of functional tests, that effort is not sustainable in my experience. I've been there. An image comparison suite of tests has the same issue, except it may defer all assertions, which is good, but it offloads a huge amount of analysis to the engineer consuming all the output. You really do need a human to check those tiny style changes it finds, a human with extensive knowledge of the product and it's ever-changing visual quirks. I also maintain a large visual testing suite like that. Cypress is not different from Selenium in that aspect. (There's not much real difference at all that I can see.) I think end to end tests have more of a value than the traditional testing pyramid would indicate, but there's pitfalls and benefits to each type. A suite of concise unit tests will find obvious bugs with precise granularity incredibly quick on the code they cover, which can never be complete coverage, but it doesn't need to be. This can be used for things like commit hooks that end to end tests just can't.

marcosdumay 6y ago
Hum... Depending on exactly what you test, it's not only the coverage that increases with a larger scope, but the tests also break less. If you engineers things for testing¹, e2e tests need much less maintenance than unit tests. Yes, their failure is also more severe, but overall, they are a large gain on that area. But then, you may discover that your current project just can not be adapted to enable e2e tests, like some projects can not be adapted to enable unity or integration tests. The single size preaching is completely flawed. 1 - Not on the ways the unity testing fans uses that phrase obviously, but if you keep your (human and machine) interfaces stable and generic enough to survive some software evolution.
username90 6y ago
> and they break more often than unit tests. That isn't my experience, unit tests breaks all the time and has to be rewritten from simple refactoring while larger tests pass as long as you keep behavior the same. This means that the bigger tests are way better for refactoring and safely making changes to your code. If most of your tests are unit tests and you don't have bigger tests covering the same things then refactoring safely is hell.

lxe 6y ago

Don't do 'unit tests' at all. Your sweet spot in testing should be the fastest possible 'semi-integrated' (real clients, faked services) tests that can be done reproducibly and hermetically, testing specific end-goal functionality.

ozim 6y ago
I would agree because I see my devs writing unit tests for plumbing code that does not have any logic. I am annoyed by that. So I am 'party pooper' when it comes to unit testing. But I would like to have unit tests for something that has calculations or non trivial logic only. Other thing I would really like 'risk based' approach, like what is the worst thing that can happen with that code. If it can drop half of the records in table, yes do checks and unit testing. If it is going to be something in the interface not displayed or displayed incorrectly then meh.
- jniedrauer 6y ago
  There is some value in unit tests for plumbing. Something I see a lot is a set of values being transformed across several domains, losing data at each step. A very simple test that inputs values on one end of the pipe, and checks the values on the other end of the pipe, makes it very easy to be sure that you didn't drop something.
  - ozim 6y ago
    If you transform values, then it has logic, so it does not fit in my definition of simple plumbing. If you check that integer value AgencyId is passed to another object as integer value with the same name of AgencyId that is basic plumbing that should not be covered by unit test. Of course someone might forget to wire that up somewhere, but it is not something I would say writing test and then maintaining it costs more than just fixing it when you find it and once it is fixed there is no real reason to keep regression on it.
jniedrauer 6y ago
> real clients, faked services I'm curious what you think this looks like in practice. I've had success with these kinds of tests, but only for very simple external services. For example, writing a fake elasticsearch backend for a few known inputs and outputs might be a good fit. But what about an RDBMS? You can't realistically write a mock backend that can talk to a real client.

bmohlenhoff 6y ago

Don't forget that generally e2e tests take longer to run and typically require more test assets. For example, if e2e tests involve simulations running in real time, then by increasing the number of distinct e2e tests you also increase the requirement on test assets and/or overall CI duration.

remote_phone 6y ago

This is why humans need to test end-to-end when there is a UI component. Trying to automate tests on a GUI is the most torturous aspect of software with no long term success unless your GUI is primitive and never changes. It’s easier to just send it off to a low-cost center and have them hand test various scenarios and create a report.

hinkley 6y ago

By moving the 'meat' of the tests up high into the hierarchy, the author has just re-invented the testing ice cream cone with a different flavor. In software, we can make pretty much any process work for 18 months, before it starts to fall apart. If you don't stay at a company for at least a couple of years after they start doing testing in earnest, you won't really see that what you're doing doesn't scale/isn't resilient to changing requirements. I have watched so many people try to rescue deeply coupled integration or E2E tests and it's just painful to watch. It's a deadly cocktail of cargo culting and Sunk Cost Fallacy - we don't know for sure all the corner cases these tests cover, so we aren't going to delete them and lack the confidence to rewrite, so we'll spend all day trying to fix them, and if that doesn't work, we'll pair with someone for day 2 to get it fixed. That's 3 man days, for a handful of tests. I've seen it many times, on different teams, rarely are there enough other people noticing how crazy this is to stage an intervention. It's crazy. A contributory reason to why fixing such tests takes so long is that they're so slow. Slow tasks have poor feedback loops. Testing, at least when done as part of CI/CD, is meant to provide fast feedback, and E2E tests fail at this (most especially at 18 months and beyond, where your E2E test is one of hundreds). There's a physics and a psychology to the tiers in the testing pyramid that I meant to write up publicly but I don't think ever escaped a corporate wiki. Here are the Cliff's Notes, based on my own metrics but corroborated by the handful of people who've inspired my testing journey: 1) Moving tests down a tier reduces the power of the test, so you need more of them (about 5x) 2) Moving tests down a tier makes them much faster. (8x common, 10x best case, depending on framework) 3) The simplest tier of tests will be rewritten or deleted when requirements change. All other tests will be 'rescued', sometimes at great expense in time and energy. Rules 1 & 2 create a pseudo-rule, the 5/8ths Rule. If you move a functional test to units, the same coverage will run about 30% faster in aggregate, and you will not have to provide ongoing support for those tests. That's a huge win. If you pull a test down 2 levels, they'll run 60-75% faster.

hinkley 6y ago
This got so long I decided to split it in two: It is important to note you can't pull all of the tests down. I tend to use a plumbing analogy that I probably have stolen from the literature or a mentor - It's likely that QA already tested all of the pipes at the factory before they were installed in your house. But every plumber will turn on the taps and flush the toilet at once before they leave, to make sure the stuff going in one end actually comes out the other (and only the other). Test all the bits in isolation, but always leave at least one happy and one unhappy path test going through the whole stack to make sure good and bad answers end up in the right place. I think here the pyramid analogy is a bit wrong, because in the pyramid there's a geometric reduction at each level, but I have a suspicion that in a best case scenario, you can achieve a log scale reduction (because your E2E tests can potentially reflect the height of the decision tree, rather than the surface area of a module) Tests higher up the pyramid rarely if ever suggest, let alone demand API improvements. If the first person writes elaborate mocks instead of simplifying the design to not need them, they've locked in the implementation in a way that unit tests don't (Sunk Cost). This can turn a masochistic programmer into a sadist. And that is the trap most of us fall into the first few times, and some never outgrow. It's easier to write greenfield E2E tests, but they're hell to maintain, and the create a codependent testing relationship, where the existence of the tests dissuades you from engaging in healthier activities. I almost think that you should ban E2E testing frameworks until the team has gotten comfortable with Unit Tests because of this. With E2E, you can get really far down the wrong road and find yourself in a blind alley that you now lack the imagination or conviction to escape, because everything else feels remedial. That is the main problem with unit tests - they feel like some sort of grade school activity. Doing something so simple feels like doing multiplication tables, which is kids work. I am a smart and functional adult, I shouldn't be doing something so boring. This is making me feel dumb. It takes a special disposition to enjoy a simple task for what it is, and many software developers - myself included - were lured into software by the idea of automating any task that is so simple. Rarely do I let production code be so dominated by idioms with so little meat, so why do I enjoy that in tests? Honestly, I don't have an answer, except "better in the tests than in the production code," which is hardly an answer at all. And importantly, all of these uneasy feelings are used as evidence by anti-testing people that it's too painful and too little gain and we should stop doing it. A form of learned helplessness.

adenta 6y ago

I write a bunch of e2e tests, and not many unit tests. I'm on the fence about if this is a good idea or not. I feel like when you are starting at zero, you need more e2e tests, before writing more unit/service tests as your product is solidified. I wish e2e tests were faster. I feel like this should be achievable, but it's a problem way out of my depths. I feel like you could run a browser within a docker container at 2x or 5x speed with enough virtualization. I've been told this would not work in practice.

Hokusai 6y ago
e2e testing is the way to go. But our current tech stack is really bad for testing. Testing in browsers seems an afterthought, browsers first were designed for features, later on debugging was added, quite good nowadays, but testing is out of the scope. Until we get easy to test browsers everything will continue being fragile and bug prone.

MengerSponge 6y ago

Science Twitter: "Everything is crab." HN two weeks later: "What if testing were crab?"

luhn 6y ago
For those not on Twitter, this went viral a couple weeks ago: https://en.wikipedia.org/wiki/Carcinisation https://en.wikipedia.org/wiki/Carcinisation

tmcneal 6y ago

I'm co-founder of an end-to-end testing product, but I don't actually agree that better end-to-end testing coverage means you should create less unit tests. The reason why is because there's a class of things you want to test that are much better served at the unit testing level. Unit tests are always going to be faster to execute and by definition have a much smaller scope, which is helpful in reasoning about what you're testing. Our product is really focused on making end-to-end tests much easier to create and maintain. Here's an example video I recorded yesterday of automating Monday.com's sign-up flow using our product: https://www.loom.com/share/1336a5bfa9f54f2190269961991802b5 https://www.loom.com/share/1336a5bfa9f54f2190269961991802b5 - looking at this I don't really see what unit tests this would replace, rather I think it gives coverage for things that you'd often have no automated coverage for - i.e. the interplay between disparate components along with interacting with email - basically the steps that form a complete workflow from the user's perspective.

dpix 6y ago
The author talks about how many other sources of bug there might be outside of your code like integration, assumptions, compilers etc that e2e tests will help catch. Well at least a large portion of those bugs might be able to be caught by unit tests! No reason to write less of those, having good unit and integration tests still make you more confident about your application. And because they are faster to run the feedback loop when they break is also faster

bluGill 6y ago

I prefer to think of the vertical dimension of the testing pyramid as time not test type. Nobody can agree on what the different between a unit test and an integration test is, much less end to end. Time is objective though. At the bottom of my pyramid I have the tests that I always run - even if there is no way they could break (I just changed the IP network stack, there is no way the add() function could break - but I'll run that test anyway. Then I have the tests that run pretty fast, but I find it worth some effort to add some dependency analysis to ensure I only run them when it is possible my code broke them. Next there are tests that are slow, so people working in the area it tests run, but everybody else disables them. These tests are just useful enough that the group that runs them doesn't stop running them, but not useful enough for any other group working on a different area of the code. Then there are tests that a very very slow, so I only run them when I am specifically worried I might have broke something that the test covers (typically never because you forget about them when you should, but there are just enough exceptions to get a mention). Last are tests that I only look at after they have failed 3 times in a row on the CI system, and then I'll spend a week wildly/blindly throwing possible fixes at the CI system before I break down and run them myself. Note that last two are for the CI system to run, not humans. In every case I've seen they take long enough that you have to combine the changes of several people before running them just so the CI system can keep up. They are still valuable to have because some important things cannot be tested any other way. However where possible you should find a different way to test that functionality.

CivBase 6y ago

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a machine works, it doesn't mean the machine itself works. That's where integration tests come in. Turns out it's usually fairly easy to make sure all the parts work together once you've verified all of the parts individually, so you usually need fewer integration tests. Once you've verified all the parts work and that they work together correctly, now all you need to do is make sure you actually made the right thing. Enter functional tests and E2E tests. These are usually especially costly to maintain, but they're also so high-level that you usually need very few of them. Writing tests is easy, but they're only useful if you run them. It's not enough for a test to work when you write it - it has to continue working into the future. Maintenance is a recurring cost and, in my experience, is where automated testing can get really expensive. The testing pyramid helps minimize that cost. In an ideal world, you would maximize testing at every level - but in the real world that's often just too expensive.

pydry 6y ago
Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.
- Tainnor 6y ago
  This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense. However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree structure and I need to transform it into another structure" and stuff like that. It's also worth keeping in mind that "unit testing" doesn't have to mean "class (or method) testing". Sometimes the class is the right level of abstraction but sometimes it can also be a small module of classes working in concert etc. I agree that tests shouldn't simply reflect implementation, but there are functional requirements that can be specified at a lower level.

alkonaut 6y ago

Tests should document behavior and aid in refactoring and maintaining boundaries. E2e tests don’t document modules (functions/types), only the highest interface, and it doesn’t maintain any separation of layers etc. All kinds of tests are needed, but I find the pyramid arises from the fact that there are a lot more low level units than high level behaviors.