23 ms·

We killed our end-to-end test suite

• maxencecornet 5y ago
  The scale of Nubank is insane
  • deleted 5y ago
    [deleted]
• phoenix3200 5y ago
  With a photo from a pre-COVID world...
• commandlinefan 5y ago
  > Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal.
  • coryrc 5y ago
    It says the majority of the problems caught in E2E were due to changed contacts, so it makes sense to have a dedicated test type for those errors. If the remaining caught bugs are few enough and their customers are willing to suffer occasional unavailability, just let them be caught in production and save that expense.
  • makeitdouble 5y ago
    A complicating factor is that teams usually start with E2E when it’s simple, and only move to other approaches when the systems have become way too complicated. At that point I don’t know if there is any specific strategy that effectively catches a lot of bugs, short of sending to production and monitor the effect. At a company we just called those tests “sanity checks”, and the goal was just to make sure the most basic use cases would still work, and nothing more.
  • anamexis 5y ago
    > but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal But, have you seen contract testing fail to catch problems that E2E tests did catch? I think both of them end up tending to be regression tests a lot of the time.
  • mbrodersen 5y ago
    Agree. They are not testing the end-to-end contracts of how concurrent services interact. I predict it will end badly.
• msoad 5y ago
  This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong types and validation mechanisms. E2E tests are very hard to maintain but in many situations they are required.
  • commandlinefan 5y ago
    > you can't say end-to-end tests are not worth it Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...
    • coryrc 5y ago
      Tests are never 100%, customers will always find issues. They made the choice to allow slightly more bugs to production than before, not to go from never having bugs in production to having some.
      • alex-freire 5y ago
        we didn't see more bugs in production when we sunseted E2E. in fact, complementing contract tests with acceptance tests we saw less bugs in production and more productive test creation and maintanence. :-D
    • rgoulter 5y ago
      > you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you. This reminds me of the point I've seen made elsewhere: E2E tests can be complemented by use of monitoring metrics, healthchecks, etc. for providing confidence that the system is working as intended (or for spotting cases where it's not working as intended).
  • 5y ago
• caust1c 5y ago
  End-to-end test suites do not excel in continuous integration workflows. They excel as part of your metrics, monitoring and alerting system running continuously for as long as the service they exercise lives. Perhaps it wouldn't be as painful if they took this approach instead.
  • alex-freire 5y ago
    that's a very good alternative way of thinking about E2E
• Pxtl 5y ago
  The "flaky" argument against end-to-end tests does not fill me with confidence in your system.
• liuliu 5y ago
  I am not the typical guy who is going to preach e2e testing when I am in big tech companies. But let's not confuse what e2e tests (or I'd like to call it, integration tests) can do. I love property-based testing, especially with these new frameworks now does coverage-guided fuzzing too. However, it only guarantees the "contract" (or "interface") at that level. For property-based testing (or contract testing this article calls) today, it is still very much to only validate the property holds true, not to exhaust all the edge cases. To give an example, a property-based testing validates function: "add(x, y) == add(y, x), given x in Int32 range", it doesn't validate edges cases what if you call "add" twice, 3 times, from different threads etc. At the end of the day, it would be hard for the property-based testing to validate your component satisfy Liskov substitution principle. Integration testing on the other hand, makes sure your system worked at integrated level. It doesn't enforce Liskov substitution principle too. However, if you have downstream components depend on your implementations (for example, there is an earlier callsite called a function before, second call must be cached), update the upstream and run integration testing makes that implicit assumption apparent. So, that's where I am arriving at. Without a powerful language that can encode all contract at programming level, relying on property-based testing only at components level cannot maintain substitution invariant. Integration testing is required.
• sexcombat 5y ago
  You mean unit tests?
  • ajacksified 5y ago
    I loled that you were downvoted, I was going to post the same thing - they discovered multiple layers of testing? Congrats?
    • sexcombat 5y ago
      right?! Gimme a f* break
• kleinsch 5y ago
  Buried ten feet deep in the article - they retired E2E tests and introduced "acceptance" tests, which are more efficient E2E tests that they still run on critical code. But I guess "We Renamed Our Eng-to-End Test Suite" isn't a very good blog post.
  • dgb23 5y ago
    I don't quite agree. A more precise title would have been: "We Disentangled our E2E Tests" If I understand correctly - what seemed to have happened is that they separated the lower level data-coupling testing and their higher level testing with contracts and acceptance testing respectively. Each of those layers don't need to know of each other, so this was a separation of concerns AKA simplification.
  • alex-freire 5y ago
    Thanks for the feedback, we didn't want to burry the "acceptance test" complement to our testing strategy in the article. If you're curious how it works we've recorded a webinar about it: https://www.youtube.com/watch?v=wKgDaD5Nie4&list=PLfqo9_UMdHhah_gNPnawXYQF6coVrNga3&index=10 https://www.youtube.com/watch?v=wKgDaD5Nie4&list=PLfqo9_UMdH...
    • alex-freire 5y ago
      And in part 2 about 12 minutes in we show the code to exemplify how our acceptance test suite works (and is very different from traditional E2E): https://www.youtube.com/watch?v=caxpxszueI0&list=PLfqo9_UMdHhah_gNPnawXYQF6coVrNga3&index=8 https://www.youtube.com/watch?v=caxpxszueI0&list=PLfqo9_UMdH...
      • gls2ro 5y ago
        I cannot watch this entire video, so it is possible that you explain conceptually how your acceptance tests are different from end to end test? I would like to undestand what were the goals of your initial end to end and what are the goals of acceptance testing, and how you define these acceptance testing from test objective perspective. I assume end to end tests could be described by this definition: "test the functionality and performance of an application under product-like circumstances and data to replicate live settings. The goal is to simulate what a real user scenario looks like from start to finish" [0] [0] https://smartbear.com/solutions/end-to-end-testing/ https://smartbear.com/solutions/end-to-end-testing/
• drewcoo 5y ago
  E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.
  • deleted 5y ago
    [deleted]
  • bryan_w 5y ago
    Catching all the bugs is generous. You won't be able to prove...basic functionally when all the components are deployed in production
  • coryrc 5y ago
    E2E can't catch all bugs either. This team decided the number of bugs their test suite caught was not enough to be worth keeping it. With a robust canary deployment, they will quickly find and rollback breakages whether or not the e2e suite would have caught it.
  • jniedrauer 5y ago
    E2E tests are still extremely limited and let bugs through, unless they also fuzz somehow. But that will make them even more flaky and difficult to debug, costing more time. It's a tradeoff. Example: You run some number of operations and then batch them. If you run the exact same operations each time you test, you may not catch conflicts between them. Instead, you'd have to run a random number of, and type of, operations. But then test failures would become extraordinarily difficult to reproduce. You'd have to hope that your tests log exactly what the inputs were, and have a semi-efficient way to recreate those inputs locally.
  • dgb23 5y ago
    Reading the article beyond the title reveals that this decision was engineering driven, measured and they ended up with a simpler, disentangled solution, by separating contracts that verify compatible schemas on one side and acceptance tests on the other side.
• jniedrauer 5y ago
  I have experienced all the same problems that they outline. E2E tests require a huge number of human-hours to maintain, they're difficult to debug when they fail, almost always false positives, and bugs still get through anyway. But for many situations, there doesn't seem to be a better solution. For most early stage startups, it seems that time would be better spent optimizing your deployments, rollbacks, and real time metrics so you can maintain a high velocity and roll back quickly when you make a mistake. For more safety critical systems, the cost of maintaining E2E tests needs to be built into the total engineering cost for the project. It's a hidden cost that is often way bigger than you'd expect.
  • mbrodersen 5y ago
    Not my experience at all. Having solid end-to-end tests means that I spend 99% of my time adding new features and 1% of my time fixing bugs before deploying. I haven’t had a bug in production for years because of solid end-to-end tests.
• toddh 5y ago
  Contracts are basically unit tests for whatever size of unit you're testing. How do you capture all the dynamic behaviors of a system without some sort of end-to-end test? Delayed timers, queues fill, missed interrupts, locks are held for too long, dead lock, live lock, priority inversion, dropped messages, out of order issues, etc. These things are not captured by contracts and are often exactly why the end-to-end tests were flakey in the first place.
  • JackFr 5y ago
    System A listens to queue B and handles every kind of message b throws at it. But somewhere, at some point in time, some coder has made the innocuous assumption that B_id's are unique.....
• kodablah 5y ago
  Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds trite, but if you don't test end-to-end you aren't going to catch bugs that only appear end-to-end (which also happen to be the ones customers see making the e2e suite a decent place for high level regressions assuming you maintain test performance of course). This is especially true in environment-specific scenarios.
  • Daishiman 5y ago
    I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.
    • kodablah 5y ago
      I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.
      • deleted 5y ago
        [deleted]
      • aidenn0 5y ago
        > I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code. I replied in two other places on this thread before seeing this comment. It's very true. Since tests don't get shipped to customers, tests don't get the same level of effort. But when your tests are known to be of poor quality, people stop trusting them, and when people don't trust the tests, they stop adding any value.
• Rd6n6 5y ago
  Almost all the bugs I’ve ever seen have been an integration or configuration issue. Race conditions are a common example of this. Even the small bugs usually involve 2-3 “units.” End to end tests and integration tests, manual or automated, are really the only way to catch these
• exdsq 5y ago
  Ahhh I've made this mistake before. You can't test every input/output while also losing the ability to accurately depict stateful user flows. Even Fintech projects I've worked on at the >£10Bn daily volume mark combining formal methods, mathematical proofs, property based tests, fuzzing, model based testing, etc.. still caught issues pre-production via good old end-to-end tests. Good luck nubank :)
  • alex-freire 5y ago
    thank you
    • exdsq 5y ago
      I'll be interested to see how it goes, will you follow up with a blog post in a couple months?
• paxys 5y ago
  It's not one vs the other. Both kinds of testing are 100% essential to a stable service. If you have a flaky, laggy E2E test suite...fix it.
  • kords 5y ago
    If you integrate with a third party service and their environment is slow, because it's not a production environment, your tests will fail due to timeouts. How would you fix this? One way I would think is to not go against their service; create a similar service and run it according to your SLA. But then, you have to make sure contracts are in sync, so you need to verify contracts from time to time.
• wly_cdgr 5y ago
  Good, kill testing. Testing is a worthless and tragic waste of human energy and creativity unless actual lives or fortunes are at stake
• daxfohl 5y ago
  FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same size had a huge E2E suite and AFAICT results in roughly the same modicum of production bugs. But feature development at the latter moves much more slowly because of all the wait queues, flaky tests, timeouts, test maintenance overhead, etc. IMO these seem to be more of a CYA thing for managers. When a bug does get to production, you need to have something to point to. (And in my previous company of course they're scrambling now to make a big E2E testing platform). But I'm not convinced they're actually worth the effort. Edit: actually maybe I think they're worth the effort, if for no other reason than when bugs do go to production, executives tend to start micromanaging if you don't have something to point at. That can be worse than dealing with flaky tests. But what I'm not convinced of is whether they actually reduce the number of bugs that go to production.
  • lowbloodsugar 5y ago
    You have been in two of three situations. 1. No e2e tests. 2. Heavy, flaky, slow e2e tests. 3. e2e as a driver of first class system integration Compare with 1. no unit tests 2. tons of unit tests, that regularly fail, nobody cares and check in more bad tests, "unit" tests that thread sleep and take minutes 3. CI/CD with 0 tolerance for failures or >1s tests The difference between "We have tons of tests" and "We drive development with tests" is night and day. Sure, if you slap on e2e with the mandate "They must exist", then you're going to have a shitty experience, e2e or unit.
    • catern 5y ago
      >3. e2e as a driver of first class system integration Could you say more about this approach? That's how I've tried to approach end-to-end tests, but I haven't found much written about this which is specifically about e2e tests.
      • lowbloodsugar 5y ago
        I haven't written a book but just take TDD philosophy but apply it to e2e. For example, if you have an e2e test that often fails because something times out, or you've had to set a high timeout, then dig down why and fix it. Turns out you have a JVM that often has a 2s GC but no alarms, and yes, it impacts customers too. Fix that problem, don't get rid of the test (or all the tests). You got a slow third party thing, put an abstraction in front of it so that username: MyE2EUser's traffic goes to a shim. Then either negotiate with third party, or make interaction with third party system asychronous. Or does resubmitting the page order the product twice?! lol. I wish those were the old days. All that being said, I haven't done this for a couple of years now, and I have the joys of vastly simpler systems at vastly huger scale, so I may be seeing the past through rose tinted glasses.
• kevindong 5y ago
  At $PRIOR_JOB, it always felt like the full E2E tests approached useless since for every bug successfully caught, it felt like there were ~20 false positives. At which point, everyone (myself included) blamed the tests and just repeatedly reran the tests until they usually passed. Every single failure would halt the pipeline anywhere from 5 minutes (in the case that rerunning the failed test shows that it was just a flaky test) up to multiple hours since everyone would rather try to diagnose/hotfix the issue rather than revert their code to unblock the pipeline. With that being said, a full run of the E2E suite at $PRIOR_JOB took very, very low double digit minutes so it wasn't that expensive. Rerunning a handful of failed tests took single digit minutes so it wasn't too terrible.
  • aidenn0 5y ago
    Was in a similar situation, and the VP of engineering banned the practice of rerunning failed tests, so flaky tests caused everybody pain. In less than 8 weeks the false positive rate dropped by about 3 orders of magnitude. There's a strong tendency to treat tests as a hurdle to get over rather than to treat them as first-class part of the development process.
    • deckard1 5y ago
      I imagine this would just turn into everyone inserting 10 second pauses on the tests that fail. Which works, but now your suite doubles the run time. Actually turning nondeterministic tests into deterministic ones is... hard. Really hard in some cases. Many devs don't even understand how to get there, even after years of E2E experience. One place I worked, the E2E suite took a full hour to run. Everyone reran the tests. Merges took a full day in many cases. Management tried to force people to fix broken tests. But they also required new tests on new features. So it was a constant treadmill. There was basically a full mutiny by the end and the company killed off their entire E2E suite.
      • aidenn0 5y ago
        If people just started throwing random sleeps into tests, I think management would shit a brick. Do people throw random sleeps into production code to fix bugs where you work as well?
• robmccoll 5y ago
  I don't know anything about what Nubank is up to or how things work there overall, but integration tests are absolutely worth doing. The argument against this to me reads like "coordination and testing of big systems is hard, so let's not do it." > Waiting. Engineers had to wait more and more to get feedback from this long-running suite; "Our tests are inefficient, not sufficiently parallelized, the setup / tear down of the test environment isn't optimized, and it isn't possible to run only the relevant subset of tests during feature development or bug triage for short feedback loops" > Lack of confidence. Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; "Our tests aren't well written (we have sleep-polling)", "we don't build-in testability into our system (we can't introspect or wait on the thing we care about in the test, so we have massive work arounds)", or possibly worst "our system is flaky and our tests reflect that". > Expensive to maintain. Manual changes in our staging environment corrupted test data fixtures and maintaining the environment “clean” was a challenge; "We haven't spent enough time developing our own tools for testing, so we have tests that are extremely fragile (think copy and paste of massive JSON blobs with comparisons just to check a handful of values)" > Failures don’t point to obvious issues. Test failures were very hard to debug, specially due to our reliance on asynchronous communication that make it hard to connect the cause of failure (a message not published to a queue) with its effect (changes not made in another system); "Our system is over-engineered and our service boundaries match our internal structure rather than clean separation in the functions of our APIs. We don't have good visibility because doing any one thing involves massive levels of coordination. We lack proper tracing and aggregation." > Slower value delivery. Queueing of commits in the End-to-End suite resulted in less frequent deployments; "Quality is hard and takes time. Let's not do it so we can move fast and break things." > Not efficient. Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug; See above about flakiness and fragility. Also, integration tests catching bugs tend to point to really bad obvious bugs. I'd be happy about the one that was caught. > Not effective. Bugs were still being found in production. "We still found bugs. This means testing must be ineffective altogether?"
• tmcneal 5y ago
  Our product is an end-to-end testing tool, so it's always interesting to see what issues companies hit with E2E tests and how they solve them. What's interesting about Nubank's experience is that after deleting their E2E suite, they realized that replacing them with integration tests wasn't providing enough value. There's a lot value in E2E tests, but so many orgs end up taking the wrong approach and ending up with a slow, flaky test suite. We wrote a guide [1] for building automated test suites based on our experience working with and talking to software orgs. Teams who get value out of E2E tests generally do the following things right: 1. They keep tests as small as possible. This makes maintenance easier and forces a separation-of-concerns in the tests. 2. They factor the tests so they can run in parallel. This, plus shorter tests, is the best way to mitigate the slowness issue brought up in the article. 3. They have a good strategy for test data management. It looks like Nubank had test data represented as fixtures, but then somehow manual testing in that same environment was clobbering test data and causing false failures. A better strategy for managing test data could have solved for this. Or maybe even just running the automated tests in an isolated environment. [1: https://reflect.run/regression-testing-guide/ https://reflect.run/regression-testing-guide/]
• continuational 5y ago
  > In our analysis, we figured out that the most frequent category of bugs caught by End-to-End tests was schema violations. Schema violations are pretty much just type errors. Fortunately these can be prevented automatically and with 100% confidence without writing even a single test.
  • fndex 5y ago
    Types are compile checks, they have nothing to do with contracts, having contracts for messages that will transit over a queue are still useful even if you are using a typed language.
    • chartpath 5y ago
      Not in all environments, e.g. clojure.spec, or with MyPy reflection etc.
      • fndex 5y ago
        clojure.spec is not a type system. Still, my point is, just using a typed language won't remove the need for contracts, you would still need to roll up something like Nubank did even if it means using MyPy reflection features. For example, imagine you have two services that communicates through a message queue. Service A produces X as a string, but Service B consumes X as an integer. You can type that, both services would compile, but it would break as soon as you tried to consume that message. And yes, you can build something using MyPy reflection or whatever, but you have to build it anyway.
        • continuational 5y ago
          Hm? Give the message a type. Service A or B would fail to compile, depending on whether the static type of X is a string or an integer.
          • fndex 5y ago
            AFAIK, kafka messages doesn't have "types", and even if they did, you would be relying on an external system, not your type system. If you are not convinced, test it by yourself, create two services and a kafka topic and produce a message from one service to another with different types on each service.
• whoisjuan 5y ago
  If your tolerance to deal with regressions and bugs in production is high and you have millions of users, then you can think of the user as the end to end tester. Maybe you ship some change and put it behind a feature flag and make it available to only 2% to 5% of the users. If you get 1000 users to go through a particular flow and you have a way to collect failure signals from production accurately and in real time, then you can just dial down that flag to 0% if you see a lot of production errors. I'm still not convinced that you can't drop e2e testing completely but maybe if your business allows it you can confidently rely of unit testing or testing contracts without having to run the app through all the user flows for every change.
• JackFr 5y ago
  > The support for messaging tests was immature in the JVM implementation: most of the critical interactions between our microservices occurs through Kafka messages (we favor mutations in asynchronous flows while HTTP calls are mostly reserved for read-only operations). Trying to wrap my head around what is meant by that. I mean I get the second half, but the first half not so much.
  • pstoll 5y ago
    “Instead of investing in making an existing tool better, we built our own thing!” Oye.
• titzer 5y ago
  I work a lot on compilers and VMs and have written tens of thousands of tests at different scales over the years. Different kinds of tests serve different purposes. Unit tests help you pinpoint errors in the code. They can exhaustively test (only) small components to make sure they are fully compliant. They are a refactoring and development aid to the extent that they are focused (don't involve too many components), quick (run in seconds or less), not too tightly coupled to the code under test (i.e. can change code under test without changing tests), and explanatory (failure output is easy to understand and points exactly at the faulting component). Making good unit tests is an art form. Some people love their mocking frameworks. Personally, I hate them. Mocks are confusing and they try to check behavior rather than input/output. They make refactoring hard because they test behavior rather than results, and they are usually confusing. Integration tests are about putting one or more systems together to test their interactions. More than just a single unit, we can put services together and test their interface. They can be more exhaustive about testing a component's interface because the combinatorics haven't exploded yet. Because there is a lot more code under test, failures are less explanatory and thus there is more work to investigate these failures. Investments here that help are to make failure modes as helpful as possible. That, too, is an art form. End-to-end tests are inherently going to be slow. We put the whole system together and run some canned interactions on it. It might be flaky (because large scale, because networks, because OOM, timeouts, etc). End to end tests are generally a bitch to debug, because essentially anything could be at fault...well, anything except the things that are clearly passing their unit tests and integration tests. Which is why you need to have good unit and integration tests, so that you don't need many end-to-end tests. It sounds from the article like they reduced or eliminated their end-to-end tests and went for more integration tests. That does seem to have paid off. Sometimes tests are slow and bad, and other kinds of tests are better. I would say though, working now on a system with many, many, distributed moving parts, you do want to at least have some end-to-end tests that make sure everything comes up properly. Nothing like committing a change that passes all the small scale tests and then a component fails to come up because some stupid command-line flag is set wrong. You gotta have tests for anything you could absent-mindedly break. And all of that testing needs to be a one button push away. You can't have tests that developers don't run, or don't know exist. Personally I like having shell scripts that are checked in and all at least one that does the whole enchilada, even if it is just a wrapper around the build system's or CI's test targets.
• lkrubner 5y ago
  Test suites will tend to fail the more your system has to work with "outside" data. I recently had a client where their own data was entirely dependent on data drawn from 23 different 3rd party APIs, which meant the bulk of their code was devoted to parsing APIs they had no control over. Those external APIs sometimes changed, and sometimes contained bugs (that is, violations of published contracts). To talk about this, I use a broad definition of "outside data". If you're a small startup, "outside data" typically refers to data that belongs to another company. But if you're working in a Fortune 500 company, "outside data" can also refer to data coming from an API run by some other division, which is nominally part of "your" company but is affectively independent. One rule I now offer to my clients: the more your system relies on outside data, the more it is helpful to have run time checks, rather than a test suite. Assuming you run your code on multiple machines or nodes or dynos or instances, you can chose to run the tests on just a percentage of your system, enough to detect problems, but without paying the performance price on 100% of your system. When a problem in your system is because of a change in an external API, your test suite won't catch it, since your test suite works with dummy data. But run time checks will catch the problem and make debugging easy -- you'll see almost instantly which API call created the problem. Code written on the JVM has the beautiful property that you can add pre and post assertions on every function, and you can pass a flag to the compiler asking that the assertions either be left in the code or stripped out. This makes it easier to build 2 copies of the code, one with the asserts and one without, and that makes it easier to, again, deploy the code in such a way that only a limited percentage of your code needs to run those run time checks.
• siliconc0w 5y ago
  'E2E' is often impossible as in most ecosystems as things are constantly changing and you will dependencies out of your control you cannot simulate. The key is faking the right dependencies with accurate-enough versions to keep test fidelity and speed, keeping the test svelte and fast enough so it can run before you merge code to eliminate the size of change being tested and thus more easily understand the outputs to determine if they represent a false positive or negative and where any problems may be. This also all requires building the infrastructure to spin up a simulated world quickly enough to simulate how the proposed change effects the simulation and then analyze the results which is also pretty hard and can get expensive. Luckily spending time optimizing often helps test speed and can control cost so there can be a good case to make for it but orgs have to be willing to pour engineering hours into that and engineers need to want to do it vs building new things which is typically more enticing.
  • mbrodersen 5y ago
    Test against API’s. Not implementation! Your API’s are supposed to be stable. If they are not then you are doing it wrong.
• it 5y ago
  If your system can't be tested end-to-end because of how slow and flaky it is to test that way, doesn't that say something about the quality of your system rather than the tests? Of course once you have such a system, it's probably the result of years of work by many people and most likely it would be hard to make it faster and more reliable. That is probably why people shy away from doing that, and choose to blame the tests instead.
• zemo 5y ago
  > Manual changes in our staging environment corrupted test data fixtures there's a lot here. Manual changes in your staging environment shouldn't affect your tests, because your tests should be isolated from other environments. Also fixtures are generally bad. Given some fixture representing an initial state S, a test utilizing this fixture along with some acceptance criteria is essentially testing that given the state S, running the tests executes some transformation T such that the state of the system is now S2; acceptance criteria evaluate S2 against some known-good value to confirm that T is the desired transformation. This is meaningless if the initial state S is not actually reachable by the system. The fixture itself does not prove that S is reachable: that S is reachable is taken as an act of faith. So how do you determine that the initial state S is reachable by the system? Well, you have some other test, that starts with an initial state of nothing, that performs some transition (generating and inserting random data instead of using a fixture, for example) and gets nothing into the state S. By doing this, you've both created the state S _and_ verified that S is a valid, reachable state. Now you run your second test after the first test in sequence. To run N tests off of initial state S, you replay the initial test that produced state S N times, once for each dependent test. Sure that's a lot of work, but each sequence of testing events can be run in isolation from the others, so they can be run in parallel.
  • aidenn0 5y ago
    Sometimes you have "state S that we got from a coredump from a customer that happened once every 30 computer-years in their deployment so we know it is reachable, but haven't ever seen it happen in-house"
• mjul 5y ago
  In summary they noticed that their e2e suite mostly caught integration errors where clients and servers had incompatible schemas for the data exchange. The novelty is that they found a much faster way to identify this kind of errors by collecting and comparing the client and server side schemas statically without even running the code. This is a great optimisation, but it did not remove all defects so they still need to define tests that validate actual application behaviour against the business rules.
  • alex-freire 5y ago
    well put! thanks for the great summary.
• sethammons 5y ago
  Why not both? Beyond unit test, we have docker compose spin up our service(s) and its dependencies. If those dependencies have too big a web, we may point at a staging instance or a fake server, but we routinely will spin up dependencies that will run a local kafka and zookeeper for them to run, are backed by mysql and redis, etc. We then test our service at its incoming edges (feed its incoming queue or call its endpoints) and verify its output (via logs, metrics, and sinks). We also have end to end tests that exercise our our services from the customer's point of view, but take place in our staging environment. These do suffer from many of the points the article points out, but we run these tests concurrently, and, when not flaky, can pass in 10 minutes. We are addressing flaky tests by addressing their root cause: flaky services in staging. We are expecting teams to have mature monitoring of services in staging and tying improvements directly to flaky failed tests. We are also improving traceability so a failed test is easier to debug to understand if it was a failed service request somewhere in the stack.
• corpMaverick 5y ago
  Don't do comprehensive E2E tests. Have a small set of E2E tests to verify the most critical functionality is working. Don't verify every single business rule. There are other techniques that help you reduce risk: - Good unit and integration tests. - Effective Monitoring and alerting. - Canary releases or Blue/Green releases. - Continuous integration. - Continuous delivery. - Ability to safely rollback the more recent release.
• DonnyV 5y ago
  If that picture is from the actual company. Doesn't look like a good place to work. That environment is horrible.
• deleted 5y ago
  [deleted]
• X6S1x6Okd1st 5y ago
  > One of our Sr Staff Engineers, Rafael Ferreira, ran some numbers and applied queueing theory. Is it just me or is queueing theory seemingly misused all over the place in software engineering. I don't know what was applied here, as it's omitted, but I've certainly seen people suggest that doing things like mandating that cycle time is decreased will increase throughput and justifying what they are saying with "queue theory". Mathematical constructs are great, but they aren't worth much if you can't ensure their constraints are met.
  • mbrodersen 5y ago
    All models are wrong. Some are useful. The problem is knowing if a specific model is useful.
• vemv 5y ago
  The middle ground that not enough teams are exploring is following the so-called Functional Architecture. If all side-effects are effectively segregated and reified, then one should be able to swap them out for determistic mocks that run instantly. So you could E2E a distributed system realistically and instantly. You can still simulate things services being slow, unavailable, etc if the code handling those is expressed as pure logic instead of coupling itself to IO. Interestingly, in addition to fixed examples, you can perform generative testing over this setup e.g. what happens for various combinations of services being slow/down.
  • alex-freire 5y ago
    exactly! well said. it's what we're trying to achieve with our acceptance testing strategy. you can see more here how we leveraged clojure to be able to simulate E2E in memory in the JVM by bypassing IO and just having one services logic+data layer talk to the other: https://www.youtube.com/playlist?list=PLfqo9_UMdHhah_gNPnawXYQF6coVrNga3 https://www.youtube.com/playlist?list=PLfqo9_UMdHhah_gNPnawX...
    • vemv 5y ago
      Nice! I missed that nuance in the article. Looking forward to eventually check out Sachem if there's a plan to share it?
• deleted 5y ago
  [deleted]
• rafaelturk 5y ago
  As a rule of thumb: If you're not in charge of testing, your users will have to test it for you. Looks like Nubanks's e2e was flaky and poorly optimized, so now their will try to `silo` tests in form of contracts.
• ArchStanton 5y ago
  Wait, what? Do people actually work in conditions like that article's picture shows? Good Lord.
• xrd 5y ago
  I'm a big advocate for testing, to state up front. E2E is problematic from the start because of the expectations set by the name. Any sufficiently interesting system is nigh impossible to test "end to end." And, you aren't testing ends, you are testing the "start" of the process, to one of many "ends." What about only doing "end testing?" Meaning, don't test the beginning. Put unit tests there. Put integration tests between the important components. It is important to make sure you have coverage with automated tests that prove that your system can work at the end of at least some of the processes. Otherwise your QA costs are massive, and that never scales, and no one will ever fix that other than adding QA. Your innovation will slow to a crawl, much worse than waiting on your test suite. I'm not sure after reading this article that the authors added a new testing methodology by calling it "contract testing." I'm still confused about what that means. Having said that, I am still confused about a lot of the boundaries between e2e and integration. It always sounds simple, but rarely in practice. The bottom line: the organization as a whole has to see the value of testing. That's harder work than writing the tests for sure.
• ridaj 5y ago
  Erm most people need all kinds of tests, it's not a story of one vs the other... Unit, integration, end to end... The end to end is always slowest and flaky so not everything should go there but with any mature ish product it's also impossible to catch issues related to system complexity without it... It's also weird how self important this account reads. The field is not new but they don't really describe how their homegrown stuff does better than other frameworks and present themselves as visionaries... Smelly
• okokwhatever 5y ago
  So, they're not testing at all. Just inputs and outputs. mmm ok.
• eyelidlessness 5y ago
  I’m generally not fond of e2e or even integration testing. At least, I prefer to keep them to a minimum, and use other tools to ensure units interact as expected. That said, where e2e tests may be valuable but costly as described in the article, it occurs to me that narrower integration tests which invert responsibility may be better. Which is to say: - Given Service A - Given Service B which depends on Service A Integration tests of Service A may provide more value if implemented in Service B. It’s SB, after all, which understands the behavior it expects from SA. (If they’re mutual dependencies, of course the inverse applies as well.) Of course, this highlights (at least for me) why integration tests should be limited in scope. If both services are well tested at the unit level, you will probably end up with a lot of redundancy between their reciprocal test suites. But at least at the idea level, this feels like a better compromise than expecting Team SA to anticipate all of the subtleties Team SB might have in mind.
  • CobrastanJorji 5y ago
    What are the tools you do use to ensure units interact as expected?
    • eyelidlessness 5y ago
      Mostly making types more specific. Within a service, ensuring semantics are part of the type so they can be checked statically, and designing function boundaries so those semantics are part of the interface. Between services, using standardized machine-usable docs (like OpenAPI and JSONSchema) to share those types in a well-defined way. This is harder because network boundaries are less flexible, but it does help a lot.
  • catern 5y ago
    Yeah, I think the easiest way to test that Service A is meeting its API obligations is to send some requests to Service B. Hyrum's Law means that that's the only way to really test the important aspects of Service A's API. And I think this can be generalized into a general philosophy of using your users as a test suite for your API: http://catern.com/usertests.html http://catern.com/usertests.html
• mirekrusin 5y ago
  So they ditched e2e in favour of something that average monorepo checks statically ie throught typescript? Then mocked functional tests and called it a day? The problem probably started when they put themselves in this microservice plague setup where they can't spawn simulated environment in ci anymore. As it turns out running system aka deployment on environment is a monolithic expression of microservice spaghetti. As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a good button, period. Taking aside fact that it inflates test runtime more than decade of mcdonalds diet - if you find yourself in setup with flaky tests, you should ask why they are flaky and amend setup so test is expressed as non-maybe-flaky, normal test. Forbit flakiness, there is no such thing as passed flaky test - those are just shitty tests.
  • _hilro 5y ago
    > As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a good button, period Or more like the test runner succumbs to non deterministic flaky behaviour. If something failed 65% of the time, it would be one of the easiest thing in the world to fix. If it fails .001% of the time, that's what the industry refers to as flaky. > Forbit flakiness, there is no such thing as passed flaky test - those are just shitty tests. Have you ever written and monitored e2e tests over a year? It's industry wide. Selenium/selenium grid always works great until it doesn't. Ditto with the new kids on the block. e2e outside of a browser is 100% fine unless there's an actual bug somewhere.
    • mirekrusin 5y ago
      Let me clarify, I think I haven't expressed myself well. What I meant to bash is the culture where you wrap e2e tests with retries and consider flaky tests green. This arrangement is a pathology. Industry did not put 1 out of 100000 (.001%) or less as threshold for calling something flaky. From my experience I've seen 20% success rate tests passing due to retries and teams are living with it as normal. > Have you ever written and monitored e2e tests over a year? It's industry wide. Yes, on high profile projects. I find myself repeating how important determinism is. Flaky tests, no mater how frequent, are indistinguishable from bugs, which implies they can't ever be considered green. Architecting testing environment in such a way that it is deterministic is fundamental. Sometimes it doesn't require big reshuffles, it just means the test has to be rephrased in deterministic terms that matter without asserting intermediate, timing based, racing middle states. As an example testing random failures by killing services doesn't have to assert intermediate client states, it has to assert that final state is eventually correct, which implies reconnects did happen and eventually state is correct, regardless of intermediate client states (ie. retry logic on the client auto healing itself on idempotent actions or erroring notifying client the service is offline, in which case user itself has to retry action - both are ok depending on how long service was offline, both can be progressed from test PoV, assertion that specific one happened is irrelevant).
• gorgoiler 5y ago
  The sad fact of E2E is that the tests genuinely find broken stuff. The “false negative” test results usually just mean false as in “something was broken, just not what the CI claimed was broken.” It’s could be anything, so you need automatic specificity as to what’s broken (hard) or buy-in from the entire organisation to be on standby for finding broken stuff (also hard.) “Anything” as in if your external DNS provider has 1 of 10 resolvers with an out of date zonefile, or a dodgy switch port to that particular resolver. It’s hard but if it’s broken then it’s likely it is a real issue one of your end users is also experiencing. A commitment to E2E is committing to a level of quality across your entire infrastructure that few people are prepared to own.
  • physicles 5y ago
    This is our goal with E2E tests and monitoring — write tests that fail if and only if customers are experiencing issues — and I don’t know how to achieve that level of I-can-sleep-well-at-night assurance of quality without it. We run our E2E tests continuously against prod from a different cloud region. As you wrote, the challenge is that there are literally 100 things that could cause a test to fail (some of which are outside your control), and as your team scales you’ll have to get smart about how to efficiently dispatch people to fix problems.
• sowbug 5y ago
  We've seen this before. At first, the E2E tests automate a bunch of tedious manual tests. Then they become the team's "automated test suite." Then nobody (except true believers) tests their code anymore, because "our automated test suite" will surely catch any problems. Then everything starts breaking down the way the author cites. The E2E test suite needs to be thought of differently from a "test suite." It's the last safety net to disprove that the build is worth [manual testing | dogfooding | beta release | prod release]. Any bug found there should be worth a postmortem -- even one as short as "oops, new guy forgot to update the unit tests! fixed, won't happen again!" Of course, bugs will get through to the safety net, even in a system that's working well. But they should always trigger the question whether and how the bug could have been caught earlier.
• mabbo 5y ago
  This is a very bad list of complaints and it actually makes me angry to read it. > Engineers had to wait more and more to get feedback from this long-running suite So speed up your tests. Run them in parallel. Find better frameworks for running tests. > Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; Fix your flaky tests! Why anyone just accepts that "Oh, sometimes that test fails and we have to restart everything" is beyond me. Root cause the problem and FIX IT. > Manual changes in our staging environment corrupted test data fixtures and maintaining the environment “clean” was a challenge; Tests should not rely on pre-existing state. Have a setup phase for each test that creates new data in the state you want it to be in. As it makes this data, also note down a reference to it with a Time To Live so that a follow-up process can clean up this unneeded data. > Test failures were very hard to debug That's not the fault of the tests, that's the fault of a complex system that is hard to debug. Improve your tracing between services. > Queueing of commits in the End-to-End suite resulted in less frequent deployments There are well known solutions to this problem. Lots of companies have overcome this already. > Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug If your tests have false-positives, you need to adjust your tests. Accepting that the test failed but there isn't a problem, and then not fixing the reason the test failed means that you don't have reliable tests. > Bugs were still being found in production Bugs will always make it to production. But after you fix a bug, you write a test so that this bug cannot happen again. Over time, the number of possible bugs that can make it to production shrinks. And lastly: > The main difference to the old E2E is that they encompass only a subset of services and don’t require spinning a production-like environment (the services run in memory on a single JVM and HTTP/Kafka communication is replaced by in-process communication). They are used in specific flows that we find too critical to only rely on Contract Tests. Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra?
  • lamontcg 5y ago
    > Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra? We actually do pretty well testing against fake infra. We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightweight server written in ruby which would never scale but is the same API. When the test suite is updated both the implementations need to be fixed. When the client code runs integration tests we can test against the lightweight ruby code and when it passes we actually have pretty high confidence that it runs against the production code. We have hundreds of those tests and they can be run as fast as spinning up a ruby process with an in-memory datastore which is trashed on every test. Compare that to end to end tests that might fire up a set of images, terraform them into production servers and clients and run a scripted interaction or set of interactions and then throws that away and does it again. At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. The right strategy is that you want to have enough of the most realistic tests to give you a high level of confidence that your faster, slightly less realistic tests are useful, and by having those faster tests you increase your amount of coverage, and on down the stack iteratively until you may get to unit tests of individual objects.
• eurasiantiger 5y ago
  For rewrites and refactors of integrations, there is no substitute for E2E testing. I’m hopeful to be proven wrong, though.
• ialyos 5y ago
  Isn't this what protos are for?
• AndrewKemendo 5y ago
  This sounds like a garden variety example of an organization moving to event-based architecture and implementing contract testing. What makes this interesting is that they had decided early on to be a Clojure/Lisp shop so there weren't great tools out there at enterprise scale. I don't understand why they decided to built everything on Lisp, I assume it was best for them at the beginning. But given the known lack of tools for Lisp to scale into large scale enterprises - this is the kind of thing I would always consider once you get past MVP phase. Otherwise you have to end up building all your own stuff. Architecture matters.
  • askonomm 5y ago
    What enterprise tools are you thinking of? Because Clojure runs on the JVM, I'd imagine what works for Java also to work for Clojure. Personally I see Clojure as a perfect choice for an ever-increasing amount of complexity, since you'd keep side-effects on the edges of your app and everything else as a pure function. Every occurrence of enterprise OO I have ever seen has been unmanageable mess that burns everyone out in a quick year, because you have no idea where something begins or ends, what goes on in-between, and instead of doing what actually matters (business logic) you spend most of your time creating abstractions.
• joshribakoff 5y ago
  I’m not sure I follow the logic that their e2e test suite would take an “infinite” amount of time to run by 2021. It seems like an obviously faulty calculation, unless someone puts an infinite loop.
  • AmpsterMan 5y ago
    I think what they meant is that at the pace they were committing coffee to production, the e2e suite would never stop running. This is the most charitable interpretation.
    • alex-freire 5y ago
      you’re right sir!
    • joshribakoff 5y ago
      The way we solve that at Brex it scales logarithmically, we use bors merge bot which batches everyone’s PRs together and will binary search for the offending PR if any build fails. I’ve hardly ever waited longer than 2x the length our suite takes. They could also decouple CI and only run code tests for upstream changes, so it wasn’t obvious what they meant, thanks for clarifying one theory. I’m not sure I’d agree that it’s a problem with e2e tests per se but rather the problem is slowness itself, and monolithic CI. Not doing e2e tests can be a valid tradeoff to avoid slowness but I’d also point out these issues can be solved without deleting the e2e suite…
• Aeolun 5y ago
  Hogwash. You did E2E testing badly, and now you’ve replaced it with something else. Which may or may not be as bad, but is certainly bad faster.
• jph 5y ago
  > One of our Sr Staff Engineers ran some numbers and applied queueing theory. Queueing theory is an excellent way to look at E2E testing for the big-picture view and also for drilling down into each of the relevant services. For a quick intro, this is a queueing theory primer that I've written and shared with HN previously: https://github.com/joelparkerhenderson/queueing-theory https://github.com/joelparkerhenderson/queueing-theory
• intricatedetail 5y ago
  "We can't develop proper end to end pipeline, so we killed it and patted ourselves on the backs".
• physicles 5y ago
  There’s the classic diagram showing tests as a pyramid[1]. At the bottom you have unit tests, white box stuff that mocks all dependencies and runs super fast. At the top you have E2E tests (or acceptance tests or whatever, names are fungible), but very few of them, to catch bugs at the boundaries between systems. In the middle you have stuff that maybe runs against a live database instead of a mock. As you go further up the pyramid, tests get more expensive in every dimension: they are slower (and are run less frequently as a result), more expensive to debug and maintain, and maybe a bit flaky, though you should still de-flakify these tests as much as is practical. The key is to push tests as far down the pyramid as possible. Never test something in an E2E test if it can be meaningfully tested in a unit test. For example, a regression test for a database date/time serialization bug should run against a real database. Meanwhile, a test verifying that an HTTP service returns the correct response code in a specific situation can run against mocks. 1 https://martinfowler.com/articles/practical-test-pyramid.html https://martinfowler.com/articles/practical-test-pyramid.htm... (holy crap that is a mountain of text, but the diagram is near the top)
• mbrodersen 5y ago
  The real problem is that they don’t know how to write fast end-to-end tests. Not having solid end-to-end tests is a massive red flag. Also, each service can be simulation tested in isolation. So even if they don’t have the skills to do end-to-end testing, they could at least do per-service simulation testing, with each service team responsible for running those tests independently.