3 ms·

> they tend to be flaky If your E2E test is flaky, then how is your production environment reliable? > they can require a lot of setup Setup vs mock. > the

> they tend to be flaky If your E2E test is flaky, then how is your production environment reliable? > they can require a lot of setup Setup vs mock. > they can be very slow, etc. We have a good handle on scaling, slow is now a product of the testing time being smeared across your org like peanut butter. > The purpose of unit testing... Does E2E replace unit testing, no. Things that are already units or esoteric (business "logic", you know it's in your code base). But if your writing mocks your better off un fucking your environment that E2E testing is viable, and make it a first class citizen. Everything is a crud app of some sort: web, api, mobile, smoke signal... Dropping data in these front ends, can be validated out of the back end, and in the app itself. To put a fine point on it: Unit tests make me feel good about checking in. End to end tests make me confident to deploy!

TuringNYC 3y ago
>> If your E2E test is flaky, then how is your production environment reliable? IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.
- colinmorelli 3y ago
  I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not. From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
  - ysavir 3y ago
    The end user isn't exposed to it. The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
    - zer00eyz 3y ago
      > The flakiness comes from needing an automated browser interaction acting against a page with changing input. This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
      - ysavir 3y ago
        I'm somewhat at a loss as to what point you're trying to make. Can you clarify what you consider to be E2E tests, and what you consider to be unit tests, with examples?
        zer00eyz 3y ago
        End to end: for your typical web app/stack: Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations). Mobile: same as browser... API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end. You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths. Unit testing: Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic"). The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!
  - TuringNYC 3y ago
    >> From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken. I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO
    - colinmorelli 3y ago
      I said nothing of tests, I said “end user.” And many studies show that, in fact, your end users may not wait 120 seconds at all.
wouldbecouldbe 3y ago
Yeah you have never wrote end to end tests or run them only in utopia. Whether you use playwright or cypress getting the correct wait events figured out is a huge pain. Only if you run pure server side applications that’s simple.
- gkbrk 3y ago
  It's almost like server-side applications are more reliable.
  - wouldbecouldbe 3y ago
    Yeah it's almost like you join a team and that's not your call, and it;s almost you might join a team that works an SPA application that's so complicated it would need real time updates.
- zer00eyz 3y ago
  >> Only if you run pure server side applications that’s simple. End to End testing is less about "making the front end do all the work" and more about being free of mocks and synthetics. Just because the front end exercises my server, does not mean server devs get a pass on testing API down. The tests I write should inform the data your sending and receiving, they should function as an example of inputs and outputs of what to expect in the system. Those timing issues are either a real front end problem, or a server problem (and my own tests should have noted that I did not meet the SLA).

TuringNYC 3y ago

>> If your E2E test is flaky, then how is your production environment reliable? IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.

colinmorelli 3y ago
I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not. From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
- ysavir 3y ago
  The end user isn't exposed to it. The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
  - zer00eyz 3y ago
    > The flakiness comes from needing an automated browser interaction acting against a page with changing input. This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
    - ysavir 3y ago
      I'm somewhat at a loss as to what point you're trying to make. Can you clarify what you consider to be E2E tests, and what you consider to be unit tests, with examples?
      - zer00eyz 3y ago
        End to end: for your typical web app/stack: Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations). Mobile: same as browser... API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end. You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths. Unit testing: Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic"). The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!
- TuringNYC 3y ago
  >> From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken. I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO
  - colinmorelli 3y ago
    I said nothing of tests, I said “end user.” And many studies show that, in fact, your end users may not wait 120 seconds at all.

wouldbecouldbe 3y ago

Yeah you have never wrote end to end tests or run them only in utopia. Whether you use playwright or cypress getting the correct wait events figured out is a huge pain. Only if you run pure server side applications that’s simple.

gkbrk 3y ago
It's almost like server-side applications are more reliable.
- wouldbecouldbe 3y ago
  Yeah it's almost like you join a team and that's not your call, and it;s almost you might join a team that works an SPA application that's so complicated it would need real time updates.
zer00eyz 3y ago
>> Only if you run pure server side applications that’s simple. End to End testing is less about "making the front end do all the work" and more about being free of mocks and synthetics. Just because the front end exercises my server, does not mean server devs get a pass on testing API down. The tests I write should inform the data your sending and receiving, they should function as an example of inputs and outputs of what to expect in the system. Those timing issues are either a real front end problem, or a server problem (and my own tests should have noted that I did not meet the SLA).