4 ms·

I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not. From an en

I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not. From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.

ysavir 3y ago
The end user isn't exposed to it. The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
- zer00eyz 3y ago
  > The flakiness comes from needing an automated browser interaction acting against a page with changing input. This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
  - ysavir 3y ago
    I'm somewhat at a loss as to what point you're trying to make. Can you clarify what you consider to be E2E tests, and what you consider to be unit tests, with examples?
    - zer00eyz 3y ago
      End to end: for your typical web app/stack: Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations). Mobile: same as browser... API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end. You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths. Unit testing: Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic"). The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!
TuringNYC 3y ago
>> From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken. I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO
- colinmorelli 3y ago
  I said nothing of tests, I said “end user.” And many studies show that, in fact, your end users may not wait 120 seconds at all.

ysavir 3y ago

The end user isn't exposed to it. The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.

zer00eyz 3y ago
> The flakiness comes from needing an automated browser interaction acting against a page with changing input. This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
- ysavir 3y ago
  I'm somewhat at a loss as to what point you're trying to make. Can you clarify what you consider to be E2E tests, and what you consider to be unit tests, with examples?
  - zer00eyz 3y ago
    End to end: for your typical web app/stack: Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations). Mobile: same as browser... API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end. You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths. Unit testing: Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic"). The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!

TuringNYC 3y ago

>> From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken. I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO

colinmorelli 3y ago
I said nothing of tests, I said “end user.” And many studies show that, in fact, your end users may not wait 120 seconds at all.