4 ms·

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite ove

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite overall. Does this help improve that?

Sholmesy 4y ago
My experience with flaky tests is that they seem to leave some state lying around, or are impacted by some randomness that wasn't accounted for in the test writing (lists in different order, DB state etc). So I actually do think this would benefit that, as these tests are run "independantly", however that's just going to hide the problem of your bad/flaky tests, and you'll never solve the underlying problem.
- agersant 4y ago
  I have the opposite experience. LocalStorage can leak between tests and that has been an issue for years: https://github.com/cypress-io/cypress/issues/2695 https://github.com/cypress-io/cypress/issues/2695
punkpeye 4y ago
It sure does. The way Cypress works is that it injects their entire driver (test API) to the browser and then loads the tests themselves, also in the browser. It is a finicky process because they need to rewrite all JS and HTML (https://github.com/cypress-io/cypress/tree/develop/packages/rewriter https://github.com/cypress-io/cypress/tree/develop/packages/...). It is slow because they have to proxy everything through HTTP 1.1 proxy (https://github.com/cypress-io/cypress/tree/develop/packages/https-proxy https://github.com/cypress-io/cypress/tree/develop/packages/...). The thing that I built takes a different approach. It parses Cypress tests, interprets the commands that Cypress wants to run, and then uses Chrome DevTool protocol (CDP) to evaluate commands such as navigation, click, etc. If you are familiar with the JavaScript ecosystem, my tool effectively transforms tests to Playwright compatible tests and then uses custom orchestration logic to run them.
- twobitshifter 4y ago
  I wouldn’t give away too many technical details of how things work.
polskibus 4y ago
What has the reputation of being less flaky then Cypresss? My experience is that it is easier to write a non flaky test in Cypress than in Selenium.
- jackblemming 4y ago
  Nothing. The problem is E2E UI tests are always going to be flaky. That’s why many recommend unit tests or pseudo E2E tests that save a copy of an API response. Unfortunately this advice is rarely followed.

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite overall. Does this help improve that?

Sholmesy 4y ago

My experience with flaky tests is that they seem to leave some state lying around, or are impacted by some randomness that wasn't accounted for in the test writing (lists in different order, DB state etc). So I actually do think this would benefit that, as these tests are run "independantly", however that's just going to hide the problem of your bad/flaky tests, and you'll never solve the underlying problem.

agersant 4y ago
I have the opposite experience. LocalStorage can leak between tests and that has been an issue for years: https://github.com/cypress-io/cypress/issues/2695 https://github.com/cypress-io/cypress/issues/2695

punkpeye 4y ago

It sure does. The way Cypress works is that it injects their entire driver (test API) to the browser and then loads the tests themselves, also in the browser. It is a finicky process because they need to rewrite all JS and HTML (https://github.com/cypress-io/cypress/tree/develop/packages/rewriter https://github.com/cypress-io/cypress/tree/develop/packages/...). It is slow because they have to proxy everything through HTTP 1.1 proxy (https://github.com/cypress-io/cypress/tree/develop/packages/https-proxy https://github.com/cypress-io/cypress/tree/develop/packages/...). The thing that I built takes a different approach. It parses Cypress tests, interprets the commands that Cypress wants to run, and then uses Chrome DevTool protocol (CDP) to evaluate commands such as navigation, click, etc. If you are familiar with the JavaScript ecosystem, my tool effectively transforms tests to Playwright compatible tests and then uses custom orchestration logic to run them.

twobitshifter 4y ago
I wouldn’t give away too many technical details of how things work.

polskibus 4y ago

What has the reputation of being less flaky then Cypresss? My experience is that it is easier to write a non flaky test in Cypress than in Selenium.

jackblemming 4y ago
Nothing. The problem is E2E UI tests are always going to be flaky. That’s why many recommend unit tests or pseudo E2E tests that save a copy of an API response. Unfortunately this advice is rarely followed.