7 ms·

I’ve had to use Cypress at work, and I really don’t like it. I find the command chaining and assertion queuing to be very unintuitive. (“Do I chain this one or

I’ve had to use Cypress at work, and I really don’t like it. I find the command chaining and assertion queuing to be very unintuitive. (“Do I chain this one or not? Maybe stick it in a callback? No idea!”) It’s been a few months since I’ve touched it, so I’m struggling to give more specific criticisms. I’ve had a really great time using Kent Dodd’s DOM Testing Library [1] for unit tests. The readability and writability of these tests has been fantastic. I wish I could use the exact same API for e2e tests. It looks like they actually do have a Cypress version of the Testing Library [2], but it seems to suffer from the same Cypress-isms that I was just complaining about. Has anyone else had a similar experience? It seems like I’m in the minority opinion. Maybe I’m just bothered because I don’t understand Cypress well. [1]: https://testing-library.com/docs/dom-testing-library/intro https://testing-library.com/docs/dom-testing-library/intro [2]: https://testing-library.com/docs/cypress-testing-library/intro https://testing-library.com/docs/cypress-testing-library/int...

swsieber 6y ago
To hop on the unhappy train, it's actually quite hard to get an internal mirror mirror working correctly because the platform for which the extra file it downloads is based on a query string, not the path. Argh.
dpc_pw 6y ago
I very dislike Cypress as well. This even model is sooo confusing, clumsy and error-prone that writing any non-trivial test is a challenge. Whole product uses async/await or Promises and suddenly you have to switch to this baroque model just to write a test. Reusing logic between production code and tests is pretty much impossible. Dynamic pages etc. often confuse cypress. I'm actually not a frontend developer, so I don't have a lot of insight while exactly that is, but I just know that it has been a huge drag. Other than that, it would be OKish. Being able to test the UI is great, but I would advice everyone not to base the whole testing strategy on it. It's just too heavy, slow and unreliable. It doesn't not scale even moderately with the product growing. e2e testing is better done on API level, IMO. You'll want redesigns, changes of UX, and suddenly your changes will require updating tests, which otherwise (if done on the API level) would require no modifications.
- themarkn 6y ago
  It’s possible (and indeed uh ... recommended) to write frontend tests that are resilient to DOM changes by not depending on things that are not essential to the functionality being tested. Eg select a form field by its label, not by its position in the DOM, and use specific data attributes if there’s no better choice. Then when you gut the HTML for UI overhaul, if the functionality hasn’t changed, the cypress test doesn’t change, you just make sure you attach the attributes to the right new elements. Of course if the functionality changed (like you split something up into 2 pages) yeah you need to update your UI tests cause you are testing a different experience now. But they are UI tests, so yeah. That’s appropriate. I dunno. It sounds like you found other parts of it unwieldy as well, but the idea that the tests are inherently brittle is wrong. Time spent maintaining UI based testing is saved many times over when compared with the manual clicking around that it saves imo.
  - eitland 6y ago
    related: testing-framework has been adapted for cypress. It encourages this kind of thinking and testing. I've used cypress at work but have suspended it since we used it end-to-end and it clogged up backend databases.
    - arcturus17 6y ago
      Can’t you set up a test database instead of testing against production? And set up timeouts to control the tempo of the tests if necessary?
      - deleted 6y ago
        [deleted]
  - dpc_pw 6y ago
    > Then when you gut the HTML for UI overhaul, if the functionality hasn’t changed, How often website redesigns are only about html? Usually everything changes. New flows, new dialogs, new toolbars, new frameworks. Often the web UI gets and Android UI and iOS UI siblings. Gets replaced altogether by a completely different UI, developed by a different team. APIs change much less often, and can be shared between many UIs. You can also version them. How are you going to version a web UI? Like reddit with "turn back on classic look"? How long can you keep the old UI around just because you have 1000s tests around that use it. > but the idea that the tests are inherently brittle is wrong I don't think so. Tests are great calcifyiers (check my blogpost about it if you want). If you test though your UI, you're calcifying your UI. The more you test though it, the more you calcifying it. No way around it. https://dpc.pw/be-warned-tests-are-great-calcifiers https://dpc.pw/be-warned-tests-are-great-calcifiers
BurningFrog 6y ago
I spent a few weeks with it, and it was very frustrating. The programming model is very odd. I found it next to impossible to share any code between tests, and most of the programming skills I've learned over the decades didn't seem to apply. It is vastly superior to anything else we've seen in stability and polish. If I only could write reasonable code for it, things would be great...
- strunz 6y ago
  Last time I checked you couldn't even organize tests in different folders, I wonder if they finally fixed that. I always find it odd using test frameworks developed by a for profit company. They pick and choose what code to merge and bugs to fix based on their own priority, not what's best for the users.
  - exdsq 6y ago
    I found this when I used Cypress at a contract last month - ended up just using a Makefile `make test` to run every suite.
- jpsimons 6y ago
  It's like Obamacare -- deeply flawed but hey at least they're trying to do something.
- Mavvie 6y ago
  What made it hard to reuse code? You can just import/export stuff like normal, unless you mean it was hard to find any logic that could be usefully extracted into a helper?
  - BurningFrog 6y ago
    We have a lot of data in UI tables, so I wanted to write code that extracted the cell values to a ROWSxCOLUMNS matrix, that I could reuse in each test. And... I just couldn't figure out a way!
tmcneal 6y ago
If you're willing to try something a bit different, we're working on a no-code approach to test automation that tries to avoid the pitfalls of code-based testing [1]. We're focused on building a E2E testing platform that (1) makes it as easy as possible to create tests and (2) makes tests easy to maintain. Easier said than done but that's our goal. [1] https://reflect.run https://reflect.run
- duncan-donuts 6y ago
  What are the benefits over selenium ide? It’s been years since I worked heavily on testing but I don’t recall there ever being a good record/playback testing tool. I’m intrigued.
  - tmcneal 6y ago
    Selenium IDE uses a Chrome or Firefox extension to record your actions, but they're rewriting it to be an Electron app. I'm not sure about the status of that rewrite though. But the benefits we have vs. what you experienced with Selenium IDE is: - Instead of using an extension to record your actions, there's nothing to install to use Reflect. We spin up an instrumented browser inside a virtual machine and screenshare that with you within our web app. It records your actions and translates those actions into a repeatable test automatically. - Since we have complete control of the test environment, we can do cool things like make recording file uploads really easy or present a nice workflow for getting coverage of visual regressions via screenshot testing. - Our recorder is more accurate :) - For every test you run within Reflect, you get a video synced up to the steps in your test, along with console and network logs from the test run. - Instead of emitting Selenium code that you then maintain like a normal code-based testing tool, we are completely no-code meaning that even updating a test is completely codeless. So for example if you need to make bigger changes to an existing test, we expose a way to re-record only the portions of the test you want to change, and keep the rest as-is.
inglor 6y ago
Yes! I had a very similar experience and I want to really just write regular JavaScript, I mention this in https://www.testim.io/blog/puppeteer-selenium-playwright-cypress-how-to-choose/ https://www.testim.io/blog/puppeteer-selenium-playwright-cyp...
hansworst 6y ago
I've found testcafe [1] to be a good alternative. [1]: https://devexpress.github.io/testcafe/ https://devexpress.github.io/testcafe/
- Guillaume86 6y ago
  Same experience here, moved from cypress to testcafe after getting frustrated by cypress API. It was some time ago so I may not remember correctly but I felt like cypress was not for people already fluent in javascript, a bit like jquery but pushed even further (you need to do everything the cypress way now, you can't even use variables or promises...).
- wdb 6y ago
  Doesn't it force you to use some form of page objects? That stopped me from using TestCafe didn't like that
  - mstapp 6y ago
    No, page objects (AKA page models) are a recommended but optional pattern in TestCafe. We use them on my team because they make the tests more readable and keep us from having copies of the same selectors in multiple places, but you can easily use TestCafe without them.
yani 6y ago
I am quite surprised. Our experience has been quite positive. We were able to do full coverage of our web app in less than a month without prior experience with cypress. This is what we learned from it: https://blog.servmask.com/8-tricks-i-learned-from-cypress/ https://blog.servmask.com/8-tricks-i-learned-from-cypress/
foolfoolz 6y ago
it took a lot of time but i do like the cypress approach. there are weird things like how you can’t get the current time because all lines execute immediately and are evaluated later. but once you get your handle on .then and .wrap it is good. they should have picked a different name than .then because it’s not a promise but looks like one
varrock 6y ago
I really enjoyed the testing library when I used it. I recently came back to it and felt like they removed a lot of the documentation that I liked about the library. I'm not quite sure why they did, but it was very helpful to see their examples for integrating with common third party libraries.
crb002 6y ago
It is visually appealing but sluggish and sucks at non-standard authentication to the point of not being usable. Also no out of the box way to run more than one instance with incognito tabs to have communication to test multi-user bugs. Perhaps radical, but I think it should support hooks with Playwright/Puppeteer that you can hand control back and forth to as a safety valve just like being able to shell out to the command line when your favorite programming language doesn't have a library for what you need. The difference here is that the browser context needs to be passed - that means some janky handoff (which with crypto will be a PITA) or using the Playwright/Puppeter connector under the hood.
jeff00seattle 6y ago
I have been developing tests using Cypress for 6+ months. I have used multiple other variations of test frameworks that were built on the Selenium WebDriver protocol. The official documentation [1] states that Cypress differs fundamentally and architecturally as compared to Selenium. Admittedly I found Cypress web testing framework frustrating initially, and yes, I have performed many re-writes because Cypress documentation behind command chaining and assertion queuing to be not intuitive. Thanks for my management's support, perseverance afforded me with the stubbornness with a purpose of checking out Cypress if it has a better DOM manipulation, how it operates directly in the browser, interactive test runner, and its API and configuration. It required 3+ months of perseverance in getting the Cypress tests organized [2] for handling web testing core functionality of a commercial-grade corporate web site. My current layout required adding a lot of extensions to the core folders under /cypress: /support, /plugins & /config. I am motivated in writing a series of posts on Medium, because I like Cypress. Is it perfect, no, some aspects are frustrating. In other words, I have not found a solution yet, but I will continue to use Cypress. From the minimal amount of Cypress development experience, suggestions: a) Switch everything to TypeScript b) Declare Commands within Cypress namespace c) Understand selectors d) Add querying attributes, like 'data-cy', to expedite finding elements. e) Wait for element to appear using 'cy.get(...).contains([ element attributes]).then(...)' f) Avoid if at all possible using XPath for selectors. g) Use 'cy.wait()' to wait for an alias to become available, and not for millisecond delays. h) Use available eslint extensions for Cypress. I will try to get a couple of posts on Medium as soon as possible. They may be helpful. [1]: https://docs.cypress.io/guides/overview/why-cypress.html#In-a-nutshell https://docs.cypress.io/guides/overview/why-cypress.html#In-... [2]: https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests.html#Folder-Structure https://docs.cypress.io/guides/core-concepts/writing-and-org...

swsieber 6y ago

To hop on the unhappy train, it's actually quite hard to get an internal mirror mirror working correctly because the platform for which the extra file it downloads is based on a query string, not the path. Argh.

dpc_pw 6y ago

I very dislike Cypress as well. This even model is sooo confusing, clumsy and error-prone that writing any non-trivial test is a challenge. Whole product uses async/await or Promises and suddenly you have to switch to this baroque model just to write a test. Reusing logic between production code and tests is pretty much impossible. Dynamic pages etc. often confuse cypress. I'm actually not a frontend developer, so I don't have a lot of insight while exactly that is, but I just know that it has been a huge drag. Other than that, it would be OKish. Being able to test the UI is great, but I would advice everyone not to base the whole testing strategy on it. It's just too heavy, slow and unreliable. It doesn't not scale even moderately with the product growing. e2e testing is better done on API level, IMO. You'll want redesigns, changes of UX, and suddenly your changes will require updating tests, which otherwise (if done on the API level) would require no modifications.

themarkn 6y ago
It’s possible (and indeed uh ... recommended) to write frontend tests that are resilient to DOM changes by not depending on things that are not essential to the functionality being tested. Eg select a form field by its label, not by its position in the DOM, and use specific data attributes if there’s no better choice. Then when you gut the HTML for UI overhaul, if the functionality hasn’t changed, the cypress test doesn’t change, you just make sure you attach the attributes to the right new elements. Of course if the functionality changed (like you split something up into 2 pages) yeah you need to update your UI tests cause you are testing a different experience now. But they are UI tests, so yeah. That’s appropriate. I dunno. It sounds like you found other parts of it unwieldy as well, but the idea that the tests are inherently brittle is wrong. Time spent maintaining UI based testing is saved many times over when compared with the manual clicking around that it saves imo.
- eitland 6y ago
  related: testing-framework has been adapted for cypress. It encourages this kind of thinking and testing. I've used cypress at work but have suspended it since we used it end-to-end and it clogged up backend databases.
  - arcturus17 6y ago
    Can’t you set up a test database instead of testing against production? And set up timeouts to control the tempo of the tests if necessary?
    - deleted 6y ago
      [deleted]
- dpc_pw 6y ago
  > Then when you gut the HTML for UI overhaul, if the functionality hasn’t changed, How often website redesigns are only about html? Usually everything changes. New flows, new dialogs, new toolbars, new frameworks. Often the web UI gets and Android UI and iOS UI siblings. Gets replaced altogether by a completely different UI, developed by a different team. APIs change much less often, and can be shared between many UIs. You can also version them. How are you going to version a web UI? Like reddit with "turn back on classic look"? How long can you keep the old UI around just because you have 1000s tests around that use it. > but the idea that the tests are inherently brittle is wrong I don't think so. Tests are great calcifyiers (check my blogpost about it if you want). If you test though your UI, you're calcifying your UI. The more you test though it, the more you calcifying it. No way around it. https://dpc.pw/be-warned-tests-are-great-calcifiers https://dpc.pw/be-warned-tests-are-great-calcifiers

BurningFrog 6y ago

I spent a few weeks with it, and it was very frustrating. The programming model is very odd. I found it next to impossible to share any code between tests, and most of the programming skills I've learned over the decades didn't seem to apply. It is vastly superior to anything else we've seen in stability and polish. If I only could write reasonable code for it, things would be great...

strunz 6y ago
Last time I checked you couldn't even organize tests in different folders, I wonder if they finally fixed that. I always find it odd using test frameworks developed by a for profit company. They pick and choose what code to merge and bugs to fix based on their own priority, not what's best for the users.
- exdsq 6y ago
  I found this when I used Cypress at a contract last month - ended up just using a Makefile `make test` to run every suite.
jpsimons 6y ago
It's like Obamacare -- deeply flawed but hey at least they're trying to do something.
Mavvie 6y ago
What made it hard to reuse code? You can just import/export stuff like normal, unless you mean it was hard to find any logic that could be usefully extracted into a helper?
- BurningFrog 6y ago
  We have a lot of data in UI tables, so I wanted to write code that extracted the cell values to a ROWSxCOLUMNS matrix, that I could reuse in each test. And... I just couldn't figure out a way!

tmcneal 6y ago

If you're willing to try something a bit different, we're working on a no-code approach to test automation that tries to avoid the pitfalls of code-based testing [1]. We're focused on building a E2E testing platform that (1) makes it as easy as possible to create tests and (2) makes tests easy to maintain. Easier said than done but that's our goal. [1] https://reflect.run https://reflect.run

duncan-donuts 6y ago
What are the benefits over selenium ide? It’s been years since I worked heavily on testing but I don’t recall there ever being a good record/playback testing tool. I’m intrigued.
- tmcneal 6y ago
  Selenium IDE uses a Chrome or Firefox extension to record your actions, but they're rewriting it to be an Electron app. I'm not sure about the status of that rewrite though. But the benefits we have vs. what you experienced with Selenium IDE is: - Instead of using an extension to record your actions, there's nothing to install to use Reflect. We spin up an instrumented browser inside a virtual machine and screenshare that with you within our web app. It records your actions and translates those actions into a repeatable test automatically. - Since we have complete control of the test environment, we can do cool things like make recording file uploads really easy or present a nice workflow for getting coverage of visual regressions via screenshot testing. - Our recorder is more accurate :) - For every test you run within Reflect, you get a video synced up to the steps in your test, along with console and network logs from the test run. - Instead of emitting Selenium code that you then maintain like a normal code-based testing tool, we are completely no-code meaning that even updating a test is completely codeless. So for example if you need to make bigger changes to an existing test, we expose a way to re-record only the portions of the test you want to change, and keep the rest as-is.

inglor 6y ago

Yes! I had a very similar experience and I want to really just write regular JavaScript, I mention this in https://www.testim.io/blog/puppeteer-selenium-playwright-cypress-how-to-choose/ https://www.testim.io/blog/puppeteer-selenium-playwright-cyp...

hansworst 6y ago

I've found testcafe [1] to be a good alternative. [1]: https://devexpress.github.io/testcafe/ https://devexpress.github.io/testcafe/

Guillaume86 6y ago
Same experience here, moved from cypress to testcafe after getting frustrated by cypress API. It was some time ago so I may not remember correctly but I felt like cypress was not for people already fluent in javascript, a bit like jquery but pushed even further (you need to do everything the cypress way now, you can't even use variables or promises...).
wdb 6y ago
Doesn't it force you to use some form of page objects? That stopped me from using TestCafe didn't like that
- mstapp 6y ago
  No, page objects (AKA page models) are a recommended but optional pattern in TestCafe. We use them on my team because they make the tests more readable and keep us from having copies of the same selectors in multiple places, but you can easily use TestCafe without them.

yani 6y ago

I am quite surprised. Our experience has been quite positive. We were able to do full coverage of our web app in less than a month without prior experience with cypress. This is what we learned from it: https://blog.servmask.com/8-tricks-i-learned-from-cypress/ https://blog.servmask.com/8-tricks-i-learned-from-cypress/

foolfoolz 6y ago

it took a lot of time but i do like the cypress approach. there are weird things like how you can’t get the current time because all lines execute immediately and are evaluated later. but once you get your handle on .then and .wrap it is good. they should have picked a different name than .then because it’s not a promise but looks like one

varrock 6y ago

I really enjoyed the testing library when I used it. I recently came back to it and felt like they removed a lot of the documentation that I liked about the library. I'm not quite sure why they did, but it was very helpful to see their examples for integrating with common third party libraries.

crb002 6y ago

It is visually appealing but sluggish and sucks at non-standard authentication to the point of not being usable. Also no out of the box way to run more than one instance with incognito tabs to have communication to test multi-user bugs. Perhaps radical, but I think it should support hooks with Playwright/Puppeteer that you can hand control back and forth to as a safety valve just like being able to shell out to the command line when your favorite programming language doesn't have a library for what you need. The difference here is that the browser context needs to be passed - that means some janky handoff (which with crypto will be a PITA) or using the Playwright/Puppeter connector under the hood.

jeff00seattle 6y ago

I have been developing tests using Cypress for 6+ months. I have used multiple other variations of test frameworks that were built on the Selenium WebDriver protocol. The official documentation [1] states that Cypress differs fundamentally and architecturally as compared to Selenium. Admittedly I found Cypress web testing framework frustrating initially, and yes, I have performed many re-writes because Cypress documentation behind command chaining and assertion queuing to be not intuitive. Thanks for my management's support, perseverance afforded me with the stubbornness with a purpose of checking out Cypress if it has a better DOM manipulation, how it operates directly in the browser, interactive test runner, and its API and configuration. It required 3+ months of perseverance in getting the Cypress tests organized [2] for handling web testing core functionality of a commercial-grade corporate web site. My current layout required adding a lot of extensions to the core folders under /cypress: /support, /plugins & /config. I am motivated in writing a series of posts on Medium, because I like Cypress. Is it perfect, no, some aspects are frustrating. In other words, I have not found a solution yet, but I will continue to use Cypress. From the minimal amount of Cypress development experience, suggestions: a) Switch everything to TypeScript b) Declare Commands within Cypress namespace c) Understand selectors d) Add querying attributes, like 'data-cy', to expedite finding elements. e) Wait for element to appear using 'cy.get(...).contains([ element attributes]).then(...)' f) Avoid if at all possible using XPath for selectors. g) Use 'cy.wait()' to wait for an alias to become available, and not for millisecond delays. h) Use available eslint extensions for Cypress. I will try to get a couple of posts on Medium as soon as possible. They may be helpful. [1]: https://docs.cypress.io/guides/overview/why-cypress.html#In-a-nutshell https://docs.cypress.io/guides/overview/why-cypress.html#In-... [2]: https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests.html#Folder-Structure https://docs.cypress.io/guides/core-concepts/writing-and-org...