4 ms·

> If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our bu

by renke1 6y ago

> If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.) Would you say the (slow and unreliable) Cypress tests are worth it still?

dzhiurgis 6y ago
That's the nature of UI tests for most part. IIRC Cypress are written declarative tools which would make them even more unreliable and slow, albeit easier to fix. Personally I've recently started using Playwright and I'm quite happy with it. There was occasional misunderstanding of their API, but 95% of time it's great. Microsoft is kicking butt these days.
mcv 6y ago
I don't know. We need some sort of e2e tests, and all e2e test frameworks are terrible in one way or another. Cypress is okay. I would prefer to only run it on production or the dev server and have alarms go off when they fail, but either the requirement is, or other developers have decided that it's necessary to pass all e2e tests before a feature branch can be merged into the master branch. And I get the reason for it; you don't want to accidentally merge breaking changes. But it does make our build pipelines very slow and unreliable. So are they worth it? I don't know. If I had my way, we'd only run them on master, and not make it a requirement for feature branches to pass them. Because if you fix one tiny thing, you now have to wait 15 minutes again for the Cypress tests to run. I think they'd be better in a different setup than what we're doing.
- pavon 6y ago
  We had similar issues with integration tests, and made them a separate jenkins job that didn't trigger automatically, but gitlab was still configured to require them to pass for merge. We would kick it off manually only after all other code review was complete. Then the only cases where we had to re-run it were the same cases where it would have failed in master if we only ran the test there, but it saved us the hassle of reverting or feeling pressured to get hotfixes into master quickly.
- smaps 6y ago
  Check out https://reflect.run/ https://reflect.run/ as a replacement for Cypress. I started using it recently to do E2E testing at work in our staging environment to run a suite of tests before we move anything to production. So far it's been great and has saved a couple of releases in a month or so of use!
holtalanm 6y ago
Cypress is horribly unreliable. We used to use it, and tests would pass, then fail on subsequent runs with no code changes, due to internal bugs within Cypress screenshot plugins, if I remember right. I have no idea if it is any better now, but we dropped it about 6 months ago in favor of pure Selenium C# for our UI tests. edit: a word

by renke1 6y ago

dzhiurgis 6y ago

That's the nature of UI tests for most part. IIRC Cypress are written declarative tools which would make them even more unreliable and slow, albeit easier to fix. Personally I've recently started using Playwright and I'm quite happy with it. There was occasional misunderstanding of their API, but 95% of time it's great. Microsoft is kicking butt these days.

mcv 6y ago

I don't know. We need some sort of e2e tests, and all e2e test frameworks are terrible in one way or another. Cypress is okay. I would prefer to only run it on production or the dev server and have alarms go off when they fail, but either the requirement is, or other developers have decided that it's necessary to pass all e2e tests before a feature branch can be merged into the master branch. And I get the reason for it; you don't want to accidentally merge breaking changes. But it does make our build pipelines very slow and unreliable. So are they worth it? I don't know. If I had my way, we'd only run them on master, and not make it a requirement for feature branches to pass them. Because if you fix one tiny thing, you now have to wait 15 minutes again for the Cypress tests to run. I think they'd be better in a different setup than what we're doing.

pavon 6y ago
We had similar issues with integration tests, and made them a separate jenkins job that didn't trigger automatically, but gitlab was still configured to require them to pass for merge. We would kick it off manually only after all other code review was complete. Then the only cases where we had to re-run it were the same cases where it would have failed in master if we only ran the test there, but it saved us the hassle of reverting or feeling pressured to get hotfixes into master quickly.
smaps 6y ago
Check out https://reflect.run/ https://reflect.run/ as a replacement for Cypress. I started using it recently to do E2E testing at work in our staging environment to run a suite of tests before we move anything to production. So far it's been great and has saved a couple of releases in a month or so of use!

holtalanm 6y ago

Cypress is horribly unreliable. We used to use it, and tests would pass, then fail on subsequent runs with no code changes, due to internal bugs within Cypress screenshot plugins, if I remember right. I have no idea if it is any better now, but we dropped it about 6 months ago in favor of pure Selenium C# for our UI tests. edit: a word