4 ms·

Anyone have experience with successfully doing UI tests (especially screenshot tests) in a very young software product? We try to keep our "plumbing" code and b

by kevsim 6y ago

Anyone have experience with successfully doing UI tests (especially screenshot tests) in a very young software product? We try to keep our "plumbing" code and backend code well tested, but the UI is changing all of the time and I'm quite worried it'd be a nightmare to maintain UI tests at this stage.

whycombagator 6y ago
> screenshot tests > the UI is changing all of the time Far from an expert in UI tests, but I’m not sure the former is possible given the latter. > it'd be a nightmare to maintain UI tests at this stage. IMO that is likely.
jdlshore 6y ago
You can use something like Quixote. (Disclaimer: I wrote it.) Screenshot tests are always going to be brittle.
jluxenberg 6y ago
It's not quite the same as screenshot testing, but I've had success using snapshot testing in React. The idea is outlined here: https://jestjs.io/docs/en/snapshot-testing https://jestjs.io/docs/en/snapshot-testing Snapshots of the intermediate representation (e.g. JSX in React) will be less brittle than e.g. a screenshot of pixels. It also works best when your UI is a function of some easily-constructed state. Other frameworks sharing these features would also work well with snapshot testing.
- harlanlewis 6y ago
  In the same vein, I'm a fan of Storybook [1], which uses Jest under the hood [2], paired with Chromatic [3]. Chromatic generates screenshots from component stories for visual change detection with an easy to use and share web UI. It's neat. [1] https://storybook.js.org https://storybook.js.org [2] https://storybook.js.org/docs/testing/structural-testing/ https://storybook.js.org/docs/testing/structural-testing/ [3] https://www.chromatic.com https://www.chromatic.com
hugs 6y ago
(Selenium project founder here.) Test maintenance cost is a sane thing to be worried about! Early on, there can be lots of code thrashing and UI changes. Better to keep the UI tests to a minimum at that stage - have them act more like smoke alarms. You only need a few alarms in the right places to tell you that something is wrong. Other general advice is that in a young product, it's often more important to make sure you're building the right thing. ("Right" is subjective, but could be defined as "has users" or "makes money".) Having a well-tested product that has no users and/or makes no money is an even bigger nightmare than UI test maintenance. Once you know you've built the "right" thing (hopefully sooner than later), you can afford to invest more in all kinds of test automation (not just UI tests).
- The_rationalist 6y ago
  By curiosity, what do you think about selenium vs https://github.com/microsoft/playwright https://github.com/microsoft/playwright vs cypress?
bickeringyokel 6y ago
I use webdriverio, which has pretty straightforward screenshot plugins. Honestly early on it's better to focus on API testing. My advice is start small for the UI and focus on a few critical components like page routing or features that would be a show stopper if they broke. I don't think screen shot tests will be of great value if your product is not very mature. When I first started I made the mistake of blindly following my predecessors user stories that weren't particularly well thought out.

by kevsim 6y ago

whycombagator 6y ago

> screenshot tests > the UI is changing all of the time Far from an expert in UI tests, but I’m not sure the former is possible given the latter. > it'd be a nightmare to maintain UI tests at this stage. IMO that is likely.

jdlshore 6y ago

You can use something like Quixote. (Disclaimer: I wrote it.) Screenshot tests are always going to be brittle.

jluxenberg 6y ago

It's not quite the same as screenshot testing, but I've had success using snapshot testing in React. The idea is outlined here: https://jestjs.io/docs/en/snapshot-testing https://jestjs.io/docs/en/snapshot-testing Snapshots of the intermediate representation (e.g. JSX in React) will be less brittle than e.g. a screenshot of pixels. It also works best when your UI is a function of some easily-constructed state. Other frameworks sharing these features would also work well with snapshot testing.

harlanlewis 6y ago
In the same vein, I'm a fan of Storybook [1], which uses Jest under the hood [2], paired with Chromatic [3]. Chromatic generates screenshots from component stories for visual change detection with an easy to use and share web UI. It's neat. [1] https://storybook.js.org https://storybook.js.org [2] https://storybook.js.org/docs/testing/structural-testing/ https://storybook.js.org/docs/testing/structural-testing/ [3] https://www.chromatic.com https://www.chromatic.com

hugs 6y ago

(Selenium project founder here.) Test maintenance cost is a sane thing to be worried about! Early on, there can be lots of code thrashing and UI changes. Better to keep the UI tests to a minimum at that stage - have them act more like smoke alarms. You only need a few alarms in the right places to tell you that something is wrong. Other general advice is that in a young product, it's often more important to make sure you're building the right thing. ("Right" is subjective, but could be defined as "has users" or "makes money".) Having a well-tested product that has no users and/or makes no money is an even bigger nightmare than UI test maintenance. Once you know you've built the "right" thing (hopefully sooner than later), you can afford to invest more in all kinds of test automation (not just UI tests).

The_rationalist 6y ago
By curiosity, what do you think about selenium vs https://github.com/microsoft/playwright https://github.com/microsoft/playwright vs cypress?

bickeringyokel 6y ago

I use webdriverio, which has pretty straightforward screenshot plugins. Honestly early on it's better to focus on API testing. My advice is start small for the UI and focus on a few critical components like page routing or features that would be a show stopper if they broke. I don't think screen shot tests will be of great value if your product is not very mature. When I first started I made the mistake of blindly following my predecessors user stories that weren't particularly well thought out.