4 ms·
When it comes to browser automation tests, everywhere I've worked at suffered from intermittently failing tests. When I was at the Ministry of Justice (UK), I c
by umaar 6y ago
When it comes to browser automation tests, everywhere I've worked at suffered from intermittently failing tests. When I was at the Ministry of Justice (UK), I configured CircleCI to run tests hundreds of times [1] (over a number of days) through cron jobs. This allowed me to reflect on all test results, and find out what failed most often and eventually solve those root causes. This strategy worked well.
Interestingly enough, just today I posted a GitHub thread [2] and asked the community to 'thumbs up' the video course they'd like for me to create. "Learn Browser Automation" is currently the highest voted. If it's the one I end up making, a huge focus will absolutely be on: How to reduce test flakiness with headless browsers.
Words of advice, avoiding sleep() and other brittle bits of code will help. But in addition, run your tests frequently to catch out the flakiness early. Invest in tooling which helps you diagnose failing tests (screenshots, devtools trace dumps). Configure something like VNC so you can remotely connect to the machine running the test.
[1] https://github.com/ministryofjustice/fb-automated-tests/blob/7c9cee58db902419abf5449aaef6e91e575502d9/.circleci/config.yml#L52 https://github.com/ministryofjustice/fb-automated-tests/blob...
[2] https://github.com/umaar/dev-tips-tracker/issues/33 https://github.com/umaar/dev-tips-tracker/issues/33