5 ms·

I see these kind of tools once in a while and it brings out my jaded side. About 15 years ago I recall some considerable effort was spent at a company I worked

I see these kind of tools once in a while and it brings out my jaded side. About 15 years ago I recall some considerable effort was spent at a company I worked at for the Quality Engineering team to build a visual regression tool for a video game UI I was working on. Using some Windows api magic it could detect buttons and click them to navigate through a UI and then take a screenshot to do a visual diff. It was the most broken useless thing ever and after 2 or 3 months of development it was scrapped. Last place I worked had a team that built a similar system using Selenium and some image diff. It worked 90% of the time. It was even integrated into their CI pipeline and the system would email you when it failed. You could make a change to one area of code and get an email from the system for a completely separate area that for whatever random reason failed. I once submitted some code to that project and when I got the email I asked one of the project maintainers what I should do to fix it. He told me to just ignore it. Their process was to check the output of the tool on those failures to see if there was any legitimate problem. When I checked the output myself a large amount of the output was garbage (about 100 application states tested and at least 10 in a garbled state). My own team even tried to integrate Selenium and we even found an external vendor where you can ship your Selenium tests along with your app and they will run it against a matrix of browsers of various versions. We barely got a Chrome version running - no hope of Safari, Firefox or IE/Edge. It was a blackhole of time just fighting to get an equivalent of hello world running consistently as a test in all browsers. One day someone will prove me wrong and get this kind of UI testing working reliably. But after nearly 20 years seeing optimistic people die on this hill - I do not support wasting more time on it.

Dowwie 7y ago
People have moved on to headless Chrome
- paulryanrogers 7y ago
  Webdriver can still work with headless modes. Firefox too has a headless mode. And they're simpler than the old XVFB.
- zelly 7y ago
  Puppeteer (js) in particular. But since it's based on protobuf rpc, you can use the underlying protocol (chrome devtools protocol) from any language. It's by far the leading player in this space and for good reason. Selenium is unreliable and too generic.
  - Dowwie 7y ago
    There are puppeteer clones for many languages now. It's impressive how successful the devtools protocol has been.
beatthatflight 7y ago
I've done a ton of selenium, appium and winappdriver (kinda like selenium) in Java, C# and python, as a contractor at at least 7 different companies. Every company experienced these issues. Selenium is great. It really is. But the level of flaky tests that generally get produced is just so painful. And as you say, getting it to run in Firefox - that's not been too bad, but any other browser as well was a pipe(line) dream. IEDriver last time I tried to use it with Grid only allowed one instance anyway, so couldn't run multiple on one machine. UIPath, AutomationAnywhere and these other 'no code required but we have a place you can write code in case' tools - I'm looking forward to seeing how they deal with these sorts of problems. When you're scraping data off a dynamically loading page, or is time dependent, it's not trivial. Even with implicit waits, retries and other 'tricks' as someone described further down, it's frustratingly difficult at times to just do a single action in an app/page. And then you hit shadow dom websites, or Windows apps that create new windows (not child popups) for every screen/dialog box. The joy :) Still, I find it an enjoyable, if frustrating challenge.
jonperl 7y ago
I have had the same experience using Selenium but not with Chrome Dev Tools Protocol. However you still need a stable environment, trustable selectors, and automatic waiting logic to have stable e2e web tests. I am working on an open source library that generates Playwright tests (uses Chrome Dev Tools protocol) and I hope we can prove you wrong about getting UI tests working reliably. https://github.com/qawolf/qawolf https://github.com/qawolf/qawolf
mintzworld 7y ago
SeleniumBase methods wait for page elements to fully load before interacting with them, which brings test reliability. There's also built-in mobile testing and cross-browser testing features included: https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/mobile_testing.md https://github.com/seleniumbase/SeleniumBase/blob/master/hel...
zelly 7y ago
The unspoken subtext is that development on these tools is intended for building spammers. Automated testing is just the cover story like a torrent client putting Big Buck Bunny in the screenshot.

Dowwie 7y ago

People have moved on to headless Chrome

paulryanrogers 7y ago
Webdriver can still work with headless modes. Firefox too has a headless mode. And they're simpler than the old XVFB.
zelly 7y ago
Puppeteer (js) in particular. But since it's based on protobuf rpc, you can use the underlying protocol (chrome devtools protocol) from any language. It's by far the leading player in this space and for good reason. Selenium is unreliable and too generic.
- Dowwie 7y ago
  There are puppeteer clones for many languages now. It's impressive how successful the devtools protocol has been.

beatthatflight 7y ago

I've done a ton of selenium, appium and winappdriver (kinda like selenium) in Java, C# and python, as a contractor at at least 7 different companies. Every company experienced these issues. Selenium is great. It really is. But the level of flaky tests that generally get produced is just so painful. And as you say, getting it to run in Firefox - that's not been too bad, but any other browser as well was a pipe(line) dream. IEDriver last time I tried to use it with Grid only allowed one instance anyway, so couldn't run multiple on one machine. UIPath, AutomationAnywhere and these other 'no code required but we have a place you can write code in case' tools - I'm looking forward to seeing how they deal with these sorts of problems. When you're scraping data off a dynamically loading page, or is time dependent, it's not trivial. Even with implicit waits, retries and other 'tricks' as someone described further down, it's frustratingly difficult at times to just do a single action in an app/page. And then you hit shadow dom websites, or Windows apps that create new windows (not child popups) for every screen/dialog box. The joy :) Still, I find it an enjoyable, if frustrating challenge.

jonperl 7y ago

I have had the same experience using Selenium but not with Chrome Dev Tools Protocol. However you still need a stable environment, trustable selectors, and automatic waiting logic to have stable e2e web tests. I am working on an open source library that generates Playwright tests (uses Chrome Dev Tools protocol) and I hope we can prove you wrong about getting UI tests working reliably. https://github.com/qawolf/qawolf https://github.com/qawolf/qawolf

mintzworld 7y ago

SeleniumBase methods wait for page elements to fully load before interacting with them, which brings test reliability. There's also built-in mobile testing and cross-browser testing features included: https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/mobile_testing.md https://github.com/seleniumbase/SeleniumBase/blob/master/hel...

zelly 7y ago

The unspoken subtext is that development on these tools is intended for building spammers. Automated testing is just the cover story like a torrent client putting Big Buck Bunny in the screenshot.