3 ms·

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best rel

by jmickey 11y ago

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browser-tests http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

crdoconnor 11y ago
>the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. What was actually going wrong during that 10%? I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this. Do you make heavy use of sleeps?
- jmickey 11y ago
  Mostly these would be cases where the browser would, seemingly at random, end up in an unpredictable state and all proceeding test scenarios would fail because of this. (Page is white, or a completely unrelated website gets opened. We have seen lots of weird situations so far) This might be exacerbated by the fact that we use the remote Browserstack Selenium hosting service so that the tests can be executed automatically as a part of our deployment process.
  - crdoconnor 11y ago
    White page and randomly ending up on an unrelated website both sound like bugs.
omouse 11y ago
>tests that run for 40 minutes This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.
- stevebmark 11y ago
  40 minutes from clicking a button to deploy is actually abysmal, especially when you need to worry about things like rollbacks, or deploying at the end of the day, or releasing quick hotfixes to users. In modern build processes, even 10 minutes seems too long.

by jmickey 11y ago

crdoconnor 11y ago

>the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. What was actually going wrong during that 10%? I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this. Do you make heavy use of sleeps?

jmickey 11y ago
Mostly these would be cases where the browser would, seemingly at random, end up in an unpredictable state and all proceeding test scenarios would fail because of this. (Page is white, or a completely unrelated website gets opened. We have seen lots of weird situations so far) This might be exacerbated by the fact that we use the remote Browserstack Selenium hosting service so that the tests can be executed automatically as a part of our deployment process.
- crdoconnor 11y ago
  White page and randomly ending up on an unrelated website both sound like bugs.

omouse 11y ago

>tests that run for 40 minutes This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.

stevebmark 11y ago
40 minutes from clicking a button to deploy is actually abysmal, especially when you need to worry about things like rollbacks, or deploying at the end of the day, or releasing quick hotfixes to users. In modern build processes, even 10 minutes seems too long.