6 ms·

Consistent Selenium Testing in Python

61 points by cdubzzz 9y ago

andrewalan 9y ago
I am Andrew Alan Escher, an honest and loyal hacker with vouches to proof validity of jobs.. I major in... Credit score increase and decrease Upgrade University Grades Facebook, Instagram, Twitter, WhatsApp, Line, Skype Hack Delete unwanted online Pictures and Videos on any website Remove Criminal Records Hack bank accounts Apps hacking Loading all MasterCard, Bank Accounts, PayPal, Bitcoin, WU, Money Gram with untraceable credit on it. etc. We also develop hacked facebook, twitter, instagram, yahoo, gmail passwords etc. Do you need to keep an eye on your spouse by gaining access to their emails? As a parent do you want to know what your kids do on a daily basis on social networks. We have 100% track records from our client as well as highest return hire rate. Best Online hacker with 100% guarantee and money back return policy. Email: andrewalanhacks@gmail.com Call/Text: +1 (323)835-1741
alistproducer2 9y ago
I've been working with selenium a lot at work recently. There are lots of little nuances to grok if you want your tests to run completely deterministic: especially on a JavaScript-heavy site.
- danidiaz 9y ago
  I try to use explicit waits whenever possible. Luckily, the Java client library provides a rich set of combinators for declaring them. You can also define your own; they are basically functions from a WebDriver object to some other thing (usually a Boolean). http://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/support/ui/ExpectedConditions.html http://seleniumhq.github.io/selenium/docs/api/java/org/openq...
  - cdubzzz 9y ago
    Oh wow, Java gives you a lot! Python's are actually pretty decent as well, I just came to rely almost entirely on "presence_of_element_located" with our Javascript-heavy app.
- hinkley 9y ago
  Have you ever gotten completely deterministic tests? I'm starting to think it may not be possible.
  - thevalermor 9y ago
    Indeed, it is possible but requires some effort. Look at this project https://github.com/Skyscanner/pages https://github.com/Skyscanner/pages. Choosing carefully your traits will do the trick.
stephen 9y ago
FWIW the most consistent (as in non-flaky) selenium tests I've worked with used hidden DOM elements to synchronize the selenium driver with the JS code: http://www.draconianoverlord.com/2011/10/14/sane-selenium-testing.html http://www.draconianoverlord.com/2011/10/14/sane-selenium-te... Basically every JS ajax call (or JS promise, though we didn't use any of those at the time) would update a hidden DOM element's inner text, e.g. the `#pendingAjaxRequests` inner text would go to 1, or 2, then back to 0 when the AJAX response came back (ideally you do this instrumentation in a single place in your app). Then on the selenium side, after every button click/other action, we'd wait until those counters went back down to zero. This post-action waiting (waiting for the app to "settle" after invoking actions) vs. pre-action waiting (...well, this the element I want to click on available yet? Hm...no...) was very reliable.
- gowan 9y ago
  sounds like an interesting pattern. it would be nice to have a browser extension that abstracts the pattern in the browser.
- cdubzzz 9y ago
  Interesting! Thanks for the reference. I think all the waiting I have dealt with is server (vs. animation) as well. I'll have to play around with this idea...
- whatnotests 9y ago
  That's not how it's suppose to work. If your program can tell when something happens then your user should be able to as well. Please, let's not consider this to be a positive design - rather this is smell. Check your UX for usability, then proceed with behavior testing.
- zootam 9y ago
  >There are a variety of ways of doing this, but you basically need a choke point in your application where all AJAX requests go through. Does wrapping the XMLHTTPrequest object to get this functionality still count as sane?
noir_lord 9y ago
I wrote a shim layer over selenium using pythons test infrastructure so I can do self.clickPseudoButton('foo') self.setInputById('fizz', 'buzz') It also maps common operations against the structure of my legacy app, its way better than using the lower level API. Tests look like pseudo code and are way more grokkable.
- cdubzzz 9y ago
  Your code available anywhere? (:
  - noir_lord 9y ago
    It's nothing that impressive but I'll post it on Monday when I'm at work, it's really just a bunch of helper methods to reduce boilerplatr.
    - cdubzzz 9y ago
      Yeah, I would just be curious to see how you organized. I kept meaning to step back and work on abstracting things but always got side tracked with other stuff. It gets real long and messy without that.
gowan 9y ago
you should skip this post. it is full of anti-patterns. anti-patterns (1) implicit wait. this will create subtle differences in behavior between drivers. it will also create long pauses when you test for negative conditions. (2) clear. selenium has built in support for clear[1]. in addition to clear you can send the null key[2] if you want to clear the input midway through a sequence of characters. (3) time wait. this does not make any sense to me. seems like a clever way to add time.sleep. [1] https://www.w3.org/TR/webdriver/#element-clear https://www.w3.org/TR/webdriver/#element-clear [2] https://www.w3.org/TR/webdriver/#element-send-keys https://www.w3.org/TR/webdriver/#element-send-keys
- cdubzzz 9y ago
  I agree that implicit wait is something of a time waster. Regarding clear, that does appear to be exactly what the Python implementation does[0], but in my experience it just seemed to fail at random. If I remember correctly it worked most of the time with geckodriver, and then hardly ever with chromedriver. Really not sure what exactly the deal is with that... And yes the time wait is a little pointless (: [0] https://github.com/SeleniumHQ/selenium/blob/master/py/selenium/webdriver/remote/remote_connection.py#L266 https://github.com/SeleniumHQ/selenium/blob/master/py/seleni...
  - Elhana 9y ago
    clear() works for me consistently. At the same time, I had cases before when sendkeys was not sending complete string, so I had to compare it with get_attribute('value') to be sure.
_pRwn_ 9y ago
... and when you're done with this beginner stuff ... use the de facto industry standard http://robotframework.org/ http://robotframework.org/
yebyen 9y ago
The best thing you can do if you are developing Selenium tests in ruby (but these lessons are probably applicable no matter what environment you're using) is learn how they changed Capybara. There was a big shift in Capybara around the time that "should" went out of the vocabulary. What that means syntactically for developers is less important than the semantic change that came with this shift. Tons of the old cheat sheets are still based on the old syntax, and many of them if you dig deep will also give the bad advice! So many people try browser testing and give up because in some ways it's hard to make the testing consistent. Especially if you use onClick events that load another page, it's so easy to forget that you might or might not yet have loaded that page when the next line of code runs! That browser-side JavaScript has made your integration now multi-threaded. The hardest one to figure out is when you grab a reference to something on the current page that looks like the element you want, but it's on the previous page... it's not what you want. Then when you get around to trying to finding nodes within it, or clicking on that node. You've loaded the page you want, but your reference is pointing to a node that is no longer present in the active DOM, because you're on the new page now! If you're composing reusable steps (like in Cucumber) Always find a way to make sure you've already loaded the DOM of the page you think you're on, before you get any references to nodes on the page. Even if you have to put #some-target-page-node on the target page so you can expect(page).to have_selector('#some-target-page-node') That will prove you've landed on the target page, and if you find it sometimes takes a long time, or if it's longer than the timeout, set 'wait' to a longer number of seconds: expect(page).to have_selector('.some-node', wait: 90) If your page has waits that long of course, it's most likely going to start affecting your conversion rates, so do something about that...
- yebyen 9y ago
  To clarify... set 'wait' to a longer number of seconds: expect(page).to have_selector('.some-node', wait: 90) If your page has waits that long of course Wait is different than sleep. Wait retries until the condition is met. This makes it a little bit confusing for "negative presence" tests where you are testing for the absence of a thing, but it ensures that you are not waiting a full 90 seconds for a thing to happen, when that thing has already happened. This is the greatest, most noticeable effect of the "shift." No more "until". This is something like explicit/implicit waiting described by the OP's article, although Capybara is careful now to avoid "wait until" you can still get this behavior using expect statements as I showed above. It's much cleaner in capybara[1] nowadays, and I'm actually surprised people work with Selenium WebDriver directly in Python by comparison. You can also wait explicitly for the AJAX events to complete, although I never do this[2] because it seems to introduce a new source of error: the AJAX completes before we get around to observing that it had started. This article seems to provide a bullet-proof way to do it though, upon review, because it's simply counting with JS variables in the DOM. You can even start multiple AJAX requests at once and wait for all of them to complete, or just go ahead and move on if you can see they already completed when you first check. No waiting. This is more reliable than adding and removing the ".ajax-processing" class to a hidden element, which you must be careful to observe so you know your AJAX has started, and then observe again as it is removed so you know it has completed. The method in [2] is a bit less "semaphore-ish" and looks more reliable than ways I've used. Another nice trick is to remove another source of threaded racing: your animations. If you have a test that triggers an animation and you want to disable animations, run once you get to the page: page.execute_script("$.fx.off=true") # or jquery.fx.off=true This runs on the DOM though, so it must be run on every page that loads if it is going to be effective. There are also ways[3][4] you can get this script to run on every page rather than running it manually just wherever needed, I haven't tried this, but I'd recommend it if you don't want to spend time by hand tracing down the source of each delay. I feel like there were about 5 things I had to learn to make my tests reliable, and I am still applying them daily. But once I had the experience of "tracking down the Heisenbug" enough times, I can avoid writing 98% of the bad tests, and tracking down the remaining 2% when they inevitably show up in my test suites as "failed once, then failure went away with no code changes" it simply hasn't been hard to track down the source of the failure anymore, as I now know really well what I'm looking for, what causes this type of failure. I personally got a bit out of this article, but I'm not sure anyone is going to read this article and just know better how to do browser testing. I think you have to go through this experience of accidentally writing sometimes-unreliable tests, and then figuring out how to fix them. I did not learn without the help of articles like this, though. The last great advice I found was to learn to mock API responses if your application uses external APIs, but don't over-rely on mocks. We have lots of tests that do hit our APIs and will fail if they are down. But we also have tests that depend on some difficult to replicate condition being met by the API servers. Those mock tests in some cases are more dangerously brittle than the tests that actually hit the API. My favorite was added recently, the test that shows what happens if the API goes down in the middle of your overnight job. Hint: You want the job to stop running at the first sign that the API has gone down, and trigger alert mail or otherwise signal an error that you can find later. With WebMock, this test was easy to write! Just send back a 500 error at a time when you know you weren't expecting it. You can usually use factories to replicate other less unusual conditions, but when your app depends heavily on external APIs, this trick can help greatly to simplify your tests. This particular test would have been impossible to write without mocks. I would not mock more than 50% of API tests though, because it is equally valuable to find out when an unrelated change that is deployed through your CI environment to one of your APIs, will break some part of your application unexpectedly. This is why I do not mock 100% of API-related tests. [1]: https://www.varvet.com/blog/why-wait_until-was-removed-from-capybara/ https://www.varvet.com/blog/why-wait_until-was-removed-from-... [2]: http://cabeca.github.io/blog/2013/06/16/waiting-for-completed-ajax-in-capybara-and-other-tricks/ http://cabeca.github.io/blog/2013/06/16/waiting-for-complete... [3]: https://makandracards.com/coffeeandcode/7503-disable-jquery-animations-during-rails-tests https://makandracards.com/coffeeandcode/7503-disable-jquery-... [4]: https://gist.github.com/keithtom/8763169 https://gist.github.com/keithtom/8763169

61 points by cdubzzz 9y ago

I am Andrew Alan Escher, an honest and loyal hacker with vouches to proof validity of jobs.. I major in... Credit score increase and decrease Upgrade University Grades Facebook, Instagram, Twitter, WhatsApp, Line, Skype Hack Delete unwanted online Pictures and Videos on any website Remove Criminal Records Hack bank accounts Apps hacking Loading all MasterCard, Bank Accounts, PayPal, Bitcoin, WU, Money Gram with untraceable credit on it. etc. We also develop hacked facebook, twitter, instagram, yahoo, gmail passwords etc. Do you need to keep an eye on your spouse by gaining access to their emails? As a parent do you want to know what your kids do on a daily basis on social networks. We have 100% track records from our client as well as highest return hire rate. Best Online hacker with 100% guarantee and money back return policy. Email: andrewalanhacks@gmail.com Call/Text: +1 (323)835-1741

alistproducer2 9y ago

I've been working with selenium a lot at work recently. There are lots of little nuances to grok if you want your tests to run completely deterministic: especially on a JavaScript-heavy site.

danidiaz 9y ago
I try to use explicit waits whenever possible. Luckily, the Java client library provides a rich set of combinators for declaring them. You can also define your own; they are basically functions from a WebDriver object to some other thing (usually a Boolean). http://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/support/ui/ExpectedConditions.html http://seleniumhq.github.io/selenium/docs/api/java/org/openq...
- cdubzzz 9y ago
  Oh wow, Java gives you a lot! Python's are actually pretty decent as well, I just came to rely almost entirely on "presence_of_element_located" with our Javascript-heavy app.
hinkley 9y ago
Have you ever gotten completely deterministic tests? I'm starting to think it may not be possible.
- thevalermor 9y ago
  Indeed, it is possible but requires some effort. Look at this project https://github.com/Skyscanner/pages https://github.com/Skyscanner/pages. Choosing carefully your traits will do the trick.

stephen 9y ago

FWIW the most consistent (as in non-flaky) selenium tests I've worked with used hidden DOM elements to synchronize the selenium driver with the JS code: http://www.draconianoverlord.com/2011/10/14/sane-selenium-testing.html http://www.draconianoverlord.com/2011/10/14/sane-selenium-te... Basically every JS ajax call (or JS promise, though we didn't use any of those at the time) would update a hidden DOM element's inner text, e.g. the `#pendingAjaxRequests` inner text would go to 1, or 2, then back to 0 when the AJAX response came back (ideally you do this instrumentation in a single place in your app). Then on the selenium side, after every button click/other action, we'd wait until those counters went back down to zero. This post-action waiting (waiting for the app to "settle" after invoking actions) vs. pre-action waiting (...well, this the element I want to click on available yet? Hm...no...) was very reliable.

gowan 9y ago
sounds like an interesting pattern. it would be nice to have a browser extension that abstracts the pattern in the browser.
cdubzzz 9y ago
Interesting! Thanks for the reference. I think all the waiting I have dealt with is server (vs. animation) as well. I'll have to play around with this idea...
whatnotests 9y ago
That's not how it's suppose to work. If your program can tell when something happens then your user should be able to as well. Please, let's not consider this to be a positive design - rather this is smell. Check your UX for usability, then proceed with behavior testing.
zootam 9y ago
>There are a variety of ways of doing this, but you basically need a choke point in your application where all AJAX requests go through. Does wrapping the XMLHTTPrequest object to get this functionality still count as sane?

noir_lord 9y ago

I wrote a shim layer over selenium using pythons test infrastructure so I can do self.clickPseudoButton('foo') self.setInputById('fizz', 'buzz') It also maps common operations against the structure of my legacy app, its way better than using the lower level API. Tests look like pseudo code and are way more grokkable.

cdubzzz 9y ago
Your code available anywhere? (:
- noir_lord 9y ago
  It's nothing that impressive but I'll post it on Monday when I'm at work, it's really just a bunch of helper methods to reduce boilerplatr.
  - cdubzzz 9y ago
    Yeah, I would just be curious to see how you organized. I kept meaning to step back and work on abstracting things but always got side tracked with other stuff. It gets real long and messy without that.

gowan 9y ago

you should skip this post. it is full of anti-patterns. anti-patterns (1) implicit wait. this will create subtle differences in behavior between drivers. it will also create long pauses when you test for negative conditions. (2) clear. selenium has built in support for clear[1]. in addition to clear you can send the null key[2] if you want to clear the input midway through a sequence of characters. (3) time wait. this does not make any sense to me. seems like a clever way to add time.sleep. [1] https://www.w3.org/TR/webdriver/#element-clear https://www.w3.org/TR/webdriver/#element-clear [2] https://www.w3.org/TR/webdriver/#element-send-keys https://www.w3.org/TR/webdriver/#element-send-keys

cdubzzz 9y ago
I agree that implicit wait is something of a time waster. Regarding clear, that does appear to be exactly what the Python implementation does[0], but in my experience it just seemed to fail at random. If I remember correctly it worked most of the time with geckodriver, and then hardly ever with chromedriver. Really not sure what exactly the deal is with that... And yes the time wait is a little pointless (: [0] https://github.com/SeleniumHQ/selenium/blob/master/py/selenium/webdriver/remote/remote_connection.py#L266 https://github.com/SeleniumHQ/selenium/blob/master/py/seleni...
- Elhana 9y ago
  clear() works for me consistently. At the same time, I had cases before when sendkeys was not sending complete string, so I had to compare it with get_attribute('value') to be sure.

_pRwn_ 9y ago

... and when you're done with this beginner stuff ... use the de facto industry standard http://robotframework.org/ http://robotframework.org/

yebyen 9y ago

The best thing you can do if you are developing Selenium tests in ruby (but these lessons are probably applicable no matter what environment you're using) is learn how they changed Capybara. There was a big shift in Capybara around the time that "should" went out of the vocabulary. What that means syntactically for developers is less important than the semantic change that came with this shift. Tons of the old cheat sheets are still based on the old syntax, and many of them if you dig deep will also give the bad advice! So many people try browser testing and give up because in some ways it's hard to make the testing consistent. Especially if you use onClick events that load another page, it's so easy to forget that you might or might not yet have loaded that page when the next line of code runs! That browser-side JavaScript has made your integration now multi-threaded. The hardest one to figure out is when you grab a reference to something on the current page that looks like the element you want, but it's on the previous page... it's not what you want. Then when you get around to trying to finding nodes within it, or clicking on that node. You've loaded the page you want, but your reference is pointing to a node that is no longer present in the active DOM, because you're on the new page now! If you're composing reusable steps (like in Cucumber) Always find a way to make sure you've already loaded the DOM of the page you think you're on, before you get any references to nodes on the page. Even if you have to put #some-target-page-node on the target page so you can expect(page).to have_selector('#some-target-page-node') That will prove you've landed on the target page, and if you find it sometimes takes a long time, or if it's longer than the timeout, set 'wait' to a longer number of seconds: expect(page).to have_selector('.some-node', wait: 90) If your page has waits that long of course, it's most likely going to start affecting your conversion rates, so do something about that...

yebyen 9y ago
To clarify... set 'wait' to a longer number of seconds: expect(page).to have_selector('.some-node', wait: 90) If your page has waits that long of course Wait is different than sleep. Wait retries until the condition is met. This makes it a little bit confusing for "negative presence" tests where you are testing for the absence of a thing, but it ensures that you are not waiting a full 90 seconds for a thing to happen, when that thing has already happened. This is the greatest, most noticeable effect of the "shift." No more "until". This is something like explicit/implicit waiting described by the OP's article, although Capybara is careful now to avoid "wait until" you can still get this behavior using expect statements as I showed above. It's much cleaner in capybara[1] nowadays, and I'm actually surprised people work with Selenium WebDriver directly in Python by comparison. You can also wait explicitly for the AJAX events to complete, although I never do this[2] because it seems to introduce a new source of error: the AJAX completes before we get around to observing that it had started. This article seems to provide a bullet-proof way to do it though, upon review, because it's simply counting with JS variables in the DOM. You can even start multiple AJAX requests at once and wait for all of them to complete, or just go ahead and move on if you can see they already completed when you first check. No waiting. This is more reliable than adding and removing the ".ajax-processing" class to a hidden element, which you must be careful to observe so you know your AJAX has started, and then observe again as it is removed so you know it has completed. The method in [2] is a bit less "semaphore-ish" and looks more reliable than ways I've used. Another nice trick is to remove another source of threaded racing: your animations. If you have a test that triggers an animation and you want to disable animations, run once you get to the page: page.execute_script("$.fx.off=true") # or jquery.fx.off=true This runs on the DOM though, so it must be run on every page that loads if it is going to be effective. There are also ways[3][4] you can get this script to run on every page rather than running it manually just wherever needed, I haven't tried this, but I'd recommend it if you don't want to spend time by hand tracing down the source of each delay. I feel like there were about 5 things I had to learn to make my tests reliable, and I am still applying them daily. But once I had the experience of "tracking down the Heisenbug" enough times, I can avoid writing 98% of the bad tests, and tracking down the remaining 2% when they inevitably show up in my test suites as "failed once, then failure went away with no code changes" it simply hasn't been hard to track down the source of the failure anymore, as I now know really well what I'm looking for, what causes this type of failure. I personally got a bit out of this article, but I'm not sure anyone is going to read this article and just know better how to do browser testing. I think you have to go through this experience of accidentally writing sometimes-unreliable tests, and then figuring out how to fix them. I did not learn without the help of articles like this, though. The last great advice I found was to learn to mock API responses if your application uses external APIs, but don't over-rely on mocks. We have lots of tests that do hit our APIs and will fail if they are down. But we also have tests that depend on some difficult to replicate condition being met by the API servers. Those mock tests in some cases are more dangerously brittle than the tests that actually hit the API. My favorite was added recently, the test that shows what happens if the API goes down in the middle of your overnight job. Hint: You want the job to stop running at the first sign that the API has gone down, and trigger alert mail or otherwise signal an error that you can find later. With WebMock, this test was easy to write! Just send back a 500 error at a time when you know you weren't expecting it. You can usually use factories to replicate other less unusual conditions, but when your app depends heavily on external APIs, this trick can help greatly to simplify your tests. This particular test would have been impossible to write without mocks. I would not mock more than 50% of API tests though, because it is equally valuable to find out when an unrelated change that is deployed through your CI environment to one of your APIs, will break some part of your application unexpectedly. This is why I do not mock 100% of API-related tests. [1]: https://www.varvet.com/blog/why-wait_until-was-removed-from-capybara/ https://www.varvet.com/blog/why-wait_until-was-removed-from-... [2]: http://cabeca.github.io/blog/2013/06/16/waiting-for-completed-ajax-in-capybara-and-other-tricks/ http://cabeca.github.io/blog/2013/06/16/waiting-for-complete... [3]: https://makandracards.com/coffeeandcode/7503-disable-jquery-animations-during-rails-tests https://makandracards.com/coffeeandcode/7503-disable-jquery-... [4]: https://gist.github.com/keithtom/8763169 https://gist.github.com/keithtom/8763169