7 ms·

Running feature specs with Capybara and Chrome headless

101 points by TimPetricola 9y ago

sparaker 9y ago
"It is also supposed to improve memory usage and stability" Did you do a comparison of memory against phantomJS?
- lloeki 9y ago
  Words from PhantomJS ex-maintainer[0]: > I think people will switch to it, eventually. Chrome is faster and more stable than PhantomJS. And it doesn't eat memory like crazy. [0] https://groups.google.com/forum/#!msg/phantomjs/9aI5d-LDuNE/5Z3SMZrqAQAJ https://groups.google.com/forum/#!msg/phantomjs/9aI5d-LDuNE/...
- TimPetricola 9y ago
  In fact no we didn't do any kind of benchmark. It's an assumption made of feedback from other users. As we knew we wanted to switch to headless Chrome anyway, we didn't investigate more into this part.
- royka118 9y ago
  Just this morning my team migrated from phantomJS to chrome headless for our Karma plus Jasmine tests. Did a bit of researching before the move and we found tests ran faster plus chrome used less memory than phantomJS. ~ 2000 specs compiling html and stuff
  - rgbrgb 9y ago
    Any guess at how much faster the tests are running?
lloeki 9y ago
Wow, I completely missed the announcement that the maintainer and sole developer of PhantomJS is stepping down.
waffle_ss 9y ago
I tried switching from poltergeist to Chrome headless last week using a very similar setup, but it was much, much slower. I assumed it was Selenium's fault, but didn't dig any deeper as it seemed to be expected from the guide I was using[1] ("I can anecdotally report that Capybara-Webkit seems significantly faster"). [1]: https://robots.thoughtbot.com/headless-feature-specs-with-chrome https://robots.thoughtbot.com/headless-feature-specs-with-ch...
thibaut_barrere 9y ago
Earlier this month I wrote an ETL extractor using Capybara & headless browser (to work-around a lack of API - PS: only do that as a last resort!). One thing I learned and wanted to warn about here is that Chrome headless currently doesn't support file downloads [1]. PhantomJS won't work either (unless you use a custom build) [2]. I also tried with capybara-webkit, but no luck either [3]. The only driver which ultimately allowed me to download files at this point was Selenium + Firefox (with some tweaking on profile options browser.download.dir, browser.download.folderList = 2, browser.helperApps.neverAsk.saveToDisk set to the MIME type of files to be downloaded). [1] https://bugs.chromium.org/p/chromium/issues/detail?id=696481 https://bugs.chromium.org/p/chromium/issues/detail?id=696481 [2] https://github.com/ariya/phantomjs/issues/10052 https://github.com/ariya/phantomjs/issues/10052 [3] https://github.com/thoughtbot/capybara-webkit/issues/691 https://github.com/thoughtbot/capybara-webkit/issues/691
- TimPetricola 9y ago
  Unfortunately, headless Chrome is still missing some features and file downloads is one of them. I believe Nightmare [1] (running on Electron) handles files download, it might be worth looking at it? [1] https://github.com/segmentio/nightmare https://github.com/segmentio/nightmare
- vadansky 9y ago
  >One thing I learned and wanted to warn about here is that Chrome headless currently doesn't support file downloads [1]. Not true. At least with C# You can use var client = new WebClient(); client.Headers[HttpRequestHeader.Cookie] = cookieString(driver); client.DownloadFile(reportURL, savePath + fileName + ".xlsx"); Along with: string cookieString(IWebDriver driver) { var cookies = driver.Manage().Cookies.AllCookies; return string.Join("; ", cookies.Select(c => string.Format("{0}={1}", c.Name, c.Value))); } Edit: Just wanted to emphasize that this ensures all the steps you did to login won't be lost since you're using the logged in cookie. The idea is to use Selenium to get the download url and then use a regular download method using the selenium cookie.
  - 9y ago
Kiro 9y ago
> To prevent some issues in PhantomJS when elements would overlap, we had a lot of calls like this: I don't understand. How would you do it when it's not overlapping?
mabbo 9y ago
Chrome headless migration has been my bane for the last two weeks. It's almost ready, but not yet. The short version of what's wrong: chrome headless does not support extensions. The chromedriver used by selenium works via starting chrome with an automation extension that gives it certain needed functionality. So things like setting the window size, taking screenshots, and a few other important features just cause chromedriver to crash or hang. There's bugs filed and some clever googlers are trying to fix it all- but it's not ready for our purposes yet. I don't want to rewrite dozens of tests that work with phantomjs to get around chrome's bugs.
- TimPetricola 9y ago
  Indeed, some features are not supported yet (setting size, alerts, ...). We're lucky not to use too many of them in our actual test suite!
  - MikeKusold 9y ago
    Just implemented headless testing for a rails app today, and discovered that alert support has been worked around in the latest Capybara (2.14.4). https://github.com/teamcapybara/capybara/commit/12c10059709346bfeadb2cea18f510174d842303 https://github.com/teamcapybara/capybara/commit/12c100597093...
- larksimian 9y ago
  I believe chromedriver 2.30(or maybe an earlier version) fixes these issues. Certainly it is possible to pass chromeOptions: { args: %w[headless disable-gpu window-size=1920x1080] } to the capabilities object and that works. Screenshots also work in Rails system tests.
  - drspacemonkey 9y ago
    >Screenshots also work in Rails system tests. Do you have a link or something about how to make that work? About two weeks ago, I was struggling with screenshots and pushed it back to my to-do list.
wingerlang 9y ago
Kind of off topic, but. Is running automation as complicated as this? I recently wanted to log in to a website, click some page and download a .csv file. I saw that chrome can be run headless, nice. So I opened it headless and there's this REPL, nice I can do JS directly into the website. Now how do I automate this. This leads me into all sorts of things that doesn't seem related at all (I is, but still). Selenium stuff, automation setups, drivers, language bindings, chrome-api-stuff and no end in sight. All I want is something like "chrome -headless -js script-flow.file http://URL" http://URL" Am I to overwhelmed or is there no simple way without buttloads of 3rd party tools and setups required?
- diggan 9y ago
  Yeah, it's actually way simpler than that since chrome (and I think Firefox now too) is exposing a API for driving the browser, what you're after is the remote debugging protocol: https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ But, you have bunch of things that abstract that for you, so you don't have to implement things hitting that API manually. Anyways, here is a handy document from Mozilla on the protocol too: http://searchfox.org/mozilla-central/source/devtools/docs/backend/protocol.md http://searchfox.org/mozilla-central/source/devtools/docs/ba...
- mrskitch 9y ago
  I'm working on a high-level API to solve a lot of what you're describing. It's still in its infancy, but soon will be runtime agnostic: https://github.com/joelgriffith/navalia https://github.com/joelgriffith/navalia. File an issue for what's missing!
- robk 9y ago
  This one will be close one day for downloading at least. Ok today for scraping. https://github.com/LucianoGanga/simple-headless-chrome https://github.com/LucianoGanga/simple-headless-chrome
- corford 9y ago
  A nice quick solution for that is SlimerJS with Casperjs.
jayflux 9y ago
Poltergeist had a network_traffic mode which we found quite useful. (Seeing if requests were made etc) Does Chrome headless have an equivalent? It's the only thing preventing us from moving across. And if not where should I raise it?
- mickeyben 9y ago
  Not that I know of, but AFAIK you can use a proxy. Even with Poltergeist I used to debug a slow test suite with it's `--proxy-server` option. You can fire up mitmproxy and see the requests flowing in real time. You can even modify them or replay them.
- rberdeen 9y ago
  Headless Chrome is controlled using the Chrome DevTools protocol, which has quite a few options for tracking and modifying network requests. Everything you get in the Network inspector in Chrome DevTools should be available in Headless Chrome. https://chromedevtools.github.io/devtools-protocol/tot/Network/ https://chromedevtools.github.io/devtools-protocol/tot/Netwo...
mrskitch 9y ago
I hear a lot of folks mentioning the pain working with headless chrome. This was the inspiration for Navalia, which is inspired by the API from nightmare. It also handles the case where you want to run simultaneous scripts by using new tabs (and even multiple chrome instances as well). It's still early in life, and I'm about to do another breaking release, but here is the repo https://github.com/joelgriffith/navalia https://github.com/joelgriffith/navalia
zimbatm 9y ago
When specs are failing you actually want to see what's happening in the browser. Xvfb + normal Chrome + ffmpeg screen capture. This can be run in Docker and thus on Macs as well.

101 points by TimPetricola 9y ago

sparaker 9y ago

"It is also supposed to improve memory usage and stability" Did you do a comparison of memory against phantomJS?

lloeki 9y ago
Words from PhantomJS ex-maintainer[0]: > I think people will switch to it, eventually. Chrome is faster and more stable than PhantomJS. And it doesn't eat memory like crazy. [0] https://groups.google.com/forum/#!msg/phantomjs/9aI5d-LDuNE/5Z3SMZrqAQAJ https://groups.google.com/forum/#!msg/phantomjs/9aI5d-LDuNE/...
TimPetricola 9y ago
In fact no we didn't do any kind of benchmark. It's an assumption made of feedback from other users. As we knew we wanted to switch to headless Chrome anyway, we didn't investigate more into this part.
royka118 9y ago
Just this morning my team migrated from phantomJS to chrome headless for our Karma plus Jasmine tests. Did a bit of researching before the move and we found tests ran faster plus chrome used less memory than phantomJS. ~ 2000 specs compiling html and stuff
- rgbrgb 9y ago
  Any guess at how much faster the tests are running?

lloeki 9y ago

Wow, I completely missed the announcement that the maintainer and sole developer of PhantomJS is stepping down.

waffle_ss 9y ago

I tried switching from poltergeist to Chrome headless last week using a very similar setup, but it was much, much slower. I assumed it was Selenium's fault, but didn't dig any deeper as it seemed to be expected from the guide I was using[1] ("I can anecdotally report that Capybara-Webkit seems significantly faster"). [1]: https://robots.thoughtbot.com/headless-feature-specs-with-chrome https://robots.thoughtbot.com/headless-feature-specs-with-ch...

thibaut_barrere 9y ago

Earlier this month I wrote an ETL extractor using Capybara & headless browser (to work-around a lack of API - PS: only do that as a last resort!). One thing I learned and wanted to warn about here is that Chrome headless currently doesn't support file downloads [1]. PhantomJS won't work either (unless you use a custom build) [2]. I also tried with capybara-webkit, but no luck either [3]. The only driver which ultimately allowed me to download files at this point was Selenium + Firefox (with some tweaking on profile options browser.download.dir, browser.download.folderList = 2, browser.helperApps.neverAsk.saveToDisk set to the MIME type of files to be downloaded). [1] https://bugs.chromium.org/p/chromium/issues/detail?id=696481 https://bugs.chromium.org/p/chromium/issues/detail?id=696481 [2] https://github.com/ariya/phantomjs/issues/10052 https://github.com/ariya/phantomjs/issues/10052 [3] https://github.com/thoughtbot/capybara-webkit/issues/691 https://github.com/thoughtbot/capybara-webkit/issues/691

TimPetricola 9y ago
Unfortunately, headless Chrome is still missing some features and file downloads is one of them. I believe Nightmare [1] (running on Electron) handles files download, it might be worth looking at it? [1] https://github.com/segmentio/nightmare https://github.com/segmentio/nightmare
vadansky 9y ago
>One thing I learned and wanted to warn about here is that Chrome headless currently doesn't support file downloads [1]. Not true. At least with C# You can use var client = new WebClient(); client.Headers[HttpRequestHeader.Cookie] = cookieString(driver); client.DownloadFile(reportURL, savePath + fileName + ".xlsx"); Along with: string cookieString(IWebDriver driver) { var cookies = driver.Manage().Cookies.AllCookies; return string.Join("; ", cookies.Select(c => string.Format("{0}={1}", c.Name, c.Value))); } Edit: Just wanted to emphasize that this ensures all the steps you did to login won't be lost since you're using the logged in cookie. The idea is to use Selenium to get the download url and then use a regular download method using the selenium cookie.
- 9y ago

Kiro 9y ago

> To prevent some issues in PhantomJS when elements would overlap, we had a lot of calls like this: I don't understand. How would you do it when it's not overlapping?

mabbo 9y ago

Chrome headless migration has been my bane for the last two weeks. It's almost ready, but not yet. The short version of what's wrong: chrome headless does not support extensions. The chromedriver used by selenium works via starting chrome with an automation extension that gives it certain needed functionality. So things like setting the window size, taking screenshots, and a few other important features just cause chromedriver to crash or hang. There's bugs filed and some clever googlers are trying to fix it all- but it's not ready for our purposes yet. I don't want to rewrite dozens of tests that work with phantomjs to get around chrome's bugs.

TimPetricola 9y ago
Indeed, some features are not supported yet (setting size, alerts, ...). We're lucky not to use too many of them in our actual test suite!
- MikeKusold 9y ago
  Just implemented headless testing for a rails app today, and discovered that alert support has been worked around in the latest Capybara (2.14.4). https://github.com/teamcapybara/capybara/commit/12c10059709346bfeadb2cea18f510174d842303 https://github.com/teamcapybara/capybara/commit/12c100597093...
larksimian 9y ago
I believe chromedriver 2.30(or maybe an earlier version) fixes these issues. Certainly it is possible to pass chromeOptions: { args: %w[headless disable-gpu window-size=1920x1080] } to the capabilities object and that works. Screenshots also work in Rails system tests.
- drspacemonkey 9y ago
  >Screenshots also work in Rails system tests. Do you have a link or something about how to make that work? About two weeks ago, I was struggling with screenshots and pushed it back to my to-do list.

wingerlang 9y ago

Kind of off topic, but. Is running automation as complicated as this? I recently wanted to log in to a website, click some page and download a .csv file. I saw that chrome can be run headless, nice. So I opened it headless and there's this REPL, nice I can do JS directly into the website. Now how do I automate this. This leads me into all sorts of things that doesn't seem related at all (I is, but still). Selenium stuff, automation setups, drivers, language bindings, chrome-api-stuff and no end in sight. All I want is something like "chrome -headless -js script-flow.file http://URL" http://URL" Am I to overwhelmed or is there no simple way without buttloads of 3rd party tools and setups required?

diggan 9y ago
Yeah, it's actually way simpler than that since chrome (and I think Firefox now too) is exposing a API for driving the browser, what you're after is the remote debugging protocol: https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ But, you have bunch of things that abstract that for you, so you don't have to implement things hitting that API manually. Anyways, here is a handy document from Mozilla on the protocol too: http://searchfox.org/mozilla-central/source/devtools/docs/backend/protocol.md http://searchfox.org/mozilla-central/source/devtools/docs/ba...
mrskitch 9y ago
I'm working on a high-level API to solve a lot of what you're describing. It's still in its infancy, but soon will be runtime agnostic: https://github.com/joelgriffith/navalia https://github.com/joelgriffith/navalia. File an issue for what's missing!
robk 9y ago
This one will be close one day for downloading at least. Ok today for scraping. https://github.com/LucianoGanga/simple-headless-chrome https://github.com/LucianoGanga/simple-headless-chrome
corford 9y ago
A nice quick solution for that is SlimerJS with Casperjs.

jayflux 9y ago

Poltergeist had a network_traffic mode which we found quite useful. (Seeing if requests were made etc) Does Chrome headless have an equivalent? It's the only thing preventing us from moving across. And if not where should I raise it?

mickeyben 9y ago
Not that I know of, but AFAIK you can use a proxy. Even with Poltergeist I used to debug a slow test suite with it's `--proxy-server` option. You can fire up mitmproxy and see the requests flowing in real time. You can even modify them or replay them.
rberdeen 9y ago
Headless Chrome is controlled using the Chrome DevTools protocol, which has quite a few options for tracking and modifying network requests. Everything you get in the Network inspector in Chrome DevTools should be available in Headless Chrome. https://chromedevtools.github.io/devtools-protocol/tot/Network/ https://chromedevtools.github.io/devtools-protocol/tot/Netwo...

mrskitch 9y ago

I hear a lot of folks mentioning the pain working with headless chrome. This was the inspiration for Navalia, which is inspired by the API from nightmare. It also handles the case where you want to run simultaneous scripts by using new tabs (and even multiple chrome instances as well). It's still early in life, and I'm about to do another breaking release, but here is the repo https://github.com/joelgriffith/navalia https://github.com/joelgriffith/navalia

zimbatm 9y ago

When specs are failing you actually want to see what's happening in the browser. Xvfb + normal Chrome + ffmpeg screen capture. This can be run in Docker and thus on Macs as well.