4 ms·

That would be so awesome. The current alternatives: 1. Use PhantomJS: It easy to script, but it lags over real browser by few years and development is stagnant

That would be so awesome. The current alternatives: 1. Use PhantomJS: It easy to script, but it lags over real browser by few years and development is stagnant. Many real website doesn't work in PhantomJS. 2. Use Chrome/Firefox xvfb. You have real browser, but scripting is hard. E.g. even hello world examples like make a thumbnail of a website takes a lot of time to get right. According to some rumors, headless Chrome existed before even Chrome was released to the public. Google use it to do web scraping. However, though headless browsers are great for developers, they are also great for spammers and ad fraud. So the main developer likely has conflict of interests whether to invest resources into making headless Chrome public.

andrea_s 10y ago
What about Selenium? Is it already covered in point 2?
- jakozaur 10y ago
  Yes you can do it with Selenium (I mean that in point 2), but IMO it is hard to setup and resource consuming. E.g. You need separate VM or Docket to have browser with virtual frame buffer. Using Web Driver also got a steep learning curve. Well, some people also ask why world ever need Ruby with Rails or Python with Django? You can write any website in Enterprise Java since late 1990s. Same Docker vs. Virtual Machines or Atom vs. Vim/Emacs. Even if capability is there, there is room for improvement. Especially if you can make it easier to use and make developers more productive by an order of magnitude.
  - curiousgal 10y ago
    As someone with a horrible internet connection, I've often resorted to using a VPS, Selenium + Firefox+ xvfb is definitely resourcee-intensive and a pain.
  - zurn 10y ago
    No functional need for a separate VM: xvfb-run -a will automatically pick the next free display number and set up the environment pointing to xvfb's $DISPLAY. Of course container/vm isolation is still a good idea for security and repeatability reasons.
  - mbell 10y ago
    > Using Web Driver also got a steep learning curve. I would recommend not working with WebDriver directly. There are many projects that wrap it in a sane API and make it very easy to use. For example if you're using Javascript then use http://webdriver.io http://webdriver.io instead of directly using the Selenium WebDriver API. > IMO it is hard to setup and resource consuming What issues have you had? It's always been very easy to setup in my experience. The only two 'issues' I've had are correct management of the Selenium server within a task running context and odd interactions with some browsers when they are driven by selenium (e.g. safari considering `window.open()` to always be a popup from selenium triggered actions).
    - jakozaur 10y ago
      I used http://nightwatchjs.org/ http://nightwatchjs.org/ as a library talk to selenium, but plethora of WebDriver wrapper rather confused me, than added value. Btw, can you point me to some tutorial that teach you in 15 minutes from zero how to setup something that can change any url into a thumbnail screen shoot. You can do that with phantomjs: brew install phantomjs http://phantomjs.org/screen-capture.html http://phantomjs.org/screen-capture.html Maybe a selenium pro can do it quickly, but for me: 1. How to install selenium? It took me few attempts until I realized that Docker containers are the best way. 2. Install Docker on MacOS and start container. 3. Find some toolkit that talks to selenium. There are tons of libraries, but just some have reasonable docs and are well maintained. 4. Configure this toolkit so it can talk to Selenium. 5. Beware if your container die. Also figuring out that website is loaded is also non-trivial in Selenium and default are worse than in PhantomJS. 6. Etc.
      - mbell 10y ago
        I'm kind of curious how you came to Docker and PhantomJS being the easiest way, that sounds like the worst possible way to me. To get things started simply with JS just follow instructions here: http://webdriver.io/guide.html http://webdriver.io/guide.html The only major diversions from that are to use Mocha or Jasmine for your test framework (instructions on the same site) and to automate starting / stopping the selenium server for which there are grunt/gulp plugins.
ddebernardy 10y ago
Another good option is Scrapy combined with Splash.
fauigerzigerk 10y ago
Another downside to the selenium API is that there is no way to get any upstream HTTP status or network error. That includes redirects unfortunately. I'm using a Chrome plugin for that purpose, which complicates the whole setup even further.
- benjiweber 10y ago
  You could use a proxy like browsermob, which you can connect via the selenium API.
  - fauigerzigerk 10y ago
    That's yet another moving part with its own error modes. And I'm not quite sure (without having it thought through) how I would do something like get the URL of the last permanent redirect and correlate it with the original request. I'm sure it's possible somehow, but it doesn't seem exactly straightforward.
m0th87 10y ago
There's also Nightmare, which is built off Atom. We ported all of our PhantomJS-backed headless tests to Nightmare recently, and it's paid in dividends: http://www.nightmarejs.org http://www.nightmarejs.org
est 10y ago
Mozilla Servo has a headless mode, but PITA to get it working.
robin_reala 10y ago
Option 3: SlimerJS[1]. We evaluated it for testing CSP2 functionality which WebKit doesn’t yet support. [1] http://slimerjs.org/ http://slimerjs.org/

andrea_s 10y ago

What about Selenium? Is it already covered in point 2?

jakozaur 10y ago
Yes you can do it with Selenium (I mean that in point 2), but IMO it is hard to setup and resource consuming. E.g. You need separate VM or Docket to have browser with virtual frame buffer. Using Web Driver also got a steep learning curve. Well, some people also ask why world ever need Ruby with Rails or Python with Django? You can write any website in Enterprise Java since late 1990s. Same Docker vs. Virtual Machines or Atom vs. Vim/Emacs. Even if capability is there, there is room for improvement. Especially if you can make it easier to use and make developers more productive by an order of magnitude.
- curiousgal 10y ago
  As someone with a horrible internet connection, I've often resorted to using a VPS, Selenium + Firefox+ xvfb is definitely resourcee-intensive and a pain.
- zurn 10y ago
  No functional need for a separate VM: xvfb-run -a will automatically pick the next free display number and set up the environment pointing to xvfb's $DISPLAY. Of course container/vm isolation is still a good idea for security and repeatability reasons.
- mbell 10y ago
  > Using Web Driver also got a steep learning curve. I would recommend not working with WebDriver directly. There are many projects that wrap it in a sane API and make it very easy to use. For example if you're using Javascript then use http://webdriver.io http://webdriver.io instead of directly using the Selenium WebDriver API. > IMO it is hard to setup and resource consuming What issues have you had? It's always been very easy to setup in my experience. The only two 'issues' I've had are correct management of the Selenium server within a task running context and odd interactions with some browsers when they are driven by selenium (e.g. safari considering `window.open()` to always be a popup from selenium triggered actions).
  - jakozaur 10y ago
    I used http://nightwatchjs.org/ http://nightwatchjs.org/ as a library talk to selenium, but plethora of WebDriver wrapper rather confused me, than added value. Btw, can you point me to some tutorial that teach you in 15 minutes from zero how to setup something that can change any url into a thumbnail screen shoot. You can do that with phantomjs: brew install phantomjs http://phantomjs.org/screen-capture.html http://phantomjs.org/screen-capture.html Maybe a selenium pro can do it quickly, but for me: 1. How to install selenium? It took me few attempts until I realized that Docker containers are the best way. 2. Install Docker on MacOS and start container. 3. Find some toolkit that talks to selenium. There are tons of libraries, but just some have reasonable docs and are well maintained. 4. Configure this toolkit so it can talk to Selenium. 5. Beware if your container die. Also figuring out that website is loaded is also non-trivial in Selenium and default are worse than in PhantomJS. 6. Etc.
    - mbell 10y ago
      I'm kind of curious how you came to Docker and PhantomJS being the easiest way, that sounds like the worst possible way to me. To get things started simply with JS just follow instructions here: http://webdriver.io/guide.html http://webdriver.io/guide.html The only major diversions from that are to use Mocha or Jasmine for your test framework (instructions on the same site) and to automate starting / stopping the selenium server for which there are grunt/gulp plugins.

ddebernardy 10y ago

Another good option is Scrapy combined with Splash.

fauigerzigerk 10y ago

Another downside to the selenium API is that there is no way to get any upstream HTTP status or network error. That includes redirects unfortunately. I'm using a Chrome plugin for that purpose, which complicates the whole setup even further.

benjiweber 10y ago
You could use a proxy like browsermob, which you can connect via the selenium API.
- fauigerzigerk 10y ago
  That's yet another moving part with its own error modes. And I'm not quite sure (without having it thought through) how I would do something like get the URL of the last permanent redirect and correlate it with the original request. I'm sure it's possible somehow, but it doesn't seem exactly straightforward.

m0th87 10y ago

There's also Nightmare, which is built off Atom. We ported all of our PhantomJS-backed headless tests to Nightmare recently, and it's paid in dividends: http://www.nightmarejs.org http://www.nightmarejs.org

est 10y ago

Mozilla Servo has a headless mode, but PITA to get it working.

robin_reala 10y ago

Option 3: SlimerJS[1]. We evaluated it for testing CSP2 functionality which WebKit doesn’t yet support. [1] http://slimerjs.org/ http://slimerjs.org/