4 ms·

It's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb https://en

It's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb https://en.wikipedia.org/wiki/Xvfb

c0nfused 10y ago
This is the way I am going. I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line. In the end unless you are severely resource constrained,it is a very good option.
- chriswarbo 10y ago
  I've done a similar thing. There are annoyances with both XVFB and Firefox instances clobbering each other though. For XVFB, each instance needs a globally unique display number. For Firefox, you need the "-new-instance" or "-no-remote" option, along with a unique profile (e.g. in a temp dir).
  - jordanpg 10y ago
    Could you say more about what you mean by "clobbering each other" here? I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT. I've always assumed it had something to do with Xvfb/browser interaction.
    - j_s 10y ago
      Firefox shares sessions (cookies, etc.) between all running instances by default.
    - chriswarbo 10y ago
      When you run "firefox", it will check if the current user already has an instance running; even if it's on a different display. To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance"). Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T". As an example, the Firefox command I'm using in one of my scripts is currently: timeout 60 firefox -safe-mode \ -profile "$FF_DIR" \ -no-remote \ -new-instance \ "$URL" XFVB is an X server, which programs connect to based on the "DISPLAY" environment variable. You can see what that's set to by running: echo "$DISPLAY" You can set it, e.g. to ":99", by doing: export DISPLAY=:99 Each running X server needs a different display number; this includes regular desktop sessions (usually running on Xorg with display :0), XVFB, Xnest, Xephyr, etc. I use a variant of the script at http://stackoverflow.com/a/30336424/884682 http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation. NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)
ggurgone 10y ago
+1 you can run things in real browsers. fwiw Travis CI supports it.
FuNe 10y ago
You can also go with a regular browser tied on a vncserver display (pros: you have real captures/video of the tests, cons: the solution gets a bit heavier than when using xvfb)
- bryanlarsen 10y ago
  It's also fairly easy to take a capture of an xvfb or an xdummy display. Dunno about video though.
bryanlarsen 10y ago
Xdummy is another headless X display that has a more modern, more complete stack. https://xpra.org/trac/wiki/Xdummy https://xpra.org/trac/wiki/Xdummy
- reacweb 10y ago
  TIL. I did not know Xdummy. I have just followed your link. It is trivial to launch. It avoids installation of vnc, has a better compatibility and is already present everywhere.
spangry 10y ago
Just to throw in another option, there's selenium grid docker containers: https://github.com/SeleniumHQ/docker-selenium https://github.com/SeleniumHQ/docker-selenium. The containers are a bit heavy IMHO (I suppose because they're aimed more at 'testing' instead of scraping), but I think there are a few Alpine Linux versions floating around (and easy enough to set up yourself). Handy for more tricky sorts of web scraping, or if you want to use chromium in mobile emulation mode (causing websites with an 'app version' to switch over to a HTTP-JSON SPA in many cases). I was particularly excited to discover you can set the 'http_proxy' env variable on a "per-browser node" basis (and do fun stuff like installing DNS caching daemons, round-robinning through OpenVPN/squid containers etc.).

It's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb https://en.wikipedia.org/wiki/Xvfb

c0nfused 10y ago

This is the way I am going. I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line. In the end unless you are severely resource constrained,it is a very good option.

chriswarbo 10y ago
I've done a similar thing. There are annoyances with both XVFB and Firefox instances clobbering each other though. For XVFB, each instance needs a globally unique display number. For Firefox, you need the "-new-instance" or "-no-remote" option, along with a unique profile (e.g. in a temp dir).
- jordanpg 10y ago
  Could you say more about what you mean by "clobbering each other" here? I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT. I've always assumed it had something to do with Xvfb/browser interaction.
  - j_s 10y ago
    Firefox shares sessions (cookies, etc.) between all running instances by default.
  - chriswarbo 10y ago
    When you run "firefox", it will check if the current user already has an instance running; even if it's on a different display. To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance"). Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T". As an example, the Firefox command I'm using in one of my scripts is currently: timeout 60 firefox -safe-mode \ -profile "$FF_DIR" \ -no-remote \ -new-instance \ "$URL" XFVB is an X server, which programs connect to based on the "DISPLAY" environment variable. You can see what that's set to by running: echo "$DISPLAY" You can set it, e.g. to ":99", by doing: export DISPLAY=:99 Each running X server needs a different display number; this includes regular desktop sessions (usually running on Xorg with display :0), XVFB, Xnest, Xephyr, etc. I use a variant of the script at http://stackoverflow.com/a/30336424/884682 http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation. NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)

ggurgone 10y ago

+1 you can run things in real browsers. fwiw Travis CI supports it.

FuNe 10y ago

You can also go with a regular browser tied on a vncserver display (pros: you have real captures/video of the tests, cons: the solution gets a bit heavier than when using xvfb)

bryanlarsen 10y ago
It's also fairly easy to take a capture of an xvfb or an xdummy display. Dunno about video though.

bryanlarsen 10y ago

Xdummy is another headless X display that has a more modern, more complete stack. https://xpra.org/trac/wiki/Xdummy https://xpra.org/trac/wiki/Xdummy

reacweb 10y ago
TIL. I did not know Xdummy. I have just followed your link. It is trivial to launch. It avoids installation of vnc, has a better compatibility and is already present everywhere.

spangry 10y ago

Just to throw in another option, there's selenium grid docker containers: https://github.com/SeleniumHQ/docker-selenium https://github.com/SeleniumHQ/docker-selenium. The containers are a bit heavy IMHO (I suppose because they're aimed more at 'testing' instead of scraping), but I think there are a few Alpine Linux versions floating around (and easy enough to set up yourself). Handy for more tricky sorts of web scraping, or if you want to use chromium in mobile emulation mode (causing websites with an 'app version' to switch over to a HTTP-JSON SPA in many cases). I was particularly excited to discover you can set the 'http_proxy' env variable on a "per-browser node" basis (and do fun stuff like installing DNS caching daemons, round-robinning through OpenVPN/squid containers etc.).