6 ms·
A list of headless web browsers
- duke360 10y agoi've tried some of these for a project ( i had to cough automate download of files from a site made in old asp.net and with a lot of new js on it), only awesomium was able to do the job, but, man, it was overwelming difficult! i had to inject fiddler as proxy to catch exactly what was happening, awesomium by itself just can't do it, we are very far from a real scriptable headless browser.
- DonHopkins 10y agoUnfortunately, Awesomium seems to have been in "lights on, nobody home" mode for over a year now, and uses an outdated version of Chrome. [1] The nail in the coffin is a predictably unfulfilled promise (see Yannou's rule #1 and #2) followed by radio silence: [2] "The product is very well alive. Version 1.7.6 will be released in a few days with many new features and an updated Unity plugin as well. Meanwhile, we are also working on our next major version (v2.0)." Perikles C. Stephanidis, Jun 15 The other big problem is that it's not free. So nobody is going to pick it up and run with it. [1] http://answers.awesomium.com/questions/6880/does-the-project-is-still-supported-developed.html http://answers.awesomium.com/questions/6880/does-the-project... [2] http://answers.awesomium.com/questions/7255/is-awesomium-dead-now.html http://answers.awesomium.com/questions/7255/is-awesomium-dea...
- dom96 10y agoAnybody else seeing something similar to this? https://i.imgur.com/Dz1FLGQ.png https://i.imgur.com/Dz1FLGQ.png. It's as if GitHub was embedded in an iframe on this repo's page.
- piotrkaminski 10y agoYup, I'm seeing the same thing. Weird.
- dom0 10y agoDitto. Probably someone messed up some piece of dynamically loaded server-side rendered stuff in the repository info bar. Since not everyone seems to have it it's probably not on full-roll-out, but just a small part of the user base.
- joefarish 10y agoIt's obviously not a headless browser but xvfb can be very useful depending on what exactly one is trying to test: https://en.wikipedia.org/wiki/Xvfb https://en.wikipedia.org/wiki/Xvfb
- c0nfused 10y agoThis is the way I am going. I spent a long time trying to find an up to date javascript engine in a headless browser a while back before I gave up and just wrote a chrome extension, installed a copy of chromium, and ran the whole thing in xvfb with the extension added by command line. In the end unless you are severely resource constrained,it is a very good option.
- chriswarbo 10y agoI've done a similar thing. There are annoyances with both XVFB and Firefox instances clobbering each other though. For XVFB, each instance needs a globally unique display number. For Firefox, you need the "-new-instance" or "-no-remote" option, along with a unique profile (e.g. in a temp dir).
- jordanpg 10y agoCould you say more about what you mean by "clobbering each other" here? I run firefox/chrome under Xvfb and very frequently get unexplaine `Unable to connect to host 127.0.0.1 on port 7055` from selenium, zombie firefox and chrome processes, and hundreds of TCP ports in status TIME_WAIT. I've always assumed it had something to do with Xvfb/browser interaction.
- j_s 10y agoFirefox shares sessions (cookies, etc.) between all running instances by default.
- chriswarbo 10y agoWhen you run "firefox", it will check if the current user already has an instance running; even if it's on a different display. To prevent that you need to run the firefox command with "-new-instance" or "-no-remote" (which implies "-new-instance"). Firefox also 'locks' profiles which are in use, so you need to use separate profiles for each Firefox instance which is running. You can do that by making a temporary directory "T" and passing it to the firefox command as "-profile T". As an example, the Firefox command I'm using in one of my scripts is currently: timeout 60 firefox -safe-mode \ -profile "$FF_DIR" \ -no-remote \ -new-instance \ "$URL" XFVB is an X server, which programs connect to based on the "DISPLAY" environment variable. You can see what that's set to by running: echo "$DISPLAY" You can set it, e.g. to ":99", by doing: export DISPLAY=:99 Each running X server needs a different display number; this includes regular desktop sessions (usually running on Xorg with display :0), XVFB, Xnest, Xephyr, etc. I use a variant of the script at http://stackoverflow.com/a/30336424/884682 http://stackoverflow.com/a/30336424/884682 to choose an unused display number for each XVFB invocation. NOTE: This is based on simple automation tasks with bash and xdotool. I don't use Selenium, so I don't know if already does any of this. Hope it helps anyway :)
- VeejayRampay 10y agoI really wish we had a good set of scriptable browser engines (i.e. everything but the graphics part) for all browsers. That would make testing so much easier. Even simple tasks like taking a screenshot of a given web page involves PhantomJS, which seems really heavy-handed. Thanks for compiling this list but the issue I'm having with this is the lack of a rating system based on usage, Github stars, downloads, repo activity, project creation, corporate support, etc. to get a measure of how reliable those tools are.
- mping 10y agoYeah, some are quite old and discontinued. I currently use Nightmare which is electron based and has a nice api. Tried a couple others but every time something went wrong.
- icebraining 10y agoThe problem is that to take a screenshot of how the page actually looks, you need almost everything from the browser. Rendering of HTML/CSS/images/videos, Javascript execution, networking, cookies... you end up with 90% of the browser, and then removing the last 10% is not worth it.
- nattaylor 10y agoIf all you want to do is take a screenshot, I've had success with http://wkhtmltopdf.org/ http://wkhtmltopdf.org/ (which also outputs to image formats.) You can't script it though.
- bzbarsky 10y agoThe problem is that the "graphics part" is often the part that knows how to deal with fonts, which you sort of need for layout.
- mbrock 10y agoHere's a thing I made to script WebDriver from the terminal or shell scripts: https://github.com/mbrock/wd https://github.com/mbrock/wd
- riston 10y agoMost of the tools listed use PhantomJS or QT core browser as the base. The problem is that PhantomJS doesn't support many new API-s or the rendering is different. There is also headless chrome browser in development, not sure if there are any prebuilt binaries out there?
- jordybg 10y agoGoing to ask same question, about headless Chrome, any recent information on that, anyone?
- gowan 10y agoit is issue 546953[1] in the chromium issue tracker. [1] https://bugs.chromium.org/p/chromium/issues/detail?id=546953 https://bugs.chromium.org/p/chromium/issues/detail?id=546953
- caleblloyd 10y agoThere is a Google Group for it: https://groups.google.com/a/chromium.org/forum/#!forum/headless-dev https://groups.google.com/a/chromium.org/forum/#!forum/headl... I check every now and again. I'm super interested in headless PDF generation (using Nigtmare now)
- gowan 10y agowhat new apis should phantomjs support? download support[1] is the only new api in development. it's not tagged for the 2.5 release[2]. [1] https://github.com/ariya/phantomjs/pull/14225 https://github.com/ariya/phantomjs/pull/14225 [2] https://github.com/ariya/phantomjs/issues/14458 https://github.com/ariya/phantomjs/issues/14458
- nmk 10y agoFileReader support is not fully implemented (addEventListener is not present). Just ran into this last week. There has been an open issue for this for three years[1]. [1] https://github.com/ariya/phantomjs/issues/11733 https://github.com/ariya/phantomjs/issues/11733
- tzaman 10y agoI know it's somewhat a vanity metric, but this list should have GitHub stars listed for each project. Would save us some work :)
- r3bl 10y agoUnless someone creates a bot that will automate the process of updating the number of stars, it's a cat and mouse game.
- jpalomaki 10y agoMaybe something like this would be an easy start: http://githubbadges.com/ http://githubbadges.com/
- segmondy 10y agonumber of contributors, last commit date too.
- DonHopkins 10y agoShouldn't Chromium Embedded Framework be on this list? [1] That seems to be robust, well maintained and widely used by many projects [3], uses a recent version of Chrome, and it supports off-screen (non-GPU-accelerated) rendering. [3] I haven't used it myself, but I'm considering it, so I'd appreciate hearing other people's thoughts about it. [1] https://bitbucket.org/chromiumembedded/cef https://bitbucket.org/chromiumembedded/cef [2] https://en.wikipedia.org/wiki/Chromium_Embedded_Framework#Applications_using_CEF https://en.wikipedia.org/wiki/Chromium_Embedded_Framework#Ap... [3] https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-header-off-screen-rendering https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage...
- to3m 10y agoIt's a strange omission given that CEF is free and seemingly in rude health while Awesomium, number 1 on the list (if only by dint of its name), looks dead, even though you have to pay for it. I'm using it for off-screen rendering. It's pretty straightforward, once past the inevitable initial hump. The code is sensible (if a little smart pointer-happy), and you're provided with a good range of hooks. Documentation is OK. Nothing you can't figure out if you've got a debugger... (I used Visual Studio with the prebuilt Spotify builds. This addin really helps: https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f-465f-9b6d-d29b7b503d7a https://visualstudiogallery.msdn.microsoft.com/a1141bff-463f...)
- DonHopkins 10y agoEspecially surprising since CEF's Wikipedia page lists 73 projects that use it (many of the well known and widely used, like Spotify, Second Life, Steam Client, Adobe Creative Cloud, Dreamweaver CC, Edge Animate and Edge Reflow). I first looked at Awesomium because it had a Unity3D interface [1], but that's gone stale (Awesomium doesn't support 64 bits, much has changed in Unity since then, not to mention Chrome itself), and some developers on the Unity discussion groups were suggesting CEF as an alternative. [2] [3] I'm interested in integrating the latest CEF with the latest Unity3D on Mac and Windows desktop, as well as for consoles. There are (non-Unity-specific, i.e. maybe too complex and modern) C# wrappers for CEF [4], which looks pretty elaborate, but I think it would be tighter to integrate the CEF C++ code directly as a native Unity extension, without thunking through too much Windows-oriented C# glue that might not be compatible with Unity's older version of C#. On iOS I'm simply using WkWebKit, and WebKit on Chrome, which both support off-screen rendering. Here's some interesting stuff comparing Awesomium and CEF and other stuff from Stratoscope. [5] It's from a 446 day old thread about CefSharp [6] so it might predate the Awesomiumocalypse. The thread is interesting reading! Really makes me wonder why Awesomium is on the list at all. It looks like Coherent UI is a commercially supported alternative to Awesomium, but I haven't looked into it yet. Their employee said good things about CEF and offered some fair criticism: "Hi, Stoyan from Coherent Labs here. My impressions of CEF are very good. They manage to keep it very close to the chromium HEAD, which is great. The only technical downside of CEF is the imperfect GPU acceleration. In many cases however the architecture of chromium itself is a hindrance. Their processes hog memory like crazy and introduce perceivable lag. We had to write a complete new product to solve the issue for games." [7] [1] http://labs.awesomium.com/unity3d-integration-tutorial-part-1/ http://labs.awesomium.com/unity3d-integration-tutorial-part-... [2] http://answers.unity3d.com/questions/945207/unity3d-v5-awesomium-currently-compatible.html http://answers.unity3d.com/questions/945207/unity3d-v5-aweso... [3] http://gamedev.stackexchange.com/questions/111846/chromium-embedded-framework-and-unity3d http://gamedev.stackexchange.com/questions/111846/chromium-e... [4] https://github.com/cefsharp/CefSharp https://github.com/cefsharp/CefSharp [5] https://news.ycombinator.com/item?id=10089645 https://news.ycombinator.com/item?id=10089645 [6] https://news.ycombinator.com/item?id=10087013 https://news.ycombinator.com/item?id=10087013 [7] https://news.ycombinator.com/item?id=10096616 https://news.ycombinator.com/item?id=10096616
- matthewhall 10y agoWhat about Electron? Multiple people, including myself, have used it to make a web browser
- m0th87 10y agofwiw, nightmare is just a thin layer above electron's browser.
- OtterCoder 10y agoI'm currently using Nightmare for a little automation. It's fantastic. A few small gotchas, but so easy to set up.
- k__ 10y agoThe problem I found with this was mostly that there are bugs that only show in one browser (only FF, only Chrome, only Edge, etc) so these tools get their own share of bugs. At the moment I use Ghostinspector (a SaaS) which works okay. Their UI is a bit clunky, but recording stuff via Chrome extension and modifying it later works okay. Also they let me run the tests with Firefox later. So I develop with Chrome and run tests with Firefox.
- natch 10y agoMissing: lynx, links (both have headless mode), curl, wget, nscurl.
- skinnymuch 10y agoPretty sure being able to render JavaScript is needed to be on the list. So not most of what you listed.
- nathancahill 10y agoIt's missing the browser engine and version that each of them use. Even latest PhantomJS only uses Chromium 40. Most important metric IMO.
- recrof 10y agoisn't PJS based on webkit?
- nathancahill 10y agoSame same
- equasar 10y agoWould be nice to include if is it maintained or their last update. There are a lot of them that aren't updated in years.
- konradzikusek 10y agoChrome will officially support headless mode: http://crbug.com/546953 http://crbug.com/546953