13 ms·

Phantompy - Headless WebKit engine for Python, like phantomjs

171 points by juanriaza 13y ago

hackerboos 13y ago
Cool you can take screenshots with it, see the test_capture_page example.
shadowmint 13y ago
OT I know, I cant help but feel a little smug when I see cool projects like this and they use cmake, and not autotools. Thank you cmake, you've made working in the C ecosystem fun again.
- greggman 13y ago
  Check out gyp and ninja https://code.google.com/p/gyp/wiki/GypVsCMake https://code.google.com/p/gyp/wiki/GypVsCMake https://github.com/martine/ninja https://github.com/martine/ninja
- deleted 13y ago
  [deleted]
- deleted 13y ago
  [deleted]
- randallu 13y ago
  CMake is pretty horrifically inefficient at compiling WebKit. The makefiles it generates fork to call CMake (to do things like printf in color) far more than they fork to call the compiler. The ninja generator is better, but it has various issues with long command lines on WebKit currently (trunk CMake fixes some of these, but then it doesn't make some directories in the output directory). Also, the CMakeLists syntax is pretty nasty IMO, but it feels like gyp, cmake and autotools are in some kind of syntax ugliness competition... (Maybe because nobody ever wants to work on a build system, so they'll just do this one little hack which inevitably grows tentacles).
_seininn 13y ago
I've been working on a project like this that's nearing completion (it works, and the api is mostly complete). It differs from this in that it uses any normal browser to fulfill requests (via extensions). The idea is to have a pythonic API to any browser that has reasonable support for extensions; i.e. a cross-browser scripting.
- laurencerowe 13y ago
  What does it offer over Selenium Webdriver? Yeah, the API is not terribly pythonic, but it's battle tested and stable, works across almost all browsers and there's always wrappers like Splinter (http://splinter.cobrateam.info/ http://splinter.cobrateam.info/) if you prefer a more pythonic interface.
fauigerzigerk 13y ago
I wonder if Phantompy is truly headless so it works on servers without GUI libraries and without xvfb. QtWebKit needs access to some GUI features out of the box, but phantomjs includes platform abstraction code that avoids the dependency on an X server. That's the feature that made me migrate my own C++ crawler code to phantomjs.
- kanzure 13y ago
  Actually, you don't even need xvfb these days (and I have no idea why anyone is telling people that they do). You could use something like xserver-xorg-video-dummy. Of course, as you have pointed out, PhantomJS doesn't require an X server (as opposed to these other projects, wtf) so that it is "really really truly headless".
druska 13y ago
Will this render Flash content?
salimmadjd 13y ago
OT - I've been looking for a headless WebKit for Go (Golang) I've searched all over with no luck. If anyone wants to become famous, please make one :)
rkrzr 13y ago
Does this actually offer full DOM access? I see a 'cssselect' in tests.py, but I wonder how much of the DOM Api is actually supported?
- jessedhillon 13y ago
  Take a look at lib/webelement.hpp -- there's not too much yet, notably absent is a way to iterate children. But you can evaluate Javascript.
encoderer 13y ago
There are so many half-baked headless browser projects, many building in some way on PyQT. I've made heavy use of Spynner and Ghost. Spynner especially. It always seemed to offer exactly what I needed but the inconsistent docs and endlessly-deep stack always seemed to leave successful implementation just beyond my grasp. Now, as an OSS maintainer myself, I sympathize. But I'd love it if this or another one of these projects started breaking away from the pack and could build some critical mass. I'm all for democratization but in this case I'd gladly trade 10 mostly working projects for a single solution ran by an iron-fisted BDFL if it meant decent docs and a mostly stable API.
neo2001 13y ago
Looks phantastic
boothead 13y ago
Is there anything wrong with the python bindings to phantom? I used phantomjs (it's great)a few years ago and I thought there were some python bindings to the original phantom C++ library - could be wrong though.
- waitwhatwhoa 13y ago
  Ariya Hidayat & co have put a monumental amount of effort into exposing the inner workings of webkit in phantomjs - I'm not sure I see the necessity for a python rewrite. Taking a quick look at the code, most everything I use phantomjs for outside of screenshotting is not available. A great start for sure, but unless there are some fundamental shortcomings with phantomjs that I'm unaware of, I agree with boothead that python bindings are probably the way to go.
  - heydenberk 13y ago
    One thing that jumps out at me is that phantompy uses Qt5 internally, while phantomjs uses Qt4. There are lots of new WebKit features and optimizations in Qt5.
    - rjh29 13y ago
      phantomjs will target Qt5 in the next release, 2.0.
- Herald_MJ 13y ago
  There were a set of Python bindings for phantom.js named PyPhantomjs, and it was included in actual phantom releases, but the maintainer moved away from the project, no-one stepped forward and it was ultimately removed from the project (https://github.com/ariya/phantomjs/issues/10344 https://github.com/ariya/phantomjs/issues/10344). So this project is really useful.
  - kanzure 13y ago
    > no-one stepped forward and it was ultimately removed from the project Here's the source code, if you are interested: https://github.com/kanzure/pyphantomjs https://github.com/kanzure/pyphantomjs
woadwarrior01 13y ago
Nice. There's also https://github.com/jeanphix/Ghost.py https://github.com/jeanphix/Ghost.py which I remember using a couple of months ago.
- killahpriest 13y ago
  Ghost.py seems promising but is painful to work with. Installing the dependencies took about an hour (compiling PyQt itself took about 25 minutes). When we did finally get it working, PyQt would crash after the third or fourth scrape and would take between 15 to 30 seconds per scrape. Where we used it: https://github.com/createch/PriceChecker.py/blob/master/pychecker/scraper.py https://github.com/createch/PriceChecker.py/blob/master/pych.... My advice to anybody looking to do headless webkit in Python: Don't use Ghost. Try out PhantomPy and try using the Selenium webdriver for Phantom. We switched over the Selenium webdriver for PhantomJS and found it to be much more stable and fast. See this SO answer by another Ghost.py user who gave up: http://stackoverflow.com/a/15699761/854025 http://stackoverflow.com/a/15699761/854025. Where we used the Selenium driver: https://github.com/createch/PriceChecker.py/blob/phantom/pychecker/scraper2.py https://github.com/createch/PriceChecker.py/blob/phantom/pyc....
  - arikfr 13y ago
    Just to give another perspective: I'm not sure on which OS you had those issues, but I'm successfully using Ghost.py on Ubuntu and OS/X. On OS/X it was harder to install PyQT and sometimes there were weird issues, but on Ubuntu it all went smooth. By now I've ran it on thousands of different websites with little issues. It's true that I'm "recycling" the processes after ~10 sites, but that's mainly because of memory leaks. EDIT: and to clarify on Ubuntu installing either PyQT or PySide (Ghost.py now supports both) was as easy as apt-get install...
    - killahpriest 13y ago
      I tried (but failed to) get it to run at all on OS X. We were able to get it to run on Ubuntu, but thats where it crashed after every three to four scrapes.
acron0 13y ago
For those of us on the outside, what's the typical use cases of a headless browser engine? Testing?
- deleted 13y ago
  [deleted]
- tlarkworthy 13y ago
  angularjs and other JS rendering techniques make the plain HTML look a mess. Google will not be able to present the page or get its content properly, so you use something like this to take snapshots of your website post rendering and expose it for indexing & website previews. (https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot https://developers.google.com/webmasters/ajax-crawling/docs/...)
  - rjh29 13y ago
    I would be surprised if Google was not doing this themselves to some extent.
    - akmiller 13y ago
      They are not (although they definitely should be). Google will read and index javascript, they do not execute it.
    - yahelc 13y ago
      There's some evidence that they are. [1] http://searchengineland.com/google-can-now-execute-ajax-javascript-for-indexing-99518 http://searchengineland.com/google-can-now-execute-ajax-java... [2] http://www.seomoz.org/ugc/can-google-really-access-content-in-javascript-really http://www.seomoz.org/ugc/can-google-really-access-content-i...
- saryant 13y ago
  I recently built a service using PhantomJS for a client. They have graphs and charts generated with d3.js that they wanted to add to their user-specific update emails. I built a backend service using Akka[1] and PhantomJS which would grab the appropriate user's page, render it to HTML, splice up the images and save everything to temp files. Then I had a Java process that read those back in then format and sent an email to the user. [1]: http://akka.io http://akka.io

171 points by juanriaza 13y ago

Cool you can take screenshots with it, see the test_capture_page example.

OT I know, I cant help but feel a little smug when I see cool projects like this and they use cmake, and not autotools. Thank you cmake, you've made working in the C ecosystem fun again.

greggman 13y ago
Check out gyp and ninja https://code.google.com/p/gyp/wiki/GypVsCMake https://code.google.com/p/gyp/wiki/GypVsCMake https://github.com/martine/ninja https://github.com/martine/ninja
deleted 13y ago
[deleted]
deleted 13y ago
[deleted]
randallu 13y ago
CMake is pretty horrifically inefficient at compiling WebKit. The makefiles it generates fork to call CMake (to do things like printf in color) far more than they fork to call the compiler. The ninja generator is better, but it has various issues with long command lines on WebKit currently (trunk CMake fixes some of these, but then it doesn't make some directories in the output directory). Also, the CMakeLists syntax is pretty nasty IMO, but it feels like gyp, cmake and autotools are in some kind of syntax ugliness competition... (Maybe because nobody ever wants to work on a build system, so they'll just do this one little hack which inevitably grows tentacles).

_seininn 13y ago

I've been working on a project like this that's nearing completion (it works, and the api is mostly complete). It differs from this in that it uses any normal browser to fulfill requests (via extensions). The idea is to have a pythonic API to any browser that has reasonable support for extensions; i.e. a cross-browser scripting.

laurencerowe 13y ago
What does it offer over Selenium Webdriver? Yeah, the API is not terribly pythonic, but it's battle tested and stable, works across almost all browsers and there's always wrappers like Splinter (http://splinter.cobrateam.info/ http://splinter.cobrateam.info/) if you prefer a more pythonic interface.

fauigerzigerk 13y ago

I wonder if Phantompy is truly headless so it works on servers without GUI libraries and without xvfb. QtWebKit needs access to some GUI features out of the box, but phantomjs includes platform abstraction code that avoids the dependency on an X server. That's the feature that made me migrate my own C++ crawler code to phantomjs.

kanzure 13y ago
Actually, you don't even need xvfb these days (and I have no idea why anyone is telling people that they do). You could use something like xserver-xorg-video-dummy. Of course, as you have pointed out, PhantomJS doesn't require an X server (as opposed to these other projects, wtf) so that it is "really really truly headless".

druska 13y ago

Will this render Flash content?

salimmadjd 13y ago

OT - I've been looking for a headless WebKit for Go (Golang) I've searched all over with no luck. If anyone wants to become famous, please make one :)

rkrzr 13y ago

Does this actually offer full DOM access? I see a 'cssselect' in tests.py, but I wonder how much of the DOM Api is actually supported?

jessedhillon 13y ago
Take a look at lib/webelement.hpp -- there's not too much yet, notably absent is a way to iterate children. But you can evaluate Javascript.

encoderer 13y ago

There are so many half-baked headless browser projects, many building in some way on PyQT. I've made heavy use of Spynner and Ghost. Spynner especially. It always seemed to offer exactly what I needed but the inconsistent docs and endlessly-deep stack always seemed to leave successful implementation just beyond my grasp. Now, as an OSS maintainer myself, I sympathize. But I'd love it if this or another one of these projects started breaking away from the pack and could build some critical mass. I'm all for democratization but in this case I'd gladly trade 10 mostly working projects for a single solution ran by an iron-fisted BDFL if it meant decent docs and a mostly stable API.

neo2001 13y ago

Looks phantastic

boothead 13y ago

Is there anything wrong with the python bindings to phantom? I used phantomjs (it's great)a few years ago and I thought there were some python bindings to the original phantom C++ library - could be wrong though.

waitwhatwhoa 13y ago
Ariya Hidayat & co have put a monumental amount of effort into exposing the inner workings of webkit in phantomjs - I'm not sure I see the necessity for a python rewrite. Taking a quick look at the code, most everything I use phantomjs for outside of screenshotting is not available. A great start for sure, but unless there are some fundamental shortcomings with phantomjs that I'm unaware of, I agree with boothead that python bindings are probably the way to go.
- heydenberk 13y ago
  One thing that jumps out at me is that phantompy uses Qt5 internally, while phantomjs uses Qt4. There are lots of new WebKit features and optimizations in Qt5.
  - rjh29 13y ago
    phantomjs will target Qt5 in the next release, 2.0.
Herald_MJ 13y ago
There were a set of Python bindings for phantom.js named PyPhantomjs, and it was included in actual phantom releases, but the maintainer moved away from the project, no-one stepped forward and it was ultimately removed from the project (https://github.com/ariya/phantomjs/issues/10344 https://github.com/ariya/phantomjs/issues/10344). So this project is really useful.
- kanzure 13y ago
  > no-one stepped forward and it was ultimately removed from the project Here's the source code, if you are interested: https://github.com/kanzure/pyphantomjs https://github.com/kanzure/pyphantomjs

woadwarrior01 13y ago

Nice. There's also https://github.com/jeanphix/Ghost.py https://github.com/jeanphix/Ghost.py which I remember using a couple of months ago.

killahpriest 13y ago
Ghost.py seems promising but is painful to work with. Installing the dependencies took about an hour (compiling PyQt itself took about 25 minutes). When we did finally get it working, PyQt would crash after the third or fourth scrape and would take between 15 to 30 seconds per scrape. Where we used it: https://github.com/createch/PriceChecker.py/blob/master/pychecker/scraper.py https://github.com/createch/PriceChecker.py/blob/master/pych.... My advice to anybody looking to do headless webkit in Python: Don't use Ghost. Try out PhantomPy and try using the Selenium webdriver for Phantom. We switched over the Selenium webdriver for PhantomJS and found it to be much more stable and fast. See this SO answer by another Ghost.py user who gave up: http://stackoverflow.com/a/15699761/854025 http://stackoverflow.com/a/15699761/854025. Where we used the Selenium driver: https://github.com/createch/PriceChecker.py/blob/phantom/pychecker/scraper2.py https://github.com/createch/PriceChecker.py/blob/phantom/pyc....
- arikfr 13y ago
  Just to give another perspective: I'm not sure on which OS you had those issues, but I'm successfully using Ghost.py on Ubuntu and OS/X. On OS/X it was harder to install PyQT and sometimes there were weird issues, but on Ubuntu it all went smooth. By now I've ran it on thousands of different websites with little issues. It's true that I'm "recycling" the processes after ~10 sites, but that's mainly because of memory leaks. EDIT: and to clarify on Ubuntu installing either PyQT or PySide (Ghost.py now supports both) was as easy as apt-get install...
  - killahpriest 13y ago
    I tried (but failed to) get it to run at all on OS X. We were able to get it to run on Ubuntu, but thats where it crashed after every three to four scrapes.

acron0 13y ago

For those of us on the outside, what's the typical use cases of a headless browser engine? Testing?

deleted 13y ago
[deleted]
tlarkworthy 13y ago
angularjs and other JS rendering techniques make the plain HTML look a mess. Google will not be able to present the page or get its content properly, so you use something like this to take snapshots of your website post rendering and expose it for indexing & website previews. (https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot https://developers.google.com/webmasters/ajax-crawling/docs/...)
- rjh29 13y ago
  I would be surprised if Google was not doing this themselves to some extent.
  - akmiller 13y ago
    They are not (although they definitely should be). Google will read and index javascript, they do not execute it.
  - yahelc 13y ago
    There's some evidence that they are. [1] http://searchengineland.com/google-can-now-execute-ajax-javascript-for-indexing-99518 http://searchengineland.com/google-can-now-execute-ajax-java... [2] http://www.seomoz.org/ugc/can-google-really-access-content-in-javascript-really http://www.seomoz.org/ugc/can-google-really-access-content-i...
saryant 13y ago
I recently built a service using PhantomJS for a client. They have graphs and charts generated with d3.js that they wanted to add to their user-specific update emails. I built a backend service using Akka[1] and PhantomJS which would grab the appropriate user's page, render it to HTML, splice up the images and save everything to temp files. Then I had a Java process that read those back in then format and sent an email to the user. [1]: http://akka.io http://akka.io