6 ms·

Ghost.py - a webkit web client written in python.

173 points by aeurielesn 14y ago

RBerenguel 14y ago
Good! A few days ago I was playing with mechanize to automate some form filling in wordpress posts (iTunes app details, automatically downloaded and then batch-add as post drafts). Gave up by the lack of AJAX-Javascript, turned instead to the Selenium web driver, which solved the problem in "seconds". I'll have to give Ghost.py a spin :)
deleted 14y ago
[deleted]
kanzure 14y ago
Individuals might also be interested in the reimplemented version of phantomjs in python (pyphantomjs): http://github.com/kanzure/pyphantomjs http://github.com/kanzure/pyphantomjs
- civilian 14y ago
  pyphantomjs is not under active development anymore, unfortunately.
sscheper 14y ago
What is a webkit web client?
- grakic 14y ago
  WebKit is a famous browser rendering engine. Client says that you can use ghost.py to "operate" or "drive" a WebKit instance as a headless browser.
- wisty 14y ago
  Webkit is an open source browser project. It's based on a rewrite of KDE's browser, mostly done by Apple. KDE was going to merge it back into KHTML, but they found Apple hard to work with (Apple made lots of big changes without explaining them very well, and KDE didn't have the resources to keep up). It powers Safari and Chrome, and a lot of smaller projects (mobile browsers, email clients, etc). "Webkit" can refer to the browser (Apple's fork Safari, or a free version of Safari), the rendering engine (WebkitCore), or the rendering and javascript components (WebkitCore and JavaScriptCore). Google uses WebkitCore (or maybe a fork) for rendering, but their own Javascript engine. I'm guessing ghost.py uses WebkitCore and JavaScriptCore, so you can find out how a page will be seen by Safari (and probably Chrome, since Google's JS engine shouldn't be that different in the way it behaves).
greattypo 14y ago
Anyone know - how does it handle file downloads?
clemesha 14y ago
Can this be used to suck in streaming Flash video? There is this streaming camera of the ocean that I check often, but it's Flash and I'd love to check it from my iPhone. Could Ghost.py be used to get the Flash video? (then turn it into images by other means). Thanks.
- pudquick 14y ago
  With the Qt framework, maybe: http://trac.webkit.org/wiki/QtWebKitPlugins http://trac.webkit.org/wiki/QtWebKitPlugins Good luck at it, though - seems the hard way to go about it.
- kanzure 14y ago
  You could just rip the RTMP stream itself, or whatever the source data is. Decompile the swf and check it out for yourself, or the source url for the video feed is probably provided in some lame xml config file. No need to write software around an entire browser to get your ocean feed.
  - eli 14y ago
    Just turn on the Net Inspector in your favorite browser's debugger and start the stream; you'll probably see what url it's loading.
  - skeletonjelly 14y ago
    And hopefully it's not RTMPS!
dlsym 14y ago
Giggle at the ghostie! (Ok - and now burn my karma)
- daeken 14y ago
  I don't know which one scares me more: the fact that there are My Little Pony references on HN, or that I actually got the reference.
DaNmarner 14y ago
This is the Python equivalence of phantom.js, which provides a programming interface for testing rendered web pages without the overhead of actually opening up a browser (a la Selenium).
deleted 14y ago
[deleted]
jc4p 14y ago
Can it run inline Javascript as the page is loaded or do I have to explicitly tell it what JS to run? I want to scrape some pages that use JS packers to obfuscate their code so that it's only loaded by real browsers, but if I just use curl all I see is JS that needs to be evaluated before I can get anything useful out of it.
- tzury 14y ago
  Can you provide some examples of such sites? I would love to learn more about this technique.
  - jc4p 14y ago
    The signature for them is that they start off with: eval(function(p,a,c,k,e,d))... I've seen them most commonly used in not-quite-so-legal streaming websites online where their biggest problem is blocking robots from scraping their site and losing their advertising revenue.
    - tzury 14y ago
      okay then, that's the usual daftlogic's[1] output. I thought you are talking about something new. [1] http://www.daftlogic.com/projects-online-javascript-obfuscator.htm http://www.daftlogic.com/projects-online-javascript-obfuscat...
  - drostie 14y ago
    I sometimes `wget` the URLs in my spambox out of genuine curiosity as to what people are actually sending me, and there are a bunch of common patterns. What's very surprising is the "obfuscated eval" statement -- some term which ultimately evaluates to 'window' is queried for something crazy like: w[(typeof 3)[4] + "va" + (document.body + "")[17]] which ultimately is 'eval'. This is often combined with some sort of self-decrypting almost-binary-looking payload hidden in a div somewhere and requested by div.innerHTML. Replacing the "eval" with "console.log" can give you the decrypted payload, usually a redirect to another redirect to something which runs a Flash script, which is where my analysis stops. I am not sure why they do this. My first thought is "to prevent being automatically taken down by The Man", but The Man could afford to automatically dispose of a computer while monitoring its network traffic, rebooting from a fixed disk image like a LiveCD afterwards. So it shouldn't be too hard to automatically discover the domains, IPs, and malicious programs involved. I don't know why you'd obfuscate a redirection.
NiekvdMaas 14y ago
In order for this to be usable for a broad range of projects, it must contain: * Cookie support (I see this is partially implemented) * File download support * Mouse movement API (move to pos X,Y - click) * NSPlugins support (Flash, etc) Of the latter ones, I cannot find a reference so I think they are not working yet. Once they are implemented, this is a nice alternative to PhantomJS.
- revenz 14y ago
  This is a wrapper around PYQT which is a wrapper around webkit in QT. So if those things are supported in QT and PYQT there is hope.
andreif 14y ago
Earlier discussion: http://news.ycombinator.com/item?id=3896441 http://news.ycombinator.com/item?id=3896441
deleted 14y ago
[deleted]
- lucian1900 14y ago
  No, because that from import will not put 'ghost' in your namespace, only Ghost.
  - RyanMcGreal 14y ago
    Damn, you're absolutely right. That's what I get for reading code too early in the morning.
dylanpyle 14y ago
Has anyone had any experience with both ghost and phantom (or any other options I may not have found), and know how they stack up in terms of rendering speed/etc? I'd imagine they're fairly similar, but if that's not the case I'd be heavily biased towards the faster of the two.
antihero 14y ago
Seems to get a timeout error on a lot of stuff, even if I set the timeout to ~60seconds with wait_timeout :(

173 points by aeurielesn 14y ago

Good! A few days ago I was playing with mechanize to automate some form filling in wordpress posts (iTunes app details, automatically downloaded and then batch-add as post drafts). Gave up by the lack of AJAX-Javascript, turned instead to the Selenium web driver, which solved the problem in "seconds". I'll have to give Ghost.py a spin :)

deleted 14y ago

[deleted]

kanzure 14y ago

Individuals might also be interested in the reimplemented version of phantomjs in python (pyphantomjs): http://github.com/kanzure/pyphantomjs http://github.com/kanzure/pyphantomjs

civilian 14y ago
pyphantomjs is not under active development anymore, unfortunately.

sscheper 14y ago

What is a webkit web client?

grakic 14y ago
WebKit is a famous browser rendering engine. Client says that you can use ghost.py to "operate" or "drive" a WebKit instance as a headless browser.
wisty 14y ago
Webkit is an open source browser project. It's based on a rewrite of KDE's browser, mostly done by Apple. KDE was going to merge it back into KHTML, but they found Apple hard to work with (Apple made lots of big changes without explaining them very well, and KDE didn't have the resources to keep up). It powers Safari and Chrome, and a lot of smaller projects (mobile browsers, email clients, etc). "Webkit" can refer to the browser (Apple's fork Safari, or a free version of Safari), the rendering engine (WebkitCore), or the rendering and javascript components (WebkitCore and JavaScriptCore). Google uses WebkitCore (or maybe a fork) for rendering, but their own Javascript engine. I'm guessing ghost.py uses WebkitCore and JavaScriptCore, so you can find out how a page will be seen by Safari (and probably Chrome, since Google's JS engine shouldn't be that different in the way it behaves).

greattypo 14y ago

Anyone know - how does it handle file downloads?

clemesha 14y ago

Can this be used to suck in streaming Flash video? There is this streaming camera of the ocean that I check often, but it's Flash and I'd love to check it from my iPhone. Could Ghost.py be used to get the Flash video? (then turn it into images by other means). Thanks.

pudquick 14y ago
With the Qt framework, maybe: http://trac.webkit.org/wiki/QtWebKitPlugins http://trac.webkit.org/wiki/QtWebKitPlugins Good luck at it, though - seems the hard way to go about it.
kanzure 14y ago
You could just rip the RTMP stream itself, or whatever the source data is. Decompile the swf and check it out for yourself, or the source url for the video feed is probably provided in some lame xml config file. No need to write software around an entire browser to get your ocean feed.
- eli 14y ago
  Just turn on the Net Inspector in your favorite browser's debugger and start the stream; you'll probably see what url it's loading.
- skeletonjelly 14y ago
  And hopefully it's not RTMPS!

dlsym 14y ago

Giggle at the ghostie! (Ok - and now burn my karma)

daeken 14y ago
I don't know which one scares me more: the fact that there are My Little Pony references on HN, or that I actually got the reference.

DaNmarner 14y ago

This is the Python equivalence of phantom.js, which provides a programming interface for testing rendered web pages without the overhead of actually opening up a browser (a la Selenium).

deleted 14y ago

[deleted]

jc4p 14y ago

Can it run inline Javascript as the page is loaded or do I have to explicitly tell it what JS to run? I want to scrape some pages that use JS packers to obfuscate their code so that it's only loaded by real browsers, but if I just use curl all I see is JS that needs to be evaluated before I can get anything useful out of it.

tzury 14y ago
Can you provide some examples of such sites? I would love to learn more about this technique.
- jc4p 14y ago
  The signature for them is that they start off with: eval(function(p,a,c,k,e,d))... I've seen them most commonly used in not-quite-so-legal streaming websites online where their biggest problem is blocking robots from scraping their site and losing their advertising revenue.
  - tzury 14y ago
    okay then, that's the usual daftlogic's[1] output. I thought you are talking about something new. [1] http://www.daftlogic.com/projects-online-javascript-obfuscator.htm http://www.daftlogic.com/projects-online-javascript-obfuscat...
- drostie 14y ago
  I sometimes `wget` the URLs in my spambox out of genuine curiosity as to what people are actually sending me, and there are a bunch of common patterns. What's very surprising is the "obfuscated eval" statement -- some term which ultimately evaluates to 'window' is queried for something crazy like: w[(typeof 3)[4] + "va" + (document.body + "")[17]] which ultimately is 'eval'. This is often combined with some sort of self-decrypting almost-binary-looking payload hidden in a div somewhere and requested by div.innerHTML. Replacing the "eval" with "console.log" can give you the decrypted payload, usually a redirect to another redirect to something which runs a Flash script, which is where my analysis stops. I am not sure why they do this. My first thought is "to prevent being automatically taken down by The Man", but The Man could afford to automatically dispose of a computer while monitoring its network traffic, rebooting from a fixed disk image like a LiveCD afterwards. So it shouldn't be too hard to automatically discover the domains, IPs, and malicious programs involved. I don't know why you'd obfuscate a redirection.

NiekvdMaas 14y ago

In order for this to be usable for a broad range of projects, it must contain: * Cookie support (I see this is partially implemented) * File download support * Mouse movement API (move to pos X,Y - click) * NSPlugins support (Flash, etc) Of the latter ones, I cannot find a reference so I think they are not working yet. Once they are implemented, this is a nice alternative to PhantomJS.

revenz 14y ago
This is a wrapper around PYQT which is a wrapper around webkit in QT. So if those things are supported in QT and PYQT there is hope.

andreif 14y ago

Earlier discussion: http://news.ycombinator.com/item?id=3896441 http://news.ycombinator.com/item?id=3896441

deleted 14y ago

[deleted]

lucian1900 14y ago
No, because that from import will not put 'ghost' in your namespace, only Ghost.
- RyanMcGreal 14y ago
  Damn, you're absolutely right. That's what I get for reading code too early in the morning.

dylanpyle 14y ago

Has anyone had any experience with both ghost and phantom (or any other options I may not have found), and know how they stack up in terms of rendering speed/etc? I'd imagine they're fairly similar, but if that's not the case I'd be heavily biased towards the faster of the two.

antihero 14y ago

Seems to get a timeout error on a lot of stuff, even if I set the timeout to ~60seconds with wait_timeout :(