12 ms·
Nope. Sorry. :) While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
by jeremybmerrill 13y ago
Nope. Sorry. :)
While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
- rst 13y agoPerhaps better phantomjs (http://phantomjs.org http://phantomjs.org), perhaps using casper (http://casperjs.org http://casperjs.org) on top to handle some of the glue code. Phantomjs is a full headless browser (it'll give you screenshots of the pages it downloads if you want them); casper is a library that makes sequencing tasks somewhat easier. Node, by itself, doesn't have full versions of a lot of the objects that Javascript on the pages would refer to (DOM, event model, etc.); phantomjs gives you all of that.
- baudehlo 13y agoOr do what we do at Hubdoc, and use both Node and Phantom. Node for performance where it's possible, and Phantom where the site has been built in such a way that scraping in Node becomes not worth the effort of figuring out all the weird stuff they've done in client side JS. We maintain a Node to Phantom bridge for this: https://github.com/baudehlo/node-phantom-simple https://github.com/baudehlo/node-phantom-simple
- kanzure 13y agoCurious, you use the webserver module in phantomjs, is that right? And that's how you do the inter-process communication? I'm curious how you chose that over websockets, or over HTTP polling from your phantomjs client against a local node server.. What about using something like node-gir, or whatever appjs does to combine the event loops of node/v8 and chromium/v8?
- saryant 13y agoI wrote a backend system using Phantom and Akka to generate graphs using D3 and rasterize them into PNGs and put them into user-specific emails. Phantom has some quirks but overall it's pretty solid.