4 ms·
If you want to go serious about web crawling and/or web scraping (within legal boundaries of course), you want to use Node.js and appropriate modules (don't rem
by ypcx 15y ago
If you want to go serious about web crawling and/or web scraping (within legal boundaries of course), you want to use Node.js and appropriate modules (don't remember the exact names right now). This is because Node.js being based on the V8 JavaScript engine, can completely emulate a real web browser - it can load and parse the HTML, as well as JavaScript. And many sites won't load properly without JavaScript.
- masklinn 15y agoWhat you're saying makes no sense whatsoever, at any level of resolution. Chrome's rendering engine, and the library used to deal with parsing HTML and building a DOM tree is Webkit's Webcore[0]. V8 and Webcore are not the same thing and V8 does not provide a DOM implementation (that's webcore's job) nor does it handle any HTML parsing (that's also) webcore's job. V8 is a javascript VM. That's it. It does not "emulate a real web browser" (let alone completely), and nor does Node. [0] http://trac.webkit.org/browser/trunk/WebCore?rev=64712 http://trac.webkit.org/browser/trunk/WebCore?rev=64712
- ypcx 15y agoThat's why I said emulate. V8 (Node) with appropriate modules can emulate the browser - both parse the DOM, and then run scripts on that DOM. PHP/Perl/etc. can't do that. Java could do that with Rhino I assume, but I'd say V8 is much closer. I'm also not saying anything about emulating exactly Chrome. I wish I had time to dig up that module for Node now, but I don't (I don't remember the name).