5 ms·

This is in response to all the node/jsdom/jquery scraping posts that are popular lately. JSDom is hopeless for scraping - try parsing some slightly malformed HT

This is in response to all the node/jsdom/jquery scraping posts that are popular lately. JSDom is hopeless for scraping - try parsing some slightly malformed HTML..

richcollins 16y ago
You should consider creating a binding to libsgml. It's written in C and its permissive.
- chrisohara 16y ago
  Interesting, thanks. I've been looking for a decent C lib to bind with node and play around with
tszming 16y ago
Currently I am using Selenium RC + jQuery for scraping. It is slow but it is the most reliable (due to using a real browser) solution you can get. Anyway, will try your method soon.
- tworats 16y ago
  Could you provide mode details on this, either in the comments here or in a blog post? I almost went with Selenium but ended up with a custom system to drive Firefox, would love to learn more about what the Selenium based solution looks like.
  - tszming 16y ago
    Just a few lines of code, no magic needed. Example: https://github.com/tszming/Selenium-Google-Scrapper https://github.com/tszming/Selenium-Google-Scrapper
DTrejo 16y ago
Hey Chris, I was just trying to share a few things I'd learned. I know I haven't done as much scraping as others (like yourself and richcollins). Glad I've helped get some discussion going :)
- chrisohara 16y ago
  Hey David, it wasn't a stab at your blog - your post was great - anything that builds some more interest in node.js is positive :) I just hate reading about people having trouble with JSDom and putting it down to the node platform. JSDom is an excellent parser, it just fails miserably when you feed it malformed HTML, and as we know, a majority of the internet falls in to this category. I needed a framework that could scrape anything on the web so I built it myself
  - tmpvar 16y ago
    just to be clear, jsdom is not a parser. By default it uses node-htmlparser which is not very lenient. Have you tried using Aria's html5 parser? I hear it works better with malformed markup.

This is in response to all the node/jsdom/jquery scraping posts that are popular lately. JSDom is hopeless for scraping - try parsing some slightly malformed HTML..

richcollins 16y ago

You should consider creating a binding to libsgml. It's written in C and its permissive.

chrisohara 16y ago
Interesting, thanks. I've been looking for a decent C lib to bind with node and play around with

tszming 16y ago

Currently I am using Selenium RC + jQuery for scraping. It is slow but it is the most reliable (due to using a real browser) solution you can get. Anyway, will try your method soon.

tworats 16y ago
Could you provide mode details on this, either in the comments here or in a blog post? I almost went with Selenium but ended up with a custom system to drive Firefox, would love to learn more about what the Selenium based solution looks like.
- tszming 16y ago
  Just a few lines of code, no magic needed. Example: https://github.com/tszming/Selenium-Google-Scrapper https://github.com/tszming/Selenium-Google-Scrapper

DTrejo 16y ago

Hey Chris, I was just trying to share a few things I'd learned. I know I haven't done as much scraping as others (like yourself and richcollins). Glad I've helped get some discussion going :)

chrisohara 16y ago
Hey David, it wasn't a stab at your blog - your post was great - anything that builds some more interest in node.js is positive :) I just hate reading about people having trouble with JSDom and putting it down to the node platform. JSDom is an excellent parser, it just fails miserably when you feed it malformed HTML, and as we know, a majority of the internet falls in to this category. I needed a framework that could scrape anything on the web so I built it myself
- tmpvar 16y ago
  just to be clear, jsdom is not a parser. By default it uses node-htmlparser which is not very lenient. Have you tried using Aria's html5 parser? I hear it works better with malformed markup.