3 ms·
The problem I've had with using jsdom in scraping web pages is that it is not very forgiving about bad HTML. There are so many pages in the wild that have malfo
by lancefisher 15y ago
The problem I've had with using jsdom in scraping web pages is that it is not very forgiving about bad HTML. There are so many pages in the wild that have malformed HTML, and jsdom just pukes on it. I started using Apricot [1] which uses HtmlParser [2] which has been better. I'd like to hear what others are using to scrape bad webpages.
[1] https://github.com/silentrob/Apricot https://github.com/silentrob/Apricot
[2] https://github.com/tautologistics/node-htmlparser https://github.com/tautologistics/node-htmlparser
- RyanMcGreal 15y agoI've been spoiled rotten by the magnificent BeautifulSoup library for Python.
- gikrauss 15y agoA couple of months ago I posted a very similar article that uses node.js, request and jQuery to achieve the same goal. In this case, as long as jQuery can handle the response you shouldn't have any problem with malformed HTML...
- gikrauss 15y agoI forgot the link http://blog.devartis.com/2012/01/05/scraping-websites-having-fun-with-node-js-and-jquery/ http://blog.devartis.com/2012/01/05/scraping-websites-having... :S
- tluyben2 15y agoI added this for that reason; https://github.com/tautologistics/node-htmlparser/issues/2 https://github.com/tautologistics/node-htmlparser/issues/2 The issue is, as the above thread shows, that the creator of node-htmlparser does not want to support illegal HTML. Unfortunately that is not realistic.