5 ms·
Ask HN: Web Scraping with Node.js
Anyone out there have a cachet of good Node web scraping libs, and/or other resources? Opinions?
I have the intuition that my Google searches are a bit inadequate. Thank you!
- pesfandiar 10y agoShould it be necessarily with Node.js? I've tried to find a good toolset in JS, but it seems other languages (e.g. Python and Java) are a lot more mature in scraping.
- jneumann004 10y agoI haven't used this for web scraping, but have you tried out http://phantomjs.org/ http://phantomjs.org/. It's a great tool that lets your process webpages; I don't think it would be that difficult to set it up to scrape sites.
- Nilef 10y agoTry this out https://github.com/IonicaBizau/scrape-it https://github.com/IonicaBizau/scrape-it
- iends 10y agoCheck out https://github.com/cheeriojs/cheerio https://github.com/cheeriojs/cheerio
- tmpfs 10y agoI found https://github.com/cgiffard/node-simplecrawler https://github.com/cgiffard/node-simplecrawler to be good, I used it for https://github.com/tmpfs/linkdown https://github.com/tmpfs/linkdown - a cli for validating web pages etc.
- paulzerkel 10y agoHere is a place to start: https://github.com/lorien/awesome-web-scraping/blob/master/javascript.md https://github.com/lorien/awesome-web-scraping/blob/master/j... You might want to check out the Nightmare project (https://github.com/segmentio/nightmare https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.