5 ms·
Noodlejs - Node.js web scraper
- AnSavvides 13y agoI get a 404 when I try to access the project's GitHub page or download it.
- rk0567 13y agosame here. (probably, they've not made the repository public yet)
- techaddict009 13y agoI too got the same error. Seems like there is some issue.
- esamek 13y agoDitto
- deleted 13y ago[deleted]
- premasagar 13y agoWhoops. Now fixed. Not sure how that happened.
- hmottestad 13y agoI can recommend formatting the JSON. Like this: demoElement.innerHTML = JSON.stringify(data, null, 4); And then use "<pre>" tag in the demoElement.
- egeozcan 13y agoThat is the first thing I did using the web inspector. One could also use a js-beautifier I guess.
- premasagar 13y agoA good idea. I've just added that to the docs site.
- BaconJuice 13y agoThis is very cool! Thanks for sharing.
- lkinc 13y agoREADME https://npmjs.org/package/noodlejs https://npmjs.org/package/noodlejs Npm package npm install noodle Npm tarball curl -kO https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz
- lkinc 13y agonoodle should be noodlejs
- harryf 13y agoHow does this compare with Cheerio? ( see discussion on HN from yesterday - https://news.ycombinator.com/item?id=6273905 https://news.ycombinator.com/item?id=6273905 )
- irickt 13y agoI can't answer your question without reading further, but I note that Cheerio is a dependency of Noodle.
- premasagar 13y agoIt uses Cheerio for scraping, but it also lets you use selector queries to grab a subset of documents, whether the document is HTML, JSON or another format. It also lets you request multiple documents in one HTTP request, and a few other things.
- nonchalance 13y agoFor those looking for the source, you can manually get it: http://npm.im/noodlejs http://npm.im/noodlejs is the NPM package https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz is the package source $ curl -kO https://registry.npmjs.org/noodlejs/-/noodlejs-0.2.0.tgz The code is pure-JS (not a C++ addin)
- premasagar 13y agoI didn't submit this, but I did oversee the project. Noodle is a Node-based web scraper that also handles JSON, XML and other file formats. It was initially built as a hack project to replace a core subset of YQL. All responses can be served as JSONP, to allow for cross-domain scraping from a website's front-end. Selector queries can be used to grab a subset of a document - e.g. CSS selectors for HTML documents and dot-notation for JSON documents. It lets you request multiple documents in a single HTTP request, and a few other things. I helped to guide an intern, Aaron Acerboni, at my company, Dharmafly, when he built it last year.
- bdcravens 13y agoUnless I'm reading wrong, this scraper is a slightly prettier API to what most do: a simple curl, structuring the data with selectors. Like most it ignores page requiring state or anything dynamically generated on the client.
- Misiek 13y agohow to get the list of urls in this example on http://noodlejs.com/ http://noodlejs.com/? the selector: 'h3.r a' allows only to get the list of names
- premasagar 13y agoSimply replace `extract: text` with `extract: href` for the anchor's href attribute. You can do this with any kind of attribute.