5 ms·
i found node.js is a phenomenal scraping tool. There's also a pretty simple / easy framework for these tasks called node.io (http://node.io/ http://node.io/).
by AdrianRossouw 15y ago
i found node.js is a phenomenal scraping tool. There's also a pretty simple / easy framework for these tasks called node.io (http://node.io/ http://node.io/).
Being able to pop up a jsdom, and extract data from the page using jquery is a lot of fun.
- rhizome 15y agoIf the home site for a language or framework returns 404/500s, it's not professional quality.
- mistercow 15y agoI don't think that's quite fair. No server is 100% reliable, and that says nothing about the quality of the code of the project itself.
- rhizome 15y agoLet's not make excuses for them. I wouldn't say that it says "nothing" about the practices and priorities oriented around (and consequently through) the project, especially when the website apparently consists of a single small static HTML page.
- chrisohara 15y agoIt's a project created and maintained entirely in my free time. I provide free support when someone needs help and I fix bugs when someone files an issue. I've written documentation and a guide in the project's Github wiki. This is all done outside of full-time work. Forgive me if I don't spend more time improving the website and uptime when it's an endeavour that brings me no income whatsoever. You're the type of person that makes me hate open-source sometimes.
- rhizome 15y agoDon't have a cow, man, it's nothing personal. What does open source have to do with the reason your site was down?
- mistercow 15y ago>Don't have a cow, man, it's nothing personal. I think that impugning the professionalism of someone's work because their project's home page was down once is pretty darned personal.
- rhizome 15y agoI know nothing of anybody behind it, how could my intent possibly be personal?
- mistercow 15y agoThat argument doesn't hold water. I don't know you, but if I say "From the looks of your page many9s.com , I'd say you're an amateur with no design sense", then I'm personally attacking you.
- rhizome 15y agoRight, because you're making it personal. How is it that you go from a simple webpage to a person's skills? I certainly didn't do that.
- chrisohara 15y agoObligatory link to your single page static HTML homepage for comparison: http://many9s.com/ http://many9s.com/
- rhizome 15y agoWas it down when you visited it?
- chrisohara 15y agoIf it looks terrible, it's not professional quality. Right?
- mistercow 15y ago> I wouldn't say that it says "nothing" about the practices and priorities oriented around (and consequently through) the project, especially when the website apparently consists of a single small static HTML page. It says nothing. All servers go down from time to time. This one was up when I went to it before, and it was up when I tried it again just now. You say it was down for you when you visited it. Fine. That happens sometimes.
- lancefisher 15y agoI've found jsdom to break for quite a bit of real-world crap HTML. Next time, I'm going to try driving phantom.js with node using https://github.com/sgentle/phantomjs-node https://github.com/sgentle/phantomjs-node I've had pretty good luck with phantom.js, but it is somewhat difficult to debug.
- chrisohara 15y agoFYI, JSDOM isn't the default parser in node.io. It uses a faster and more forgiving parser if you can survive with a subset of jQuery functionality (https://github.com/chriso/node.io/wiki/API---CSS-Selectors-and-Traversal-methods https://github.com/chriso/node.io/wiki/API---CSS-Selectors-a...)