6 ms·
Web scraping: Reliably and efficiently pull data from pages that don't expect it
- AdrianRossouw 15y agoi found node.js is a phenomenal scraping tool. There's also a pretty simple / easy framework for these tasks called node.io (http://node.io/ http://node.io/). Being able to pop up a jsdom, and extract data from the page using jquery is a lot of fun.
- rhizome 15y agoIf the home site for a language or framework returns 404/500s, it's not professional quality.
- mistercow 15y agoI don't think that's quite fair. No server is 100% reliable, and that says nothing about the quality of the code of the project itself.
- rhizome 15y agoLet's not make excuses for them. I wouldn't say that it says "nothing" about the practices and priorities oriented around (and consequently through) the project, especially when the website apparently consists of a single small static HTML page.
- chrisohara 15y agoIt's a project created and maintained entirely in my free time. I provide free support when someone needs help and I fix bugs when someone files an issue. I've written documentation and a guide in the project's Github wiki. This is all done outside of full-time work. Forgive me if I don't spend more time improving the website and uptime when it's an endeavour that brings me no income whatsoever. You're the type of person that makes me hate open-source sometimes.
- rhizome 15y agoDon't have a cow, man, it's nothing personal. What does open source have to do with the reason your site was down?
- mistercow 15y ago>Don't have a cow, man, it's nothing personal. I think that impugning the professionalism of someone's work because their project's home page was down once is pretty darned personal.
- rhizome 15y agoI know nothing of anybody behind it, how could my intent possibly be personal?
- mistercow 15y agoThat argument doesn't hold water. I don't know you, but if I say "From the looks of your page many9s.com , I'd say you're an amateur with no design sense", then I'm personally attacking you.
- rhizome 15y agoRight, because you're making it personal. How is it that you go from a simple webpage to a person's skills? I certainly didn't do that.
- chrisohara 15y agoObligatory link to your single page static HTML homepage for comparison: http://many9s.com/ http://many9s.com/
- rhizome 15y agoWas it down when you visited it?
- chrisohara 15y agoIf it looks terrible, it's not professional quality. Right?
- mistercow 15y ago> I wouldn't say that it says "nothing" about the practices and priorities oriented around (and consequently through) the project, especially when the website apparently consists of a single small static HTML page. It says nothing. All servers go down from time to time. This one was up when I went to it before, and it was up when I tried it again just now. You say it was down for you when you visited it. Fine. That happens sometimes.
- lancefisher 15y agoI've found jsdom to break for quite a bit of real-world crap HTML. Next time, I'm going to try driving phantom.js with node using https://github.com/sgentle/phantomjs-node https://github.com/sgentle/phantomjs-node I've had pretty good luck with phantom.js, but it is somewhat difficult to debug.
- chrisohara 15y agoFYI, JSDOM isn't the default parser in node.io. It uses a faster and more forgiving parser if you can survive with a subset of jQuery functionality (https://github.com/chriso/node.io/wiki/API---CSS-Selectors-and-Traversal-methods https://github.com/chriso/node.io/wiki/API---CSS-Selectors-a...)
- superasn 15y agoMy favorite language when it comes to scraping the web has always been Perl. With tools like LWP, Mechanize and Win32::Mechanize (OLE) scraping any site is a breeze. Unfortunately, I haven't seen many good modules on CPAN for DOM processing. Of course there is the Tokeparser and XPath but those generally don't work well with street-HTML (which most sites are) and are no-where near as fast or friendly as jQuery selectors. By the way, there is one module called pQuery which is Perl's port of jQuery but that only supports a handful of selectors and doesn't work with Mechanize. If only there was a module in Perl which could marry Mechanize and jQuery (without using IE or OLE) it would make the best scraper in the world!
- mapgrep 15y agoRegarding Perl DOM parsing, you might want to check out HTML::Query. Its selectors still fall short of jQuery but when I was evaluating it against pQuery it came out ahead (IIRC).
- mickeyp 15y agoActually, libxml (lxml for Python) is very, very good at handling real-life HTML content. I'm currently on a contract where I do a significant amount of ETL and web scraping as part of the project, and I almost exclusively use lxml and XPath for parsing real-world HTML. XPath is, without a doubt, the best tool for DOM manipulation that I can think of. And that's just XPath 1.0 -- 2.0 is reputedly even better, but no 2.0 support is forthcoming for lxml as near as I can tell.
- sureshv 15y ago+1 for lxml and related features (XPath, Html Cleaner & Soup-parser).
- gauravk92 15y agoXpath is good when the content is well formed, but usually doesn't work well with even slightly messed up tags. Mojolicious supports broken content much better. And it has full css3 selector support, which we all know is hands down the best way to access Dom elements.
- epoxyhockey 15y agoMy last web scraping activity consisted of using PhantomJS to drive and injecting my own javascript that submitted the pertinent data to my own web service. http://phantomjs.org/ http://phantomjs.org/
- mistercow 15y agoWow, I had used node+jsdom+jQuery to do scraping, which was a big step up from anything I had done previously. But after playing around with PhantomJS, I feel like I finally found the hammer in the toolbox after years of trying to pound nails in with duct-taped-together screwdrivers.
- epoxyhockey 15y agoCheck that.. I wanted to use js to submit to my own web service, but forgot about the cross-domain issues, so I just used console.log to output the data. I suppose I could have used the iframe querystring parameter trick.
- easp 15y agoHmmm, seems like something you could have worked around by running phantom.js against a proxy that intercepted and rerouted requests to the data collection server.
- zackzackzack 15y agoIt's hella slow but there is always this: https://github.com/sgentle/phantomjs-node https://github.com/sgentle/phantomjs-node
- uptown 15y ago"I suppose I could have used the iframe querystring parameter trick." How's that work? Pass whatever data you want to process with your web service as URL parameters to the web service src URL of your iframe and it circumvents cross-site-scripting protections?
- deleted 15y ago[deleted]
- deleted 15y ago[deleted]
- hk_kh 15y agoAlso, tor + privoxy as a rule-of-thumb when your scraper IP is getting banned. And - I know, it is (now) so old - memcached as a good place to store things.
- duskwuff 15y agoWhen your scraper IP is getting banned, that's typically a sign that you should stop, talk to the site owner, and/or reconsider what you're doing.
- hk_kh 15y agoThat's not always the case. While government or private contractors do not allow to gather that data by any citizen (there are examples of cities allowing it), I find myself in the right to do so, and also, in the right to redistribute that data freely so other developers can play / investigate / learn with it. Why? Well, for starters: 1) Their own apps suck (or are inexistent) 2) They don't want to help their own users. 3) It's fun 4) Their services are paid with public money. 5) It raises awareness on the need of public data legislation. There's a lot more to talk on the subject. If you want to check it, http://citybik.es http://citybik.es I am helping projects and visualizations like: http://bikes.oobrien.com/ http://bikes.oobrien.com/ or my own http://citybik.es/realtime/ http://citybik.es/realtime/
- nicksergeant 15y agoKind of surprised I haven't seen PyQuery in here yet: http://pypi.python.org/pypi/pyquery http://pypi.python.org/pypi/pyquery -- highly recommend. I've used it on several projects.
- uptown 15y agoRecently I've been using this method to inject JQuery into pages that may not already have it available, then executing some custom javascript to extract the page contents: http://www.guru.net.nz/blog/2009/06/screen-scraping-with-jquery.html http://www.guru.net.nz/blog/2009/06/screen-scraping-with-jqu...
- elchief 15y agoThis one uses webdriver to get the full html from ajax pages: http://vancouverdata.blogspot.com/2012/02/less-painful-ajax-javascript-web.html http://vancouverdata.blogspot.com/2012/02/less-painful-ajax-...
- salvadors 15y agohttps://scraperwiki.com/ https://scraperwiki.com/ is a great source of thousands of pre-written scrapers to use / copy / extend etc. It's sort of like github except you can actually schedule the scrapers to run at regular intervals, and then just access the scraped data over a standard API.
- ricksta 15y agoAnyone recommend some equivalent tools for Ruby?
- rb2k_ 15y agoThere is Mechanize (http://mechanize.rubyforge.org/ http://mechanize.rubyforge.org/) which uses Nokogiri (nokogiri.org/) internally and thus nicely supports CSS and XPATH expressions. There is also Capybara, usually used as a testing framework, but you can easily navigate, chose a backend (Selenium/Webkit for compatibility, mechanize for speed): https://github.com/jnicklas/capybara https://github.com/jnicklas/capybara
- tszming 15y agoWe at Rewritely (http://www.rewritely.com http://www.rewritely.com) have quite some experience in dealing large scale content migration for clients (not just a single HTML page, but a whole site) mainly using scraping techniques. We have seen so many invalid, funny, ugly markups and edge cases and eventually, we found out the only reliable parser is actually the "browser" you use everyday - which is heavily tested by hundreds of millions of users! We chose to run the scraper in a real browser so what you scrape is what you see, not to mention you can fire up events, Ajax requests, inject JavaScript or even play Flash video... they are possible only when you have a real browser. Also, the scraper codes need to be concise and expressive (keys: maintainable & testable), because your codes are going to break sooner or later - if you are doing serious scraping business. Less LOC = more easy to change. JavaScript & jQuery are the obvious winners in this category and what is more, they are fun to work with. Disclaimer: we have no relationship with PhantomJS :)