I've had a lot of success scraping websites with Capybara [1]. It's intended for writing acceptance tests of web apps, but it works remarkably well for scraping

I've had a lot of success scraping websites with Capybara [1]. It's intended for writing acceptance tests of web apps, but it works remarkably well for scraping websites. It's written in Ruby, but the DSL it provides for interacting with web pages should be pretty understandable to anybody who's programmed before. It also supports multiple browsers, which means you can tradeoff along these axes: - Headless vs. Not - JS support vs. Not I put a repo together with a sample script [2] for scraping leads off of a website which I will not name, but whose name rhymes with 'help'. It uses the PhantomJS browser for headless JS support. It also includes a Vagrantfile so you can avoid installing all the dependencies on your local machine. [1]: https://github.com/jnicklas/capybara https://github.com/jnicklas/capybara [2]: https://github.com/toasterlovin/scraping-yalp https://github.com/toasterlovin/scraping-yalp