With so many single-page and ajax-powered websites these days, I've pretty much abandoned traditional fetching tools like Curl; in favor of headless browsers li

With so many single-page and ajax-powered websites these days, I've pretty much abandoned traditional fetching tools like Curl; in favor of headless browsers like PhantomJS, Selenium, etc. Here's a pretty good list: http://stackoverflow.com/questions/18539491/headless-browser-and-scraping-solutions http://stackoverflow.com/questions/18539491/headless-browser... Of course this only covers the actual gathering of the content, not storing/indexing in an efficient way.