Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a si

Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.