A nice write up. At a previous company I built a solution (was forced into java but ultimately the same process...) that used many of these techniques. Some

A nice write up. At a previous company I built a solution (was forced into java but ultimately the same process...) that used many of these techniques. Some suggested next steps/additional enhancements if you need to do this repeatedly and at scale. Implement global throttling on a per domain basis. Consider some abstraction. I implemented an abstract fetcher (with a number of concrete fetchers that were runtime selectable), and an abstract/concrete parser. Compose a Scraper with these two things. Allow for a runtime switch that will determine which fetcher to use (javascript enabled, straight requests, etc.). If you want to get really fancy, in your database of urls, you can flag urls that need to use a heavier, full fledged browser. For the fetcher, use the selenium bindings. We tested phantomjs and chrome and chrome outperformed phantom. It might have been the java bindings (ghostdriver) but w/e, something you just have to test for yourself. Once we settled on chrome, I built a chrome plugin to block ads and other unrelated calls. This adds LOADs of time. Its pretty tricky but you can inject in a list of well crafted regexes and it drops initial load times dramatically. For the parser, you may want to consider a fallback system. Often times the particular piece of data you want (say a title) can be found in a handful of places on the page. It will make your parsing much more reliable. Compose a 'bot runner' from the Scraper. We had json documents that described fields we were abstracting and all the fallback rules used to locate the needed data. Lastly, the bot runner can be expanding to include things like navigation and other fancier tricks. If you go for broke, build a system for generating bots, think chrome plugin. Don't forget, pruning dead URLs is a tricky little problem but an important one. To scale this whole operation linearly, we used a queue (redis at first, eventually kafka) and storm. Storm allowed us to arbitrary expand and contract our bot runners. Scraping is a problem that just about everyone encounters and alot of the most standard solutions seem to really fall short. Your article is an excellent start.