I feel like more companies are building their businesses around web crawling and parsing data. There are lots of players in the eCommerce space that monitor pri

I feel like more companies are building their businesses around web crawling and parsing data. There are lots of players in the eCommerce space that monitor pricing, search relevance, and product integrity. Each one of these companies has to build some sort of a templating system for defining crawl jobs, a definition of parsing rules to extract the data, and monitoring system to alert when the underlining HTML of a site has changed from their predefined rules. I'm interested in these aspects. Building a distributed crawler is easier than ever.