A sample crawler i wrote to harvest Yelp results, Feel free to gain insight on how it was written. Might not work as yelp might have had cosmetic changes. But t

A sample crawler i wrote to harvest Yelp results, Feel free to gain insight on how it was written. Might not work as yelp might have had cosmetic changes. But the theme would help you write one on your own ! https://github.com/deepanprabhu/yelp-crawler https://github.com/deepanprabhu/yelp-crawler . I also have an advanced scraper, than can harvest AJAX heavy site like http://venture-capital-firms.findthecompany.com/ http://venture-capital-firms.findthecompany.com/. I completely scraped their site, using a chrome plugin, exporting results through a web server. Kind of a complex procedure as we have to be inside a live browser to hijack their results. The VC site, even avoid headless browsers so it was tricky. I can share the code, in case you are interested. And scaling scraping, is an interesting process.