Any heavy scraping can be detected and blocked (or worse, corrupted), so you'd need a pool of proxies and some smart robots. You are essentially depending on yo

Any heavy scraping can be detected and blocked (or worse, corrupted), so you'd need a pool of proxies and some smart robots. You are essentially depending on your competitors to have data available and not change their website layout. By the way, if you are using interesting scraping techniques, would you mind posting a thread or two on HN?