If you are going to scrape other people's web sites there are a few basic courtesies that you are expected to observe. 1. Respect the instructions in robots.tx

If you are going to scrape other people's web sites there are a few basic courtesies that you are expected to observe. 1. Respect the instructions in robots.txt, if they don't want you crawling anything under /foo/* don't request /foo/bar or /foo/baz. A corollary is to identify your bot as a robot in the UserAgent string you send. 2. Observe response times and be gentle with the sites you are scraping, remember that you are not a paying customer and that you should not be requesting pages faster than the site can produce them. If response times suddenly rise, back off the rate at which you are requesting pages. Similarly if you start getting 500's or other errors stop and wait at least 10 minutes before resuming. FWIW, IANAL, etc. I've been told that it's perfectly fine to read information that is publicly posted to the web, the place you get in trouble is where you start republishing material without attribution and without transforming it in some manner. I have worked with sites that gathered a lot of pricing information in specific product categories in order to generate pricing recommendations for merchants, and there was no problem with using that information that I know of. But your specific situation may be different enough that my experience is not at all applicable. Your mileage will vary, ask your own lawyer, Hacker News is not legal advice.