read the oreilly book. On your sources: read ToS, robots.txt, and don't pound on anybody servers, even if you're iwthin ToS. http://news.ycombinator.com/item?

read the oreilly book. On your sources: read ToS, robots.txt, and don't pound on anybody servers, even if you're iwthin ToS. http://news.ycombinator.com/item?id=158902 http://news.ycombinator.com/item?id=158902 theres a bunch of python scrapers in py pkg index http://pypi.python.org/pypi?%3Aaction=search&term=crawler&submit=search http://pypi.python.org/pypi?%3Aaction=search&term=crawle... until you're sure your IP won't be blacklisted, test crawler from your friend's houses. I've heard that many cablecos/telcos regard a request for a new IP as very suspicious!