It sounds like you're pretty green in this area, so I'll give you a few tips from what I've read. * You may not be allowed to crawl everything, and your use of

It sounds like you're pretty green in this area, so I'll give you a few tips from what I've read. * You may not be allowed to crawl everything, and your use of that data may not always be in line with sites' terms of service. Just be aware. * It is indeed an ungodly task to crawl the web. To do the whole thing, you'd probably need thousands of servers (to crawl in a reasonable amount of time). * 80legs offers you their crawling infrastructure using cloud based crawlers. Check them out. What are you wanting to crawl / what ultimate use?