Just one IP for now. You are right to worry about being blocked from crawling however, it has happened to me already on a few sites. The key things to help mi

Just one IP for now. You are right to worry about being blocked from crawling however, it has happened to me already on a few sites. The key things to help mitigate against this are: 1. Always identify your crawler via a consistent user-agent string, that explains its a web search crawler and not a generic web browser. 2. Always obey the directives in robots.txt. 3. Make sure your crawler is not too aggressive (low frequency of requests). (updated for formatting)