4 ms·
Do you have multiple IPs? I am trying to build something which needs just the published at and updated at date fields for thousands of links and I am afraid my
by busymom0 1y ago
Do you have multiple IPs? I am trying to build something which needs just the published at and updated at date fields for thousands of links and I am afraid my IP will get blocked quickly.
- saltysalt 1y agoJust one IP for now. You are right to worry about being blocked from crawling however, it has happened to me already on a few sites. The key things to help mitigate against this are: 1. Always identify your crawler via a consistent user-agent string, that explains its a web search crawler and not a generic web browser. 2. Always obey the directives in robots.txt. 3. Make sure your crawler is not too aggressive (low frequency of requests). (updated for formatting)