3 ms·
so how polite does one need to be? One hit per x seconds?
by joshu 14y ago
so how polite does one need to be? One hit per x seconds?
- yorhel 14y agoIf the /robots.txt does not mention a Crawl-delay, one page per 3 seconds is often a safe value. Of course this rather heavily depends on the site. In any case, if you have any specific need, always contact the people responsible for the site. I occasionaly run custom queries against the database on request, for example.