Its also missing some form of lock to prevent slow requests cascading (to the same host or IP) when the next instance of your crawler comes along. I did this on

Its also missing some form of lock to prevent slow requests cascading (to the same host or IP) when the next instance of your crawler comes along. I did this once where I had a crawler spin up in a cron job. No problem, unless the previous crawler was still running. Speaking from experience its far better to lock your crawler down from the beginning then to do it after you kill someones site. I did expose a few slow pages for some sites, but not before crashing the whole thing.