Another good trick is to melt the fuse if the request is merely slow. This can often back off before the timeout scenario occurs. I used this to great advantage

Another good trick is to melt the fuse if the request is merely slow. This can often back off before the timeout scenario occurs. I used this to great advantage when I crawled all duels of Quake Live years ago. If the site became slow, I would cease crawling for 15 minutes, and fuse handled that nicely.