Let us flip this around: If your crawler regularly knocks websites offline, you've clearly done something wrong. There's no chance every single website in exis

Let us flip this around: If your crawler regularly knocks websites offline, you've clearly done something wrong. There's no chance every single website in existence is going to have a flawless setup. That's guaranteed simply from the number of websites, and how old some of them are.