Having written a web crawler recently, it's surprising just how many misbehaving servers are actually out there. Without proper error handling, a wide randomly

Having written a web crawler recently, it's surprising just how many misbehaving servers are actually out there. Without proper error handling, a wide randomly seeded crawl would hit some form of malformed response or bizarre header within 15 minutes at the most. I eventually gave up on trying to parse all the myriad odd behaviours and now just dump them all onto a blacklist and move the crawler on.