I can't imagine using anything other than a stream interface when dealing with web requests in a crawler. You need that to protect against not only these types

I can't imagine using anything other than a stream interface when dealing with web requests in a crawler. You need that to protect against not only these types of shenanigans, but also large or slow responses.