4 ms·
After a decently large RAID failure, I needed to gzip and send as many large files and send it over to S3 as quickly as possible on the risk of another failure.
by bpchaps 11y ago
After a decently large RAID failure, I needed to gzip and send as many large files and send it over to S3 as quickly as possible on the risk of another failure. The script would gzip the file and then sync it up to s3, all in its own backgrounded processes. If two large files would get sent at the same time, both would die, then /leave incomplete files/.
After leaving that running over night, all of the files appeared to be uploaded... until the owner of the company needed to use them.
I'm still not sure if that's an exceptional use case, but it left a pretty bad taste in my mouth about S3 ever since.
- donavanm 11y agoIt sounds like you were missing the "Content-MD5" header on your put requests. As i recall S3 will return an HTTP error response if the complete object does not match the Content-MD5 the client sends. The other issue with the HTTP protocol is that the request body doesnt have a mandatory delimiter. The client/server cant really distinguish between a terminated TCP connection and a complete HTTP body without the optional Content-Length/Content-MD5 headers. It really sounds like one or more of your latge files were timing out somewhere and the checksum was t sent.