2 ms·

I've read a rumor here of AWS S3 silently dropping objects when your collections get to really gargantuan sizes (billions of objects), and now I wonder if the p

I've read a rumor here of AWS S3 silently dropping objects when your collections get to really gargantuan sizes (billions of objects), and now I wonder if the people who ran into that are using PAR2 to compensate.

dehrmann 6y ago
Just because you said AWS, I was backing up a lot of data in Glacier, so I mailed them a hard drive. I included par files in case something happened. There was a read issue with the hard drive (iirc, I wouldn't reproduce it when I got the drive back, but they wiped it, so who knows), so they stopped the import halfway through. A couple of months later, I noticed a surprisingly large Glacier bill, but I couldn't find the objects. After contacting support, I found out that the temp files were essentially in a hidden archive in my account. I was pretty annoyed by the whole thing.
- yourapostasy 6y ago
  I had wrongly assumed chain of custody manifests are a thing in Snowball and AWS Import/Export. We just discussed on here about ECC memory where the consensus was ECC RAM even in laptops is good (especially at 32GB+), yet somehow mass storage dealing with orders of magnitude greater volume of data is given a pass on transits. Thanks for sharing your experience, good to know I have to preserve our manifests discipline outside our data lakes' borders. Also interesting to find out AWS uses their customers' resources for temporary storage in the transit process, instead of elastically using process-only-bound ephemeral resources outside customer space in a more cloud-native fashion. Temporary consumption in the customers' resource space in a solution pattern gives me nightmare scenarios of stray code that scribbles the temporary objects into customer-owned data, or accidentally dropped into the wrong location and read by customer processes. Would be curious to hear the trade-offs involved in that decision, they could not have made it lightly, I always try to choose fail-safe design modes and at that level of solutioning I'm sure their teams are way smarter than I am so I'd love to learn from this use case.

dehrmann 6y ago

Just because you said AWS, I was backing up a lot of data in Glacier, so I mailed them a hard drive. I included par files in case something happened. There was a read issue with the hard drive (iirc, I wouldn't reproduce it when I got the drive back, but they wiped it, so who knows), so they stopped the import halfway through. A couple of months later, I noticed a surprisingly large Glacier bill, but I couldn't find the objects. After contacting support, I found out that the temp files were essentially in a hidden archive in my account. I was pretty annoyed by the whole thing.

yourapostasy 6y ago
I had wrongly assumed chain of custody manifests are a thing in Snowball and AWS Import/Export. We just discussed on here about ECC memory where the consensus was ECC RAM even in laptops is good (especially at 32GB+), yet somehow mass storage dealing with orders of magnitude greater volume of data is given a pass on transits. Thanks for sharing your experience, good to know I have to preserve our manifests discipline outside our data lakes' borders. Also interesting to find out AWS uses their customers' resources for temporary storage in the transit process, instead of elastically using process-only-bound ephemeral resources outside customer space in a more cloud-native fashion. Temporary consumption in the customers' resource space in a solution pattern gives me nightmare scenarios of stray code that scribbles the temporary objects into customer-owned data, or accidentally dropped into the wrong location and read by customer processes. Would be curious to hear the trade-offs involved in that decision, they could not have made it lightly, I always try to choose fail-safe design modes and at that level of solutioning I'm sure their teams are way smarter than I am so I'd love to learn from this use case.