27 ms·
Did I understand it right that the inability to locate a single object for a single customer affected all your customers? Almost right. The Tarsnap server agg
by cperciva 16y ago
Did I understand it right that the inability to locate a single object for a single customer affected all your customers?
Almost right. The Tarsnap server aggregates together data from multiple machines into a single S3 object; I don't know how many users had data stored in that object, but it's probably more than 1 and less than 10.
But the problem wasn't really caused by the object going missing; rather, it was caused by S3 doing Something Which S3 Should Never Do, plus the Tarsnap server code not being designed with that possibility in mind. I've since adjusted the code so that an error like this will be handled less severely.
(That said, I doubt I'll ever see this S3 glitch again -- I got a phone call from Amazon providing some additional details about what caused this and it was clear that they were taking it very seriously.)
- jacquesm 16y ago> That said, I doubt I'll ever see this S3 glitch again -- I got a phone call from Amazon providing some additional details about what caused this and it was clear that they were taking it very seriously. They'd better. I'm as surprised as you are (well, probably not because I'm not using S3, but I know a bit about how it is put together and I can't believe the Amazon people are happy about having this happen to them, it's the exact opposite of what should happen in a 'cloud' storage situation). I think there will be some pretty high level meetings on this glitch, the one thing you don't want is customer data going absent-without-leave, even if only on a holiday instead of a permanent departure. Isn't it against your instinct to have data from different customers live in what amounts to be the same file? I understand you've got it encrypted to the hilt but that seems 'un-Colinish' ;)
- die_sekte 16y agoAs far as I understand he is using S3 as a dumb block-level store. He has implemented a file system on top of S3. All customer data is stored in different files, it's just that those files don't map 1:1 to S3 objects.
- rlpb 16y agoThe tarsnap client does the encryption. The server never sees plaintext. I think it's reasonable to consider the tarsnap protocol as adequate separation, since an error on the server wouldn't be able to leak customer data (except for stats about size and access patterns, perhaps, but that could happen regardless of how the data is chunked together).