5 ms·

I have Photos set to download all the files locally, and those get backed up with Time Machine and Backblaze. People forget even s3 will lose files. The reliab

by pram 3y ago

I have Photos set to download all the files locally, and those get backed up with Time Machine and Backblaze. People forget even s3 will lose files. The reliability is high sure but it’s still sub 100%

stefandesu 3y ago
My concern about my own backup strategy (which, for Photos, is exactly the same as yours) is that I don't have a system in place that notifies me if files get corrupted or randomly deleted, and I'm not sure how I would go about implementing something like that. Any ideas?
- pram 3y ago
  I use a synology for TM (raid 1) and it uses btrfs which has checksums to prevent bitrot. Either that or something running ZFS would be the obvious options: https://kb.synology.com/en-us/DSM/tutorial/How_to_enable_file_self_healing_on_DSM https://kb.synology.com/en-us/DSM/tutorial/How_to_enable_fil...
sitzkrieg 3y ago
s3 losing a file is a statistical anomaly and would get a lot more than a blog post and support deflection
nicoburns 3y ago
Has S3 ever lost a file yet? I know that the reliability isn't 100% as that isn't possible in the real world. But I've never heard a report of a file that was confirmed as stored being lost, and I was under the impression that this was in the category "bound to happen eventually, but hasn't happened yet".
- michaelt 3y ago
  Probably not in the way you're thinking. But with cloud systems you can always lose your files when a hacker gets access to your root account and holds it hostage, or you lose the phone with the 2fa secret on, or your payment doesn't go through and you miss the reminders, or your account is wrongly closed for abuse by a malfunctioning bot.
- tedivm 3y ago
  In December of 2012 AWS admitted that four files were lost: two completely deleted and two more truncated.
- vlovich123 3y ago
  S3 is designed for 11 9s of durability. What this means is that they model a "correctly functioning system" (i.e. replication and/or erasure coding) and the durability guarantee is about HW failures only. However, such models do not (& really cannot) account for the existence of bugs or introduction of new ones. That's a huge part of why S3 doesn't really do a whole lot of feature development (well that + it's hard to maintain a 20 yold codebase). Also, we're talking about Google Drive here which isn't GCS (Google's S3 competitor) but a higher-level product layered on top of GCS but with it's own book keeping / ACLs etc. Thus there's more room for error. My hunch is that the data is permanently lost. Additionally, S3 stores an enormous amount of data such that probabilistically they're bound to lose something to HW failure. 2 years ago, S3 stored 100 trillion objects [1]. With 11 9s of durability annually you'd expect to lose 100 objects a year. The saving grace is that most objects aren't accessed (maybe not ever again) & they detect & correct durability errors on access to ensure that accessed objects definitely aren't lost. So while they "admit" to 4 objects, that's likely an under count because I wouldn't expect them to regularly check if all 100 trillion objects are accessible because of how long that would take. [1] https://www.zdnet.com/article/aws-s3-storage-now-holds-over-100-trillion-objects/ https://www.zdnet.com/article/aws-s3-storage-now-holds-over-...

by pram 3y ago

stefandesu 3y ago

My concern about my own backup strategy (which, for Photos, is exactly the same as yours) is that I don't have a system in place that notifies me if files get corrupted or randomly deleted, and I'm not sure how I would go about implementing something like that. Any ideas?

pram 3y ago
I use a synology for TM (raid 1) and it uses btrfs which has checksums to prevent bitrot. Either that or something running ZFS would be the obvious options: https://kb.synology.com/en-us/DSM/tutorial/How_to_enable_file_self_healing_on_DSM https://kb.synology.com/en-us/DSM/tutorial/How_to_enable_fil...

sitzkrieg 3y ago

s3 losing a file is a statistical anomaly and would get a lot more than a blog post and support deflection

nicoburns 3y ago

Has S3 ever lost a file yet? I know that the reliability isn't 100% as that isn't possible in the real world. But I've never heard a report of a file that was confirmed as stored being lost, and I was under the impression that this was in the category "bound to happen eventually, but hasn't happened yet".

michaelt 3y ago
Probably not in the way you're thinking. But with cloud systems you can always lose your files when a hacker gets access to your root account and holds it hostage, or you lose the phone with the 2fa secret on, or your payment doesn't go through and you miss the reminders, or your account is wrongly closed for abuse by a malfunctioning bot.
tedivm 3y ago
In December of 2012 AWS admitted that four files were lost: two completely deleted and two more truncated.
vlovich123 3y ago
S3 is designed for 11 9s of durability. What this means is that they model a "correctly functioning system" (i.e. replication and/or erasure coding) and the durability guarantee is about HW failures only. However, such models do not (& really cannot) account for the existence of bugs or introduction of new ones. That's a huge part of why S3 doesn't really do a whole lot of feature development (well that + it's hard to maintain a 20 yold codebase). Also, we're talking about Google Drive here which isn't GCS (Google's S3 competitor) but a higher-level product layered on top of GCS but with it's own book keeping / ACLs etc. Thus there's more room for error. My hunch is that the data is permanently lost. Additionally, S3 stores an enormous amount of data such that probabilistically they're bound to lose something to HW failure. 2 years ago, S3 stored 100 trillion objects [1]. With 11 9s of durability annually you'd expect to lose 100 objects a year. The saving grace is that most objects aren't accessed (maybe not ever again) & they detect & correct durability errors on access to ensure that accessed objects definitely aren't lost. So while they "admit" to 4 objects, that's likely an under count because I wouldn't expect them to regularly check if all 100 trillion objects are accessible because of how long that would take. [1] https://www.zdnet.com/article/aws-s3-storage-now-holds-over-100-trillion-objects/ https://www.zdnet.com/article/aws-s3-storage-now-holds-over-...