9 ms·

More information about "safety freez" from an backblaze engineer: https://old.reddit.com/r/backblaze/comments/hvcbpw/safety_freeze/ https://old.reddit.com/r/bac

More information about "safety freez" from an backblaze engineer: https://old.reddit.com/r/backblaze/comments/hvcbpw/safety_freeze/ https://old.reddit.com/r/backblaze/comments/hvcbpw/safety_fr...

cloogshicer 5y ago
Thanks! I already found that, unfortunately, they also just say: Do a full restore via shipped hard drive (expensive for non-USA).
CamperBob2 5y ago
"The log files that list what Backblaze has backed up are called "bz_done" files. They list 'what has been done'. Here is where they are located on your computer: ... WARNING: don't edit those files -> you are guaranteed to corrupt your backup. You'll lose everything." Why does the integrity of the backup rely on files stored on the computer being backed up? This seems so stupid that I'm sure I'm missing a clue.
- cloogshicer 5y ago
  Yup. Sounds pretty terrifying.
  - CamperBob2 5y ago
    That can't possibly be how it works. No way.
    - cloogshicer 5y ago
      Well, their suggested fix basically says that I should "unlink" my computer from the online backup, and relink it (by re-installing the Backblaze app). But before I do that, I should download any missing files, since they will be deleted from the backup upon re-link (if they can't be found on my computer). But I can't do that without knowing which files are missing.
- gruez 5y ago
  >Why does the integrity of the backup rely on files stored on the computer being backed up? This seems so stupid that I'm sure I'm missing a clue. Reading the explanation in the reddit thread, that's not the impression I got at all. 1. If your computer exploded, your backup integrity would not be compromised 2. If gremlins in your computer did mess with the file, your backups could be compromised. That sounds bad, until you realize that gremlins in your computer could also compromise the executable to do other things that could compromise the backup, (eg. telling the server to delete existing backup data because the retention period has passed or whatever, or simply uploading bad data and waiting for the retention period of 30 days to pass). Moral of the story: if the computer doing the backup can't be trusted to operate correctly, all bets are off.
- jfrunyon 5y ago
  Because otherwise a trashed computer could backup corrupted data over the good backup.
  - lowbloodsugar 5y ago
    It's not a backup then. It's a synced copy.
    - gruez 5y ago
      It's a backup with a retention policy. In backblaze's case, it's 30 days. I'm fairly certain that still counts as a backup. Do you think that only immutable, append-only storage counts as "backup"?
    - KingMachiavelli 5y ago
      That's really what the personal 'backups' are, it's just a synced copy that will happily mirror corrupted data and delete files that were deleted (not sure if there's any delay). If you want real backups then use the B2 storage directly.
jfrunyon 5y ago
> What was the bug? It was a threading bug. lol
jiggawatts 5y ago
Wow, that engineer just... "doesn't get it". He doesn't understand the fundamentals of the problem he's solving, and is actively writing code that basically puts tools down and starts shouting "EVERYONE STOP!" in response to expected scenarios. Most filesystems provide no guarantees at all by default on writes. NTFS journals metadata writes, but not data writes. Append-only files are absolutely expected to be truncated. The application must deal with this either by being insensitive to rollback, or by explicitly requesting a write cache flush! This is very well known to anyone that has ever written any kind of write-ahead-log, database engine, etc... There's a bazillion articles about how this is intended behaviour and no amount of screaming and shouting will make it go away. Learn the storage API guarantees, or STOP writing code that has critical dependencies on storage! This quote especially seemed childish and immature to me: > "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it" No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested, consumer HDD and SSD drives are well known to ignore this and keep things in volatile RAM cache to boost their IOPS numbers at the expense of durability. Only "enterprise" drives honour this API properly, and even then there are corner-cases around things like 512/4K sectors and torn writes. Similarly, consumer drives don't have power protection, so partial sector writes are entirely possible and should be expected if they lose power mid-write, or just crash at an inopportune time. To be more constructive: The correct thing to do is that a log writer must always end log writes with a checksum of some sort. If the log is truncated (for any reason!), then it must recover starting from the last-known-good checksum. Throwing your hands up and saying "NO MORE BACKUPS FOR YOU! EVER!" is not the right response. Retrying backups from the last-known-good point automatically is the correct response. PS: Some of his other rants are also a lack of understanding of thread safety and/or the lack of ECC RAM in consumer-grade PCs causing random misbehaviour. Heck, you'd also have to deal with bad sectors, corrupt filesystems, and a bunch of other corner cases that just makes this guy scream and shout instead of writing robust code... PPS: I just realised that the reddit post is by 'CTO and Founder of Backblaze'. Wow. Note to self, do not use Blackbaze for anything, ever. If the CTO is this clueless, then their products can't be trusted.
- brianwski 5y ago

More information about "safety freez" from an backblaze engineer: https://old.reddit.com/r/backblaze/comments/hvcbpw/safety_freeze/ https://old.reddit.com/r/backblaze/comments/hvcbpw/safety_fr...

cloogshicer 5y ago

Thanks! I already found that, unfortunately, they also just say: Do a full restore via shipped hard drive (expensive for non-USA).

CamperBob2 5y ago

"The log files that list what Backblaze has backed up are called "bz_done" files. They list 'what has been done'. Here is where they are located on your computer: ... WARNING: don't edit those files -> you are guaranteed to corrupt your backup. You'll lose everything." Why does the integrity of the backup rely on files stored on the computer being backed up? This seems so stupid that I'm sure I'm missing a clue.

cloogshicer 5y ago
Yup. Sounds pretty terrifying.
- CamperBob2 5y ago
  That can't possibly be how it works. No way.
  - cloogshicer 5y ago
    Well, their suggested fix basically says that I should "unlink" my computer from the online backup, and relink it (by re-installing the Backblaze app). But before I do that, I should download any missing files, since they will be deleted from the backup upon re-link (if they can't be found on my computer). But I can't do that without knowing which files are missing.
gruez 5y ago
>Why does the integrity of the backup rely on files stored on the computer being backed up? This seems so stupid that I'm sure I'm missing a clue. Reading the explanation in the reddit thread, that's not the impression I got at all. 1. If your computer exploded, your backup integrity would not be compromised 2. If gremlins in your computer did mess with the file, your backups could be compromised. That sounds bad, until you realize that gremlins in your computer could also compromise the executable to do other things that could compromise the backup, (eg. telling the server to delete existing backup data because the retention period has passed or whatever, or simply uploading bad data and waiting for the retention period of 30 days to pass). Moral of the story: if the computer doing the backup can't be trusted to operate correctly, all bets are off.
jfrunyon 5y ago
Because otherwise a trashed computer could backup corrupted data over the good backup.
- lowbloodsugar 5y ago
  It's not a backup then. It's a synced copy.
  - gruez 5y ago
    It's a backup with a retention policy. In backblaze's case, it's 30 days. I'm fairly certain that still counts as a backup. Do you think that only immutable, append-only storage counts as "backup"?
  - KingMachiavelli 5y ago
    That's really what the personal 'backups' are, it's just a synced copy that will happily mirror corrupted data and delete files that were deleted (not sure if there's any delay). If you want real backups then use the B2 storage directly.

jfrunyon 5y ago

> What was the bug? It was a threading bug. lol

jiggawatts 5y ago

Wow, that engineer just... "doesn't get it". He doesn't understand the fundamentals of the problem he's solving, and is actively writing code that basically puts tools down and starts shouting "EVERYONE STOP!" in response to expected scenarios. Most filesystems provide no guarantees at all by default on writes. NTFS journals metadata writes, but not data writes. Append-only files are absolutely expected to be truncated. The application must deal with this either by being insensitive to rollback, or by explicitly requesting a write cache flush! This is very well known to anyone that has ever written any kind of write-ahead-log, database engine, etc... There's a bazillion articles about how this is intended behaviour and no amount of screaming and shouting will make it go away. Learn the storage API guarantees, or STOP writing code that has critical dependencies on storage! This quote especially seemed childish and immature to me: > "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it" No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested, consumer HDD and SSD drives are well known to ignore this and keep things in volatile RAM cache to boost their IOPS numbers at the expense of durability. Only "enterprise" drives honour this API properly, and even then there are corner-cases around things like 512/4K sectors and torn writes. Similarly, consumer drives don't have power protection, so partial sector writes are entirely possible and should be expected if they lose power mid-write, or just crash at an inopportune time. To be more constructive: The correct thing to do is that a log writer must always end log writes with a checksum of some sort. If the log is truncated (for any reason!), then it must recover starting from the last-known-good checksum. Throwing your hands up and saying "NO MORE BACKUPS FOR YOU! EVER!" is not the right response. Retrying backups from the last-known-good point automatically is the correct response. PS: Some of his other rants are also a lack of understanding of thread safety and/or the lack of ECC RAM in consumer-grade PCs causing random misbehaviour. Heck, you'd also have to deal with bad sectors, corrupt filesystems, and a bunch of other corner cases that just makes this guy scream and shout instead of writing robust code... PPS: I just realised that the reddit post is by 'CTO and Founder of Backblaze'. Wow. Note to self, do not use Blackbaze for anything, ever. If the CTO is this clueless, then their products can't be trusted.

brianwski 5y ago