5 ms·

> If I read the article correctly, Time Machine's switching to something similar to overlay FS, but nested? I really hope they get it right: backup is where my

> If I read the article correctly, Time Machine's switching to something similar to overlay FS, but nested? I really hope they get it right: backup is where my tolerance for bugs is really low. ... I've been generally happy with Time Machine's stability, but this is getting me a little worried. I agree backups are important to get right. I'm not as confident as you are in Time Machine today. I've gotten this message a few times: > Time Machine completed a verification of your backups on "nas.<mydomain>". To improve reliability, Time Machine must create a new backup for you. which I'd paraphrase as "oops, we corrupted your backup. try again maybe?"

RJIb8RBYxzAMX9u 6y ago
Yeah, my "solution" to that problem is to backup to a zfs NAS, and through shitty^Wclever scripting, create a snapshot after each successful backup. If TM backup fails for any reason, I roll back on the NAS end and start again.
- pfranz 6y ago
  I have a nightly snapshot and do the same if there's an issue. To be fair, I've only ever hit these "verification" errors when I manually mounted and mucked around with the backup and a backup started at the same time--all over the network. I've never had this issue when using an external hard drive connected via USB and I've never had it happen when I wasn't mucking with it. For awhile I was able to "fix" these errors by running diskutil and resetting the state (my Time Machine server was a flaky Raspberry Pi at the time). All in all, it is concerning when you "lose" your backup no matter the circumstances.
  - scottlamb 6y ago
    In my case, the NAS is a Synology server. I don't know how to determine authoritatively where the fault lies, but I trust it more than I trust Time Machine. (Although I am using btrfs on that volume so maybe it's a wash.) I'm sure I've disconnected my Mac from my Thunderbolt dock (and thus the active NIC) and/or put my Mac to sleep mid-backup many times, though. This is something that I would like it to be robust to but may be "the problem". Interesting idea to reset the state. I think I can take snapshots with Synology. I'm unsure how to manage the details though: * I let Time Machine starts backups automatically whenever it pleases. Does it have a hook to run a command to take a snapshot? * Does it do the verification every time, or (say) 1 in 10? If the latter, going back to the most recent snapshot isn't enough. And if you do say 3 backups then get rid of #3, can it take incremental #4 on top of #2, or will it complain that the #3 state it expects is absent? I really should just set up a different backup system. I do at least have Google Drive sync on most everything important. It's not a "real" backup as another poster pointed out, but it's better than nothing.
    - RJIb8RBYxzAMX9u 6y ago
      > I let Time Machine starts backups automatically whenever it pleases. Does it have a hook to run a command to take a snapshot? I don't know, but you can trigger backups on demand using tmutil(8), and that's what I do. > And if you do say 3 backups then get rid of #3, can it take incremental #4 on top of #2, or will it complain that the #3 state it expects is absent? You can delete arbitrary backup(s). "Classic" Time Machine backups are snapshots deduped using hard links. HFS+ even supports directory hard links.
    - pfranz 6y ago
      > In my case, the NAS is a Synology server. Same here. I'm also using BTRFS (I've not had a problem and my understanding is Synology doesn't use the flakier features of BTRFS). I can't remember if I'm using AFP or SMB (SMB I think is the only one Apple supports), but my backup is encrypted and I have a specific user. I'm not sure BTRFS matters all that much because the backup is a disk image. When I've had them get corrupted I've run disk utility on the disk image to fix things--there's a lot of layers to unwind. I hear stories of people having issues with similar setups, but I haven't (I know that's a sucky answer to hear). Years and years ago I felt like I would get a corrupt backup when I closed my lid during a network backup (I believe this was pre-APFS), but I haven't noticed an issue in years. I have it auto-backup and it usually does it at night when the lid is closed and it's charging. In Snapshot Replication I have a daily snapshot created at midnight with a 2 snapshot retention. I tend not to run out of space, so I'm fine with any deletes taking a few days to roll off. "Recover" under "Recovery" can restore old snapshots easily. I do this for most all of my Shared Folders.
    - pfranz 6y ago
      > * Does it do the verification every time, or 1 in 10? If the latter, going back to the most recent snapshot isn't enough. And if you do say 3 backups then get rid of #3, can it take incremental #4 on top of #2, or will it complain that the #3 state it expects is absent? I don't believe it does the verification every time. I can't remember what the criteria is (I assumed it was similar to fsck on boot for Linux). You can run verify manually (Option-click the menu bar) and I'm sure there's a command line--so maybe you want to schedule your backups explicitly and verify every time? I've always had a corrupt backup when I've manually mounted the disk image and mucked around, so it's easy to restore within two days. I thought I might have issues with the client/server state mismatch, but haven't in practice (I've only needed to two this a few times). I haven't kept close track to see if I lose any in between backups--but I restore so rarely I wouldn't miss them. > I really should just set up a different backup system. I do at least have Google Drive sync on most everything important. It's not a "real" backup as another poster pointed out, but it's better than nothing. I've kept to the 3-2-1 backup rule and kept bootable backups local and non-bootable ones remote.
nikisweeting 6y ago
That error means TM detected bit rot via failing checksums or due to an IO error during a read operation, usually because of a failing disk, or other software on the NAS you're using corrupting your backups. It's not caused by TM itself corrupting its own backups.
- scottlamb 6y ago
  Hard to completely disprove but doesn't seem terribly likely. Name-brand NAS (Synology) with stock up-to-date firmware and nothing weird running on it, RAID-1, WD Red (pre-SMR) drives, happy SMART output. Maybe the most sketchy thing is that I have an ancient SSD stuck in there as cache. I could try removing it, though its smart output also looks good. Next sketchiest is that it's using btrfs, but other people tell me this is trustworthy. /shruggie The error is rare enough that I'll never really know it has gone away permanently or what fixed it if it has, because I'm never going to go for say a year without changing more than one variable. For example, I'm not going to stop doing macOS or Synology updates for that long.
  - jmnicolas 6y ago
    > happy SMART output I wouldn't put too much trust on SMART[0]: since yesterday I have a disk in a raid 5 array making loud clicking noises, mdadm says everything is fine and all disks report perfect SMART data. [0] I trust SMART when it says it's not OK though
    - zaarn 6y ago
      SMART tells you what the disk knows. Clicking in HDDs is usually caused by the head not moving fluidly over the platter. When the firmware detects this, it tries to reset the head, usually by parking it for a moment and then unparking it. Most of the time this solves the issue. This inherently doesn't cause data loss or corruption, just high latency when using the disk. It can lead to an increased number of high fly writes as the head is unparked and is usually a bit too high for a bit until it finds it's air cushion again. The usual suspects in your smart data are "Recalibration Retries", "Seek Error Rate", "Head Stability" (if WD), "High-Fly Writes", "GMR Head Amplitude" and "Head Flying Hours". All of these generally don't cause runtime issues with the drive, but a clicking sound or an increase in these numbers means the mechanical assembly of the HDD, while still in spec and good enough to operate the HDD almost normaly, is degraded. Of course, your head may crash down on the platter the next recelibration try, the clicking causes a lot of head movement and parking, so it also causes a lot of wear. Drives which are clicking age a lot faster and in reverse, clicking is an easy indicator the drive is beginning the end of it's useful lifecycle.

RJIb8RBYxzAMX9u 6y ago

Yeah, my "solution" to that problem is to backup to a zfs NAS, and through shitty^Wclever scripting, create a snapshot after each successful backup. If TM backup fails for any reason, I roll back on the NAS end and start again.

pfranz 6y ago
I have a nightly snapshot and do the same if there's an issue. To be fair, I've only ever hit these "verification" errors when I manually mounted and mucked around with the backup and a backup started at the same time--all over the network. I've never had this issue when using an external hard drive connected via USB and I've never had it happen when I wasn't mucking with it. For awhile I was able to "fix" these errors by running diskutil and resetting the state (my Time Machine server was a flaky Raspberry Pi at the time). All in all, it is concerning when you "lose" your backup no matter the circumstances.
- scottlamb 6y ago
  In my case, the NAS is a Synology server. I don't know how to determine authoritatively where the fault lies, but I trust it more than I trust Time Machine. (Although I am using btrfs on that volume so maybe it's a wash.) I'm sure I've disconnected my Mac from my Thunderbolt dock (and thus the active NIC) and/or put my Mac to sleep mid-backup many times, though. This is something that I would like it to be robust to but may be "the problem". Interesting idea to reset the state. I think I can take snapshots with Synology. I'm unsure how to manage the details though: * I let Time Machine starts backups automatically whenever it pleases. Does it have a hook to run a command to take a snapshot? * Does it do the verification every time, or (say) 1 in 10? If the latter, going back to the most recent snapshot isn't enough. And if you do say 3 backups then get rid of #3, can it take incremental #4 on top of #2, or will it complain that the #3 state it expects is absent? I really should just set up a different backup system. I do at least have Google Drive sync on most everything important. It's not a "real" backup as another poster pointed out, but it's better than nothing.
  - RJIb8RBYxzAMX9u 6y ago
    > I let Time Machine starts backups automatically whenever it pleases. Does it have a hook to run a command to take a snapshot? I don't know, but you can trigger backups on demand using tmutil(8), and that's what I do. > And if you do say 3 backups then get rid of #3, can it take incremental #4 on top of #2, or will it complain that the #3 state it expects is absent? You can delete arbitrary backup(s). "Classic" Time Machine backups are snapshots deduped using hard links. HFS+ even supports directory hard links.
  - pfranz 6y ago
    > In my case, the NAS is a Synology server. Same here. I'm also using BTRFS (I've not had a problem and my understanding is Synology doesn't use the flakier features of BTRFS). I can't remember if I'm using AFP or SMB (SMB I think is the only one Apple supports), but my backup is encrypted and I have a specific user. I'm not sure BTRFS matters all that much because the backup is a disk image. When I've had them get corrupted I've run disk utility on the disk image to fix things--there's a lot of layers to unwind. I hear stories of people having issues with similar setups, but I haven't (I know that's a sucky answer to hear). Years and years ago I felt like I would get a corrupt backup when I closed my lid during a network backup (I believe this was pre-APFS), but I haven't noticed an issue in years. I have it auto-backup and it usually does it at night when the lid is closed and it's charging. In Snapshot Replication I have a daily snapshot created at midnight with a 2 snapshot retention. I tend not to run out of space, so I'm fine with any deletes taking a few days to roll off. "Recover" under "Recovery" can restore old snapshots easily. I do this for most all of my Shared Folders.
  - pfranz 6y ago
    > * Does it do the verification every time, or 1 in 10? If the latter, going back to the most recent snapshot isn't enough. And if you do say 3 backups then get rid of #3, can it take incremental #4 on top of #2, or will it complain that the #3 state it expects is absent? I don't believe it does the verification every time. I can't remember what the criteria is (I assumed it was similar to fsck on boot for Linux). You can run verify manually (Option-click the menu bar) and I'm sure there's a command line--so maybe you want to schedule your backups explicitly and verify every time? I've always had a corrupt backup when I've manually mounted the disk image and mucked around, so it's easy to restore within two days. I thought I might have issues with the client/server state mismatch, but haven't in practice (I've only needed to two this a few times). I haven't kept close track to see if I lose any in between backups--but I restore so rarely I wouldn't miss them. > I really should just set up a different backup system. I do at least have Google Drive sync on most everything important. It's not a "real" backup as another poster pointed out, but it's better than nothing. I've kept to the 3-2-1 backup rule and kept bootable backups local and non-bootable ones remote.

nikisweeting 6y ago

That error means TM detected bit rot via failing checksums or due to an IO error during a read operation, usually because of a failing disk, or other software on the NAS you're using corrupting your backups. It's not caused by TM itself corrupting its own backups.

scottlamb 6y ago
Hard to completely disprove but doesn't seem terribly likely. Name-brand NAS (Synology) with stock up-to-date firmware and nothing weird running on it, RAID-1, WD Red (pre-SMR) drives, happy SMART output. Maybe the most sketchy thing is that I have an ancient SSD stuck in there as cache. I could try removing it, though its smart output also looks good. Next sketchiest is that it's using btrfs, but other people tell me this is trustworthy. /shruggie The error is rare enough that I'll never really know it has gone away permanently or what fixed it if it has, because I'm never going to go for say a year without changing more than one variable. For example, I'm not going to stop doing macOS or Synology updates for that long.
- jmnicolas 6y ago
  > happy SMART output I wouldn't put too much trust on SMART[0]: since yesterday I have a disk in a raid 5 array making loud clicking noises, mdadm says everything is fine and all disks report perfect SMART data. [0] I trust SMART when it says it's not OK though
  - zaarn 6y ago
    SMART tells you what the disk knows. Clicking in HDDs is usually caused by the head not moving fluidly over the platter. When the firmware detects this, it tries to reset the head, usually by parking it for a moment and then unparking it. Most of the time this solves the issue. This inherently doesn't cause data loss or corruption, just high latency when using the disk. It can lead to an increased number of high fly writes as the head is unparked and is usually a bit too high for a bit until it finds it's air cushion again. The usual suspects in your smart data are "Recalibration Retries", "Seek Error Rate", "Head Stability" (if WD), "High-Fly Writes", "GMR Head Amplitude" and "Head Flying Hours". All of these generally don't cause runtime issues with the drive, but a clicking sound or an increase in these numbers means the mechanical assembly of the HDD, while still in spec and good enough to operate the HDD almost normaly, is degraded. Of course, your head may crash down on the platter the next recelibration try, the clicking causes a lot of head movement and parking, so it also causes a lot of wear. Drives which are clicking age a lot faster and in reverse, clicking is an easy indicator the drive is beginning the end of it's useful lifecycle.