4 ms·

That error means TM detected bit rot via failing checksums or due to an IO error during a read operation, usually because of a failing disk, or other software o

That error means TM detected bit rot via failing checksums or due to an IO error during a read operation, usually because of a failing disk, or other software on the NAS you're using corrupting your backups. It's not caused by TM itself corrupting its own backups.

scottlamb 6y ago
Hard to completely disprove but doesn't seem terribly likely. Name-brand NAS (Synology) with stock up-to-date firmware and nothing weird running on it, RAID-1, WD Red (pre-SMR) drives, happy SMART output. Maybe the most sketchy thing is that I have an ancient SSD stuck in there as cache. I could try removing it, though its smart output also looks good. Next sketchiest is that it's using btrfs, but other people tell me this is trustworthy. /shruggie The error is rare enough that I'll never really know it has gone away permanently or what fixed it if it has, because I'm never going to go for say a year without changing more than one variable. For example, I'm not going to stop doing macOS or Synology updates for that long.
- jmnicolas 6y ago
  > happy SMART output I wouldn't put too much trust on SMART[0]: since yesterday I have a disk in a raid 5 array making loud clicking noises, mdadm says everything is fine and all disks report perfect SMART data. [0] I trust SMART when it says it's not OK though
  - zaarn 6y ago
    SMART tells you what the disk knows. Clicking in HDDs is usually caused by the head not moving fluidly over the platter. When the firmware detects this, it tries to reset the head, usually by parking it for a moment and then unparking it. Most of the time this solves the issue. This inherently doesn't cause data loss or corruption, just high latency when using the disk. It can lead to an increased number of high fly writes as the head is unparked and is usually a bit too high for a bit until it finds it's air cushion again. The usual suspects in your smart data are "Recalibration Retries", "Seek Error Rate", "Head Stability" (if WD), "High-Fly Writes", "GMR Head Amplitude" and "Head Flying Hours". All of these generally don't cause runtime issues with the drive, but a clicking sound or an increase in these numbers means the mechanical assembly of the HDD, while still in spec and good enough to operate the HDD almost normaly, is degraded. Of course, your head may crash down on the platter the next recelibration try, the clicking causes a lot of head movement and parking, so it also causes a lot of wear. Drives which are clicking age a lot faster and in reverse, clicking is an easy indicator the drive is beginning the end of it's useful lifecycle.
    - jmnicolas 6y ago
      Thanks for the detailed answer. I managed to identify the faulty disk, the array has been rebuilt last night. The SMART data changed, I think it was the seek error rate that went through the roof.
- masklinn 6y ago
  > happy SMART output SMART will tell you that your disk is failing, not that it’s not failing. Unhappy smart = unhappy disk but happy smart /= happy disk.
  - scottlamb 6y ago
    As I said, hard to disprove but seems unlikely. Backblaze says 76.7% chance these SMART attributes will be bad on a failed disk. [1] More importantly, I said I'm using RAID-1, so a single bad disk shouldn't be a problem. Another poster suggested this happens with AFP, which would likely mean a macOS defect. I'm using AFP. I think I'll switch to SMB... [1] https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/ https://www.backblaze.com/blog/what-smart-stats-indicate-har...
kalleboo 6y ago
The number one cause of that error seems to be using AFP for backups. Had the same issue on a Apple Time Capsule and a Synology NAS. Went away on the Synology once I disabled AFP in favor of SMB.
- sbuk 6y ago
  Wasn't AFP deprecated in Mavericks? I'm fairly certain that it hasn't had an update for some time. At this stage, it's certainly advisable not using it for critical applications.
  - kalleboo 6y ago
    It probably is (and e.g. you can't do AFP shares of APFS drives), but Apple's SMB implementation is fairly bad, and sometimes the only workaround is to connect via AFP instead (sometimes I've also had much better performance over AFP)
    - azinman2 6y ago
      Isn’t it just samba?
      - formerly_proven 6y ago
        Samba is GPL so it is certainly not used by Apple.
      - kalleboo 6y ago
        It used to be, but Samba relicensed to GPLv3, so Apple switched to a custom SMB implementation
- scottlamb 6y ago
  I'm using AFP. I'll switch. Thanks for the idea!
nonninz 6y ago
There is a long standing bug that AFAIK it was never fixed, that it corrupts the backups if you close/disconnect the laptop while it's doing a backup it will leave the disk in a corrupted state. It's not the NAS fault usually. I've switched to an external HD always plugged on my "docking station" as Time Machine backup for this reason. Although it's annoying to remember to unmount the HD (and check if a Time Machine backup is in progress) before disconnecting the laptop :/

scottlamb 6y ago

Hard to completely disprove but doesn't seem terribly likely. Name-brand NAS (Synology) with stock up-to-date firmware and nothing weird running on it, RAID-1, WD Red (pre-SMR) drives, happy SMART output. Maybe the most sketchy thing is that I have an ancient SSD stuck in there as cache. I could try removing it, though its smart output also looks good. Next sketchiest is that it's using btrfs, but other people tell me this is trustworthy. /shruggie The error is rare enough that I'll never really know it has gone away permanently or what fixed it if it has, because I'm never going to go for say a year without changing more than one variable. For example, I'm not going to stop doing macOS or Synology updates for that long.

jmnicolas 6y ago
> happy SMART output I wouldn't put too much trust on SMART[0]: since yesterday I have a disk in a raid 5 array making loud clicking noises, mdadm says everything is fine and all disks report perfect SMART data. [0] I trust SMART when it says it's not OK though
- zaarn 6y ago
  SMART tells you what the disk knows. Clicking in HDDs is usually caused by the head not moving fluidly over the platter. When the firmware detects this, it tries to reset the head, usually by parking it for a moment and then unparking it. Most of the time this solves the issue. This inherently doesn't cause data loss or corruption, just high latency when using the disk. It can lead to an increased number of high fly writes as the head is unparked and is usually a bit too high for a bit until it finds it's air cushion again. The usual suspects in your smart data are "Recalibration Retries", "Seek Error Rate", "Head Stability" (if WD), "High-Fly Writes", "GMR Head Amplitude" and "Head Flying Hours". All of these generally don't cause runtime issues with the drive, but a clicking sound or an increase in these numbers means the mechanical assembly of the HDD, while still in spec and good enough to operate the HDD almost normaly, is degraded. Of course, your head may crash down on the platter the next recelibration try, the clicking causes a lot of head movement and parking, so it also causes a lot of wear. Drives which are clicking age a lot faster and in reverse, clicking is an easy indicator the drive is beginning the end of it's useful lifecycle.
  - jmnicolas 6y ago
    Thanks for the detailed answer. I managed to identify the faulty disk, the array has been rebuilt last night. The SMART data changed, I think it was the seek error rate that went through the roof.
masklinn 6y ago
> happy SMART output SMART will tell you that your disk is failing, not that it’s not failing. Unhappy smart = unhappy disk but happy smart /= happy disk.
- scottlamb 6y ago
  As I said, hard to disprove but seems unlikely. Backblaze says 76.7% chance these SMART attributes will be bad on a failed disk. [1] More importantly, I said I'm using RAID-1, so a single bad disk shouldn't be a problem. Another poster suggested this happens with AFP, which would likely mean a macOS defect. I'm using AFP. I think I'll switch to SMB... [1] https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/ https://www.backblaze.com/blog/what-smart-stats-indicate-har...

kalleboo 6y ago

The number one cause of that error seems to be using AFP for backups. Had the same issue on a Apple Time Capsule and a Synology NAS. Went away on the Synology once I disabled AFP in favor of SMB.

sbuk 6y ago
Wasn't AFP deprecated in Mavericks? I'm fairly certain that it hasn't had an update for some time. At this stage, it's certainly advisable not using it for critical applications.
- kalleboo 6y ago
  It probably is (and e.g. you can't do AFP shares of APFS drives), but Apple's SMB implementation is fairly bad, and sometimes the only workaround is to connect via AFP instead (sometimes I've also had much better performance over AFP)
  - azinman2 6y ago
    Isn’t it just samba?
    - formerly_proven 6y ago
      Samba is GPL so it is certainly not used by Apple.
    - kalleboo 6y ago
      It used to be, but Samba relicensed to GPLv3, so Apple switched to a custom SMB implementation
scottlamb 6y ago
I'm using AFP. I'll switch. Thanks for the idea!

nonninz 6y ago

There is a long standing bug that AFAIK it was never fixed, that it corrupts the backups if you close/disconnect the laptop while it's doing a backup it will leave the disk in a corrupted state. It's not the NAS fault usually. I've switched to an external HD always plugged on my "docking station" as Time Machine backup for this reason. Although it's annoying to remember to unmount the HD (and check if a Time Machine backup is in progress) before disconnecting the laptop :/