9 ms·

APFS doesn't do proper checksumming on data, and won't mirror or RAID, so it doesn't have the data recovery properties implied by the article. http://dtrace.or

APFS doesn't do proper checksumming on data, and won't mirror or RAID, so it doesn't have the data recovery properties implied by the article. http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/

dilap 9y ago
Well, that's disappointing to hear. I've run into multiple MP3s bit-rotting on my MB air over several years. Perhaps soley the fault of HFS+, but trusting the underlying hardware just doesn't seem like a good idea.
- itsoggy 9y ago
  Yep, I had a similar thing with audio files corrupting on my old MacBook, I was using time machine to backup and guess what.... the copy in the backup was fried too! Switched back to vinyl for my music, my 20 year old Texhnics 1210 will probably out last my current laptop and the next!
  - roryisok 9y ago
    I smell a hipster.
  - amdavidson 9y ago
    You switched to a playback means that is inherently destructive to the media as a means to protect against data loss?
  - DugFin 9y ago
    The physical vinyl won't outlast your digital, it'll just fail more gracefully, slowly losing fidelity every time you scrape it with a needle. You're just replacing a small chance of catastrophic failure with a guaranteed gradual failure.
    - semi-extrinsic 9y ago
      Wikipedia says "Record wear can be reduced to virtual insignificance, however, by the use of a high-quality, correctly adjusted turntable and tonearm, a high-compliance magnetic cartridge with a high-end stylus in good condition, and careful record handling, with non-abrasive removal of dust before playing and other cleaning if necessary." In other words, record wear is negligible if you buy good equipment and take good care of it. Just like the risk of bit rot, catastrophic crashes etc. is negligible if you buy good storage equipment and take good care of your data (good backups etc.) Perhaps the one big advantage of vinyl over digital is that the shelf life of unused vinyl is larger than a human lifetime. A disk stored on a shelf, however, can suffer damage in many ways and is guaranteed to be very hard to connect to your computer in ten-twenty years.
    - itsoggy 9y ago
      With a little calibration of the TT once in a while it will esily outlast me!
- Symbiote 9y ago
  I add parity archives to add some redundancy to my photographs. I've done it for years, but I think it's useful not to rely on a filesystem to handle this. For example: par2create -r5 -n2 example.par2 *.jpg creates two files, between them giving 5% redundancy. I think that should be more than enough to repair bitrot within a file, but depending how many photos there are, losing a whole file could be more than 5%. par2verify example.par2 will verify, and par2repair will repair corrupted or missing files. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
  - indolering 9y ago
    It would be neat if there was a parity scheme fast enough to preserve 2% of of all files on disk. It could even be tucked away behind savings from file-level compression.
  - mfukar 9y ago
    > it's useful not to rely on a filesystem to handle this In what way?
    - wtallis 9y ago
      Relatively few filesystems offer thorough data checksumming. Hardly any offer erasure coding. RAID at the filesystem layer is a bit more common, but also more inconvenient. Doing erasure coding at the file archive level rather than in the filesystem gives you the freedom to move your archives onto standard everyday filesystems and devices without silently losing the protection.
      - jen20 9y ago
        You don't need _a lot_ of filesystems to offer it - just the one you use - and if this is important to you, just use ZFS.
        rovr138 9y ago
        What if I use multiple computers and multiple operating systems want to be able to work on this data on them. Par2 would allow me to create the needed data on any computer and test it on any computer. The other alternative is having a server that's up and running all the time, exposed to the internet (or complicate the setup with a VPN), so I can sync. Operations would take a long time (via the internet) or I would anyway need to transfer the data to my computer, work on it, sync it back. During this time, any protections that ZFS offers are null since anything could happen in my computer and I can't test for it locally. ZFS is great. But it's not the answer to everything.
- beagle3 9y ago
  Damaged files (that can still be read from disk without error) are more likely than not a result of bad memory. It is incredibly unlikely to have bits mutated on disk without triggering a CRC error -- approx 1 in 4 billion errors would go undetected in a CRC-32; and it has to be spread over more than 32-bits-wide. The fact that you had multiple damaged files basically guarantees that it was a faulty memory (or bus, or controller -- but not actual magnetic media corruption) issue. ZFS can't really help you with that as the data was likely damaged in transit rather than on disk; though with its own 256 bit hash, it is likely to detect those faulty system components earlier than later.
  - pstadler 9y ago
    To mitigate the likeliness of damage during transit, it's recommened to use ECC memory with ZFS. I had FreeNAS running on a system with a faulty memory module. ZFS correctly discovered damage in random files I just transferred over. Luckily, it wasn't too late to replace the bad module and repeat the transfer. Lessons learned: - Run MemTest86 on hardware before using it as storage - Use an FS that does checksums Unfortunately, the latter doesn't seem to apply to APFS.
    - voidz 9y ago
      Can you always just buy ECC memory (I have DDR3, I guess it's more affordable now since it's a little bit older), or does something else (cpu, motherboard) need to support it as well?
      - pstadler 9y ago
        You need a compatible motherboard and CPU.
        tw04 9y ago
        Which is why Intel kinda sucks. It would cost them basically nothing to have ECC enabled on all of their hardware, but they insist on using it as a differentiator between server and desktop parts. AMD (at least in the past), includes it on all parts so that it's up to the consumer to choose.
hyperknot 9y ago
Isn't it possible that Apple developed this filesystem primarily to be used with their SSD based devices which are using their custom controllers, which simply takes care of this low level protection? SSD controllers are said to be very complicated piece of proprietary technologies, and I believe they are using sophisticated algorithms for integrity protection.
- vetinari 9y ago
  Apple users use external (USB, Thunderbolt) drives too, where they would not get this special hypothetical protection.
  - justinjlynn 9y ago
    Clearly telegraphing Apple's intent to release new "Apple approved (and licensed)" storage "accessories" for their systems.
    - toyg 9y ago
      Lock-in all the things!
      - justinjlynn 9y ago
        Indeed, sure would be a shame for anything to happen to your data because you chose not to use drives equipped with iStoreSafe(tm) technology by Apple(tm).
        kevin_thibedeau 9y ago
        Brings back the days when Mac users were paying a 3x markup for standard SCSI drives.
        deleted 9y ago
        [deleted]

dilap 9y ago

Well, that's disappointing to hear. I've run into multiple MP3s bit-rotting on my MB air over several years. Perhaps soley the fault of HFS+, but trusting the underlying hardware just doesn't seem like a good idea.

itsoggy 9y ago
Yep, I had a similar thing with audio files corrupting on my old MacBook, I was using time machine to backup and guess what.... the copy in the backup was fried too! Switched back to vinyl for my music, my 20 year old Texhnics 1210 will probably out last my current laptop and the next!
- roryisok 9y ago
  I smell a hipster.
- amdavidson 9y ago
  You switched to a playback means that is inherently destructive to the media as a means to protect against data loss?
- DugFin 9y ago
  The physical vinyl won't outlast your digital, it'll just fail more gracefully, slowly losing fidelity every time you scrape it with a needle. You're just replacing a small chance of catastrophic failure with a guaranteed gradual failure.
  - semi-extrinsic 9y ago
    Wikipedia says "Record wear can be reduced to virtual insignificance, however, by the use of a high-quality, correctly adjusted turntable and tonearm, a high-compliance magnetic cartridge with a high-end stylus in good condition, and careful record handling, with non-abrasive removal of dust before playing and other cleaning if necessary." In other words, record wear is negligible if you buy good equipment and take good care of it. Just like the risk of bit rot, catastrophic crashes etc. is negligible if you buy good storage equipment and take good care of your data (good backups etc.) Perhaps the one big advantage of vinyl over digital is that the shelf life of unused vinyl is larger than a human lifetime. A disk stored on a shelf, however, can suffer damage in many ways and is guaranteed to be very hard to connect to your computer in ten-twenty years.
  - itsoggy 9y ago
    With a little calibration of the TT once in a while it will esily outlast me!
Symbiote 9y ago
I add parity archives to add some redundancy to my photographs. I've done it for years, but I think it's useful not to rely on a filesystem to handle this. For example: par2create -r5 -n2 example.par2 *.jpg creates two files, between them giving 5% redundancy. I think that should be more than enough to repair bitrot within a file, but depending how many photos there are, losing a whole file could be more than 5%. par2verify example.par2 will verify, and par2repair will repair corrupted or missing files. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- indolering 9y ago
  It would be neat if there was a parity scheme fast enough to preserve 2% of of all files on disk. It could even be tucked away behind savings from file-level compression.
- mfukar 9y ago
  > it's useful not to rely on a filesystem to handle this In what way?
  - wtallis 9y ago
    Relatively few filesystems offer thorough data checksumming. Hardly any offer erasure coding. RAID at the filesystem layer is a bit more common, but also more inconvenient. Doing erasure coding at the file archive level rather than in the filesystem gives you the freedom to move your archives onto standard everyday filesystems and devices without silently losing the protection.
    - jen20 9y ago
      You don't need _a lot_ of filesystems to offer it - just the one you use - and if this is important to you, just use ZFS.
      - rovr138 9y ago
        What if I use multiple computers and multiple operating systems want to be able to work on this data on them. Par2 would allow me to create the needed data on any computer and test it on any computer. The other alternative is having a server that's up and running all the time, exposed to the internet (or complicate the setup with a VPN), so I can sync. Operations would take a long time (via the internet) or I would anyway need to transfer the data to my computer, work on it, sync it back. During this time, any protections that ZFS offers are null since anything could happen in my computer and I can't test for it locally. ZFS is great. But it's not the answer to everything.
beagle3 9y ago
Damaged files (that can still be read from disk without error) are more likely than not a result of bad memory. It is incredibly unlikely to have bits mutated on disk without triggering a CRC error -- approx 1 in 4 billion errors would go undetected in a CRC-32; and it has to be spread over more than 32-bits-wide. The fact that you had multiple damaged files basically guarantees that it was a faulty memory (or bus, or controller -- but not actual magnetic media corruption) issue. ZFS can't really help you with that as the data was likely damaged in transit rather than on disk; though with its own 256 bit hash, it is likely to detect those faulty system components earlier than later.
- pstadler 9y ago
  To mitigate the likeliness of damage during transit, it's recommened to use ECC memory with ZFS. I had FreeNAS running on a system with a faulty memory module. ZFS correctly discovered damage in random files I just transferred over. Luckily, it wasn't too late to replace the bad module and repeat the transfer. Lessons learned: - Run MemTest86 on hardware before using it as storage - Use an FS that does checksums Unfortunately, the latter doesn't seem to apply to APFS.
  - voidz 9y ago
    Can you always just buy ECC memory (I have DDR3, I guess it's more affordable now since it's a little bit older), or does something else (cpu, motherboard) need to support it as well?
    - pstadler 9y ago
      You need a compatible motherboard and CPU.
      - tw04 9y ago
        Which is why Intel kinda sucks. It would cost them basically nothing to have ECC enabled on all of their hardware, but they insist on using it as a differentiator between server and desktop parts. AMD (at least in the past), includes it on all parts so that it's up to the consumer to choose.

hyperknot 9y ago

Isn't it possible that Apple developed this filesystem primarily to be used with their SSD based devices which are using their custom controllers, which simply takes care of this low level protection? SSD controllers are said to be very complicated piece of proprietary technologies, and I believe they are using sophisticated algorithms for integrity protection.

vetinari 9y ago
Apple users use external (USB, Thunderbolt) drives too, where they would not get this special hypothetical protection.
- justinjlynn 9y ago
  Clearly telegraphing Apple's intent to release new "Apple approved (and licensed)" storage "accessories" for their systems.
  - toyg 9y ago
    Lock-in all the things!
    - justinjlynn 9y ago
      Indeed, sure would be a shame for anything to happen to your data because you chose not to use drives equipped with iStoreSafe(tm) technology by Apple(tm).
      - kevin_thibedeau 9y ago
        Brings back the days when Mac users were paying a 3x markup for standard SCSI drives.
        deleted 9y ago
        [deleted]