5 ms·

Is it just me or is it absolutely insane that zfs and btrfs are the only common filesystems out there that do data checksumming? I don't want or need the extra

Is it just me or is it absolutely insane that zfs and btrfs are the only common filesystems out there that do data checksumming? I don't want or need the extra complexity of either of them in a lot of cases, but I'd sure as hell like to know if my data is corrupt...

ianlevesque 8y ago
Yeah it should be standard. Equally crazy is that APFS is brand new and didn't add it. I belive the justification was that the hardware does it on flash memory for their devices, but ZFS really exposed how unreliable hardware is.
- AlphaSite 8y ago
  I mean Apple has their own flash controllers, so they’re really the only company that owns the FS and the flash controller. So really they can depend on it.
  - skrause 8y ago
    What about external hard disks with terrible USB controllers? That's where data checksumming would be even more important and where Apple doesn't produce the hardware.
    - tobias3 8y ago
      Maybe APFS was mainly developed for iOS?
    - derefr 8y ago
      Apple doesn’t recommend using APFS for external media.
      - pvg 8y ago
        Really? Can you remember where you've seen this non-recommendation?
        unicornporn 8y ago
        Apple isn't really embracing APFS for external drives as I see it. Time Machine backups would be one of the main uses for external drives, but… > Time Machine can’t back up to an APFS-formatted disk. If you select a back up disk formatted as APFS, Time Machine offers to reformat it as HFS+.[1] [1] https://support.apple.com/guide/mac-help/types-of-disks-you-can-use-with-time-machine-mh15139/mac https://support.apple.com/guide/mac-help/types-of-disks-you-...
        pvg 8y ago
        This has more to do with the implementation details of TimeMachine which relies on features of HFS+ not available in APFS. It's not Apple telling you not to use APFS on external drives.
        4ad 8y ago
        I can't recall if I ever saw it, but by default it will use HFS+ for external drives.
    - jxi 8y ago
      Every time I unplug an external USB prematurely, the next time I plug it into my mac, it does a full filesystem check of it which takes a few mins.
  - leonroy 8y ago
    There have been at least two occasions in the past three years that I remember where Apple had to recall or issue firmware updates for SSDs due to corruption issues. It’s a big IF to depend solely on flash based error checking. I think Apple likely figured the cost of implementation and system overhead wasn’t worth it for Macs and iOS devices. I’d be very surprised if they use APFS or a non checksumning file system on their iCloud server farm though.
burtonator 8y ago
A lot of applications and databases do this themselves. Part of the problem is that the app might know what the proper checksum is and it would be nice to be able to write a document ahead of time and tell the FS that you already know the checksum. Won't save a massive amount of CPU but it would be worth it if it was easy to implement.
hrez 8y ago
There is also windows ReFS and Dragonfly's HAMMER2.
- deleted 8y ago
  [deleted]
z3t4 8y ago
There is however no point in having data checks if there are no way of recovering, so running ZFS only make sense if you have multiple disks. And don't forget to scrub your pool!
- tux3 8y ago
  Knowing that your data is corrupt is always better than not knowing. Whether or not you have a RAID you should still have backups, and you don't want to replace a good backup by a corrupt one.
  - z3t4 8y ago
    OK, so you know your HDD is corrupted, and you now have to take the site down to recover from backup, which is also old, so there will be data missing. Compared to if the FS can just fix itself. Sometimes not knowing can be a bliss though. :P
- elfchief 8y ago
  This is like saying "there's no point in ECC if it can't correct every error" ... which simply isn't true. It's still far better to know that corruption exists -- so you can know something is invalid and potentially take action -- than to have corruption silently hanging around doing corrupted data things.
  - z3t4 8y ago
    That is not a fair comparison as ECC is Error Correcting, while just check-summing is not. If you however have a mirror or raid-Z the file system will be able to correct.
- mrighele 8y ago
  You can set the number of copies to keep at filesystem level. This is in addition to any redundancy provided at pool level (if any). It won't save you from a failed disk, but it will be able to transparently correct any checksum error.
- 8y ago
equalunique 8y ago
BcacheFS has it & is under active development: https://bcachefs.org/ https://bcachefs.org/

ianlevesque 8y ago

Yeah it should be standard. Equally crazy is that APFS is brand new and didn't add it. I belive the justification was that the hardware does it on flash memory for their devices, but ZFS really exposed how unreliable hardware is.

AlphaSite 8y ago
I mean Apple has their own flash controllers, so they’re really the only company that owns the FS and the flash controller. So really they can depend on it.
- skrause 8y ago
  What about external hard disks with terrible USB controllers? That's where data checksumming would be even more important and where Apple doesn't produce the hardware.
  - tobias3 8y ago
    Maybe APFS was mainly developed for iOS?
  - derefr 8y ago
    Apple doesn’t recommend using APFS for external media.
    - pvg 8y ago
      Really? Can you remember where you've seen this non-recommendation?
      - unicornporn 8y ago
        Apple isn't really embracing APFS for external drives as I see it. Time Machine backups would be one of the main uses for external drives, but… > Time Machine can’t back up to an APFS-formatted disk. If you select a back up disk formatted as APFS, Time Machine offers to reformat it as HFS+.[1] [1] https://support.apple.com/guide/mac-help/types-of-disks-you-can-use-with-time-machine-mh15139/mac https://support.apple.com/guide/mac-help/types-of-disks-you-...
        pvg 8y ago
        This has more to do with the implementation details of TimeMachine which relies on features of HFS+ not available in APFS. It's not Apple telling you not to use APFS on external drives.
      - 4ad 8y ago
        I can't recall if I ever saw it, but by default it will use HFS+ for external drives.
  - jxi 8y ago
    Every time I unplug an external USB prematurely, the next time I plug it into my mac, it does a full filesystem check of it which takes a few mins.
- leonroy 8y ago
  There have been at least two occasions in the past three years that I remember where Apple had to recall or issue firmware updates for SSDs due to corruption issues. It’s a big IF to depend solely on flash based error checking. I think Apple likely figured the cost of implementation and system overhead wasn’t worth it for Macs and iOS devices. I’d be very surprised if they use APFS or a non checksumning file system on their iCloud server farm though.

burtonator 8y ago

A lot of applications and databases do this themselves. Part of the problem is that the app might know what the proper checksum is and it would be nice to be able to write a document ahead of time and tell the FS that you already know the checksum. Won't save a massive amount of CPU but it would be worth it if it was easy to implement.

hrez 8y ago

There is also windows ReFS and Dragonfly's HAMMER2.

deleted 8y ago
[deleted]

z3t4 8y ago

There is however no point in having data checks if there are no way of recovering, so running ZFS only make sense if you have multiple disks. And don't forget to scrub your pool!

tux3 8y ago
Knowing that your data is corrupt is always better than not knowing. Whether or not you have a RAID you should still have backups, and you don't want to replace a good backup by a corrupt one.
- z3t4 8y ago
  OK, so you know your HDD is corrupted, and you now have to take the site down to recover from backup, which is also old, so there will be data missing. Compared to if the FS can just fix itself. Sometimes not knowing can be a bliss though. :P
elfchief 8y ago
This is like saying "there's no point in ECC if it can't correct every error" ... which simply isn't true. It's still far better to know that corruption exists -- so you can know something is invalid and potentially take action -- than to have corruption silently hanging around doing corrupted data things.
- z3t4 8y ago
  That is not a fair comparison as ECC is Error Correcting, while just check-summing is not. If you however have a mirror or raid-Z the file system will be able to correct.
mrighele 8y ago
You can set the number of copies to keep at filesystem level. This is in addition to any redundancy provided at pool level (if any). It won't save you from a failed disk, but it will be able to transparently correct any checksum error.
8y ago

equalunique 8y ago

BcacheFS has it & is under active development: https://bcachefs.org/ https://bcachefs.org/