3 ms·

It's interesting that XFS is the recommended filesystem for databases (event distributed ones like CochroachDB), when it does not have integrity checks that ZFS

It's interesting that XFS is the recommended filesystem for databases (event distributed ones like CochroachDB), when it does not have integrity checks that ZFS or BTRFS have. Is performance really more important than integrity for databases?

georgia_peach 4y ago
XFS has a long and solid track record, but without the workload-dependent regressions which can be encountered in a copy-on-write filesystem. Decent hard drives have had internal checksums for ages. If the drive is even halfway functioning, it will catch the bad blocks--making that particular ZFS feature largely redundant, and databases that matter are going to be on a RAID anyway, which has its own additional layer of parity. ZFS is fantastic for some applications, not so much for others.
- z3t4 4y ago
  Ive had many checksum errors with ZFS, really no difference between enterprise Hdd or cheapest possible hdd (new ones). Or i just got unlucky. But some hdds are 15+ years, bad sectors but still usable.
  - georgia_peach 4y ago
    Checksum vs parity. I would wager that the drive firmware also detected the error. This is why I think ZFS checksumming is largely redundant. Parity / error correction over a zpool being another issue entirely. This is not to ding ZFS. If I were in Sun's position of having to take responsibility for 3rd-party manufactured drives, I'd have done the same thing. And circa 2006, the reliability of in-drive error detection may have been in a poorer state. Never had a corruption issue the past 10 years that wasn't preceded by months of SMART errors/warnings.
    - tmikaeld 4y ago
      It is absolutely not redundant, ZFS RAIDZ can reconstruct damaged data, thanks to the extra checksum provided by ZFS. While traditional RAID does not recover from silent data corruption because it does not know which block to reconstruct. This is exceptionally sensitive when backups are encrypted, because even a few bits off and you can't decrypt the backups. Since this happens transparently, regularly testing backups are very important in non-ZFS systems.
      - georgia_peach 4y ago
        Read what I wrote more carefully. Checksumming is error detection only, while parity schemes can handle both error detection & correction. Checksumming blocks in software that are already being checksummed in hardware is redundant. Parity, on the other hand, is an essential part of any data redundancy system. For a transactional database, XFS + hardware RAID is going to be a more reliable setup than ZFS + JBOD. Database logs are similar in function to CoW, & the fancier DBs have time-machine features similar to snapshots. Having filesystem & software struggling against each other to solve the same problem, but with different solutions, is a situation that invites so many Heisenbugs. That, and a good RAID controller is going to handle drive-reported errors sooner and more gracefully than a ZFS scrub.
mrweasel 4y ago
I don't have the answer, but it's an interesting question. Given that XFS doesn't do replication, like ZFS or BTRFS does, having integrity checks is a bit weird. If a check fails, what is XFS suppose to do about it? If the filesystem can "recover" or deal with failed integrity checks, it makes sense to have them at that layer. If it can't, I'd say that the application, in this case a database, should do it's own integrity checks. If it's just a few rows or tables, the operator can easily recover those from a backup. Having the filesystem, which knows nothing about the data, report the failure will mean having to do a full restore from backups.
- tmikaeld 4y ago
  ZFS RAIDZ can reconstruct damaged data, thanks to the extra checksum provided by ZFS. While traditional RAID does not recover from silent data corruption because it does not know which block to reconstruct. This is exceptionally sensitive when backups are encrypted, because even a few bits off and you can't decrypt the backups. Since this happens transparently, regularly testing backups are very important in non-ZFS systems.
  - dijit 4y ago
    ZFS is more akin to a logical volume manager (inc. software raid) and a filesystem though. XFS is just a filesystem, it's not trying to solve the same issues. Higher end RAID cards also do block level validations in the background. It isn't filesystem aware but it is sufficient to suss out bad clusters and relocate data proactively. The point of ZFS systems is to avoid such hardware based checksumming systems, but RAID cards have been doing this validation for many years now. The hardware solutions we have now provide silent corruption protection from the storage controller down to the actual media. ZFS extends that protection to the filesystem layer in the kernel and when used in isolation of modern hardware solutions can protect the whole stack.

georgia_peach 4y ago

XFS has a long and solid track record, but without the workload-dependent regressions which can be encountered in a copy-on-write filesystem. Decent hard drives have had internal checksums for ages. If the drive is even halfway functioning, it will catch the bad blocks--making that particular ZFS feature largely redundant, and databases that matter are going to be on a RAID anyway, which has its own additional layer of parity. ZFS is fantastic for some applications, not so much for others.

z3t4 4y ago
Ive had many checksum errors with ZFS, really no difference between enterprise Hdd or cheapest possible hdd (new ones). Or i just got unlucky. But some hdds are 15+ years, bad sectors but still usable.
- georgia_peach 4y ago
  Checksum vs parity. I would wager that the drive firmware also detected the error. This is why I think ZFS checksumming is largely redundant. Parity / error correction over a zpool being another issue entirely. This is not to ding ZFS. If I were in Sun's position of having to take responsibility for 3rd-party manufactured drives, I'd have done the same thing. And circa 2006, the reliability of in-drive error detection may have been in a poorer state. Never had a corruption issue the past 10 years that wasn't preceded by months of SMART errors/warnings.
  - tmikaeld 4y ago
    It is absolutely not redundant, ZFS RAIDZ can reconstruct damaged data, thanks to the extra checksum provided by ZFS. While traditional RAID does not recover from silent data corruption because it does not know which block to reconstruct. This is exceptionally sensitive when backups are encrypted, because even a few bits off and you can't decrypt the backups. Since this happens transparently, regularly testing backups are very important in non-ZFS systems.
    - georgia_peach 4y ago
      Read what I wrote more carefully. Checksumming is error detection only, while parity schemes can handle both error detection & correction. Checksumming blocks in software that are already being checksummed in hardware is redundant. Parity, on the other hand, is an essential part of any data redundancy system. For a transactional database, XFS + hardware RAID is going to be a more reliable setup than ZFS + JBOD. Database logs are similar in function to CoW, & the fancier DBs have time-machine features similar to snapshots. Having filesystem & software struggling against each other to solve the same problem, but with different solutions, is a situation that invites so many Heisenbugs. That, and a good RAID controller is going to handle drive-reported errors sooner and more gracefully than a ZFS scrub.

mrweasel 4y ago

I don't have the answer, but it's an interesting question. Given that XFS doesn't do replication, like ZFS or BTRFS does, having integrity checks is a bit weird. If a check fails, what is XFS suppose to do about it? If the filesystem can "recover" or deal with failed integrity checks, it makes sense to have them at that layer. If it can't, I'd say that the application, in this case a database, should do it's own integrity checks. If it's just a few rows or tables, the operator can easily recover those from a backup. Having the filesystem, which knows nothing about the data, report the failure will mean having to do a full restore from backups.

tmikaeld 4y ago
ZFS RAIDZ can reconstruct damaged data, thanks to the extra checksum provided by ZFS. While traditional RAID does not recover from silent data corruption because it does not know which block to reconstruct. This is exceptionally sensitive when backups are encrypted, because even a few bits off and you can't decrypt the backups. Since this happens transparently, regularly testing backups are very important in non-ZFS systems.
- dijit 4y ago
  ZFS is more akin to a logical volume manager (inc. software raid) and a filesystem though. XFS is just a filesystem, it's not trying to solve the same issues. Higher end RAID cards also do block level validations in the background. It isn't filesystem aware but it is sufficient to suss out bad clusters and relocate data proactively. The point of ZFS systems is to avoid such hardware based checksumming systems, but RAID cards have been doing this validation for many years now. The hardware solutions we have now provide silent corruption protection from the storage controller down to the actual media. ZFS extends that protection to the filesystem layer in the kernel and when used in isolation of modern hardware solutions can protect the whole stack.