4 ms·

I think one of ZFS's most significant contributions was embracing the specific ways in which disks and HBAs often fail and then building mechanisms to ensure da

by dap 9y ago

I think one of ZFS's most significant contributions was embracing the specific ways in which disks and HBAs often fail and then building mechanisms to ensure data integrity in the face of those failures. Bit rot and phantom writes are the filesystem's problem, even though they're not the filesystem's fault. ZFS did a lot of work to ensure that integrity: storing checksums in parent blocks, storing metadata redundantly, fixing bad copies with the good ones when corruption is detected, and scrubbing. In many filesystems still in use today, applications can easily receive garbage data from the system. I understand this filesystem is still nascent, but shouldn't data integrity at least be one of the design goals?

deleted 9y ago
[deleted]
ticki_ 9y ago
> I understand this filesystem is still nascent, but shouldn't data integrity at least be one of the design goals? What makes you think it isn't? It definitely is. In fact, it borrows several ideas from ZFS wrt/ integrity. For example, it uses parent block checksums like ZFS.
- cmurf 9y ago
  I can't tell from the code whether this is metadata only checksumming or if data is also checksummed. ZFS and Btrfs checksum both metadata and data.
  - ticki_ 9y ago
    The data is summed too.
- dap 9y ago
  > What makes you think it isn't? The first section in the README is called "Design goals", with 13 items. None of them is "data integrity", and none of them even talks about validating the data or handling any failures aside from power loss. By contrast, in the canonical slide deck on ZFS[1], the first slide talks about "provable end-to-end data integrity". In the paper[2], "design principles" section 2.6 is "error detection and correction". I'm glad to hear that's also a focus for TFS. With ZFS, the emphasis on data integrity resulted in significant architectural choices -- I'm not sure it's something that can just be bolted on later. As a reader, I wouldn't have assumed TFS had the same emphasis. I think it's pretty valuable to spell this out early and clearly, with details, because it's actually quite a differentiator compared with most other systems. [1] https://wiki.illumos.org/download/attachments/1146951/zfs_last.pdf https://wiki.illumos.org/download/attachments/1146951/zfs_la... [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.184.3704&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.184...
  - ticki_ 9y ago
    > The first section in the README is called "Design goals", with 13 items. None of them is "data integrity", and none of them even talks about validating the data or handling any failures aside from power loss. Fair enough.

by dap 9y ago

deleted 9y ago

[deleted]

ticki_ 9y ago

> I understand this filesystem is still nascent, but shouldn't data integrity at least be one of the design goals? What makes you think it isn't? It definitely is. In fact, it borrows several ideas from ZFS wrt/ integrity. For example, it uses parent block checksums like ZFS.

cmurf 9y ago
I can't tell from the code whether this is metadata only checksumming or if data is also checksummed. ZFS and Btrfs checksum both metadata and data.
- ticki_ 9y ago
  The data is summed too.
dap 9y ago
> What makes you think it isn't? The first section in the README is called "Design goals", with 13 items. None of them is "data integrity", and none of them even talks about validating the data or handling any failures aside from power loss. By contrast, in the canonical slide deck on ZFS[1], the first slide talks about "provable end-to-end data integrity". In the paper[2], "design principles" section 2.6 is "error detection and correction". I'm glad to hear that's also a focus for TFS. With ZFS, the emphasis on data integrity resulted in significant architectural choices -- I'm not sure it's something that can just be bolted on later. As a reader, I wouldn't have assumed TFS had the same emphasis. I think it's pretty valuable to spell this out early and clearly, with details, because it's actually quite a differentiator compared with most other systems. [1] https://wiki.illumos.org/download/attachments/1146951/zfs_last.pdf https://wiki.illumos.org/download/attachments/1146951/zfs_la... [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.184.3704&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.184...
- ticki_ 9y ago
  > The first section in the README is called "Design goals", with 13 items. None of them is "data integrity", and none of them even talks about validating the data or handling any failures aside from power loss. Fair enough.