14 ms·

I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesyste

I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?

formerly_proven 4y ago
Unless you are also doing FS-level deduplication using the same checksums, it generally makes no sense for these to be cryptographic hashes, so they're not necessarily suitable for this purpose. IIRC neither ZFS nor btrfs use cryptographic hashes for checksumming by default.
- throw0101a 4y ago
  > on is a short hand for fletcher4 for non-deduped datasets and sha256 for deduped datasets * https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Checksums.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec... * https://people.freebsd.org/~asomers/fletcher.pdf https://people.freebsd.org/~asomers/fletcher.pdf * https://en.wikipedia.org/wiki/Fletcher%27s_checksum https://en.wikipedia.org/wiki/Fletcher%27s_checksum Strangely enough SHA-512 is actually (50%) faster than -256: > ZFS actually uses a special version of SHA512 called SHA512t256, it uses a different initial value, and truncates the results to 256 bits (that is all the room there is in the block pointer). The advantage is only that it is faster on 64 bit CPUs. * https://twitter.com/klarainc/status/1367199461546614788 https://twitter.com/klarainc/status/1367199461546614788
  - deleted 4y ago
    [deleted]
  - slavik81 4y ago
    It seems that SHA512t256 is another name for SHA-512/256. It's a shame that the initialization is different from SHA-512, as it would have been very useful to be able to convert a SHA-512 hash into a SHA-512/256 hash.
js2 4y ago
> I guess zfs send and similar are better solutions. It depends. I recently built a new zfs pool server and needed to transfer a few TB of data from the old pool to the new pool, but I built the new pool with a larger record size. If I’d used zfs send the files would have retained their existing record size. So rsync it was.
GekkePrutser 4y ago
ZFS send does not do any kind of cross checking with the receiving end so no, not really ideal. Even incremental ZFS sends don't do this, it keeps state on the sending side only. It's ok for its intended usecases but it's not a direct rsync replacement. It's also hard/impossible to restore individual files out of a ZFS send stream without restoring the whole thing so I've reverted to using tarballs of ZFS snapshots for backups instead of ZFS send. Again, it was never really meant for this so it was my mistake trying to use it that way.

formerly_proven 4y ago

Unless you are also doing FS-level deduplication using the same checksums, it generally makes no sense for these to be cryptographic hashes, so they're not necessarily suitable for this purpose. IIRC neither ZFS nor btrfs use cryptographic hashes for checksumming by default.

throw0101a 4y ago
> on is a short hand for fletcher4 for non-deduped datasets and sha256 for deduped datasets * https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Checksums.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec... * https://people.freebsd.org/~asomers/fletcher.pdf https://people.freebsd.org/~asomers/fletcher.pdf * https://en.wikipedia.org/wiki/Fletcher%27s_checksum https://en.wikipedia.org/wiki/Fletcher%27s_checksum Strangely enough SHA-512 is actually (50%) faster than -256: > ZFS actually uses a special version of SHA512 called SHA512t256, it uses a different initial value, and truncates the results to 256 bits (that is all the room there is in the block pointer). The advantage is only that it is faster on 64 bit CPUs. * https://twitter.com/klarainc/status/1367199461546614788 https://twitter.com/klarainc/status/1367199461546614788
- deleted 4y ago
  [deleted]
- slavik81 4y ago
  It seems that SHA512t256 is another name for SHA-512/256. It's a shame that the initialization is different from SHA-512, as it would have been very useful to be able to convert a SHA-512 hash into a SHA-512/256 hash.

js2 4y ago

> I guess zfs send and similar are better solutions. It depends. I recently built a new zfs pool server and needed to transfer a few TB of data from the old pool to the new pool, but I built the new pool with a larger record size. If I’d used zfs send the files would have retained their existing record size. So rsync it was.

GekkePrutser 4y ago

ZFS send does not do any kind of cross checking with the receiving end so no, not really ideal. Even incremental ZFS sends don't do this, it keeps state on the sending side only. It's ok for its intended usecases but it's not a direct rsync replacement. It's also hard/impossible to restore individual files out of a ZFS send stream without restoring the whole thing so I've reverted to using tarballs of ZFS snapshots for backups instead of ZFS send. Again, it was never really meant for this so it was my mistake trying to use it that way.