3 ms·

Caveat emptor: I've run ZFS on a two-digit number of machines for >10 years. But no corporate deployment. To me, a big benefit of ZFS' rampant layering violati

Caveat emptor: I've run ZFS on a two-digit number of machines for >10 years. But no corporate deployment. To me, a big benefit of ZFS' rampant layering violation is that it shrinks the surface of weird interactions and edge cases to learn about. You get to know the FS, play with it, and then off you go. Most other solutions require a lot of thought and planning. Perhaps there is a better way to tune the storage setup to exactly what you want, but with ZFS you aren't seduced into getting creative with the architecture. So I'm in favor of native ZFS encryption due to its usability. That said, it is not as mature as I'd like, and there is at least one very big issue to know about: I recently sent an encrypted FS with zfs send/receive, and the received copy ended up being unreadable. I'll see if I can find the bug again. So that means you should test your setup precisely. [edit] Here is the issue: https://github.com/openzfs/zfs/issues/12594 https://github.com/openzfs/zfs/issues/12594 The technical detail from that thread: "This happens because when sending raw encrypted datasets the userspace accounting is present when it's not expected to be. This leads to the subsequent mount failure due a checksum error when verifying the local mac. I tried unsuccessfully to tackle this in #11300. See also: #10523, #11221, #11294. Edit: If you have critical data lost due to this case I could help you recover them." And there's this comment [https://github.com/openzfs/zfs/issues/12594#issuecomment-929941596 https://github.com/openzfs/zfs/issues/12594#issuecomment-929...] which has a lot more pointers to recent problems.

3np 4y ago
If I read this right these issues may actually be resolved as of 2.1.3 ? At least there's nothing else open on the tracker.
- uniqueuid 4y ago
  Yes all the issues should be resolved by now. But given that the bugs exist, anyone using encryption should test and verify all the functionality before using it in production. The worst thing about this bug is that it doesn't manifest on the local machine but in a situation where things seem ok.
- rincebrain 4y ago
  Nope, not even close.
  - 3np 4y ago
    Care to elaborate?
    - rincebrain 4y ago
      Sure, have an explanation and a link to my (currently missing one or two newer bugs) spreadsheet. https://news.ycombinator.com/item?id=32342900 https://news.ycombinator.com/item?id=32342900
josteink 4y ago
> So I'm in favor of native ZFS encryption due to its usability. Another reason to favour it is because if you do ZFS Send/Recieve to do incremental backups (like with rsync.net), you can backup your encrypted datasets to an off-site location without shipping the encryption keys. That means you don't have to be afraid of your backups somehow resulting in breaking your confidentiality. So you can still retain your incremental ZFS based backup-flow, and you don't have to change any tooling to use it. With LUKS you will either have to store your backups unencrypted, or you lose the ability to do incremental ZFS-based backups.
- dev_hugepages 4y ago
  The LUKS headers doesn't contain the encryption keys, they're encrypted from the user's passphrase through a KDF. Additionally, you can make the LUKS header "detached" (external) see https://wiki.archlinux.org/title/Dm-crypt/Specialties#Encrypted_system_using_a_detached_LUKS_header https://wiki.archlinux.org/title/Dm-crypt/Specialties#Encryp...
  - josteink 4y ago
    But my point was that if you want to backup such a volume (the encrypted LUKS volume), you will have to backup the whole volume, and can't do incremental backup. If you want to do incremental ZFS send/recieve backups, you have to do it on the unencrypted ZFS volume, and thus your backup wont be encrypted. That is you can't have both encrypted and incremental backups at the same time. With native ZFS encryption you do get that.
dsp_person 4y ago
yup see rincebrain's first reply where nearly every word in the sentence is a url to a different encryption issue > [I recommend](https://github.com/openzfs/zfs/issues/11679 https://github.com/openzfs/zfs/issues/11679) [not using](https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014) [native encryption](https://www.illumos.org/issues/14003 https://www.illumos.org/issues/14003) [until it gets](https://github.com/openzfs/zfs/issues/12270 https://github.com/openzfs/zfs/issues/12270) [a fair bit](https://github.com/openzfs/zfs/issues/12439 https://github.com/openzfs/zfs/issues/12439) [more polish](https://github.com/openzfs/zfs/issues/12418 https://github.com/openzfs/zfs/issues/12418) [in the future](https://github.com/openzfs/zfs/issues/11983 https://github.com/openzfs/zfs/issues/11983) ([I'm only so hopeful](https://openzfs.topicbox.com/groups/developer/T86620b1082ed947a/would-anyone-be-able-to-help-with-11679 https://openzfs.topicbox.com/groups/developer/T86620b1082ed9...)). Unfortunately this knowledge isn't wide-spread, and as far as I see the zfs devs seem to NOT recommend using encryption right now without accepting the risks of dataloss
- uniqueuid 4y ago
  Right, that's the comment I linked to.
- rincebrain 4y ago
  Most of the devs seem to be of the opinion "we don't use it so we have no opinion", or there wouldn't be horrifying bugs in it over 3 years old. That said, very few if any of them have involved actual irrecoverable data loss versus just requiring lots of handholding to get your data back. edit: I suppose at the point where I'm being quoted on HN about this, I might as well link my list of encryption bugs. https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/edit?usp=sharing https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww... I still need to add a couple of new bugs, but almost all of them are dupes or related to one of two flaws - a narrow race in the ARC (...probably), where you can sometimes NULL dereference for fun and sadness, and zfs change-key and send -i resulting in the receiver incorrectly resetting the metadata for which key an encryptionroot is encrypted by.
65a 4y ago
I filed #11294, it's a non-issue for me now. Seemed like an API change stability issue, which happens. I think what a crypto-user might care about more in ZFS vs LUKS is precisely what gets encrypted: LUKS can be the entire disk, whereas ZFS is going to need some metadata to know what is there, which keys unlock it, etc on at least a per-dataset basis. It's not a tradeoff that affects my particular threat model, but it's worth understanding as a crypto user.

3np 4y ago

If I read this right these issues may actually be resolved as of 2.1.3 ? At least there's nothing else open on the tracker.

uniqueuid 4y ago
Yes all the issues should be resolved by now. But given that the bugs exist, anyone using encryption should test and verify all the functionality before using it in production. The worst thing about this bug is that it doesn't manifest on the local machine but in a situation where things seem ok.
rincebrain 4y ago
Nope, not even close.
- 3np 4y ago
  Care to elaborate?
  - rincebrain 4y ago
    Sure, have an explanation and a link to my (currently missing one or two newer bugs) spreadsheet. https://news.ycombinator.com/item?id=32342900 https://news.ycombinator.com/item?id=32342900

josteink 4y ago

> So I'm in favor of native ZFS encryption due to its usability. Another reason to favour it is because if you do ZFS Send/Recieve to do incremental backups (like with rsync.net), you can backup your encrypted datasets to an off-site location without shipping the encryption keys. That means you don't have to be afraid of your backups somehow resulting in breaking your confidentiality. So you can still retain your incremental ZFS based backup-flow, and you don't have to change any tooling to use it. With LUKS you will either have to store your backups unencrypted, or you lose the ability to do incremental ZFS-based backups.

dev_hugepages 4y ago
The LUKS headers doesn't contain the encryption keys, they're encrypted from the user's passphrase through a KDF. Additionally, you can make the LUKS header "detached" (external) see https://wiki.archlinux.org/title/Dm-crypt/Specialties#Encrypted_system_using_a_detached_LUKS_header https://wiki.archlinux.org/title/Dm-crypt/Specialties#Encryp...
- josteink 4y ago
  But my point was that if you want to backup such a volume (the encrypted LUKS volume), you will have to backup the whole volume, and can't do incremental backup. If you want to do incremental ZFS send/recieve backups, you have to do it on the unencrypted ZFS volume, and thus your backup wont be encrypted. That is you can't have both encrypted and incremental backups at the same time. With native ZFS encryption you do get that.

dsp_person 4y ago

yup see rincebrain's first reply where nearly every word in the sentence is a url to a different encryption issue > [I recommend](https://github.com/openzfs/zfs/issues/11679 https://github.com/openzfs/zfs/issues/11679) [not using](https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014) [native encryption](https://www.illumos.org/issues/14003 https://www.illumos.org/issues/14003) [until it gets](https://github.com/openzfs/zfs/issues/12270 https://github.com/openzfs/zfs/issues/12270) [a fair bit](https://github.com/openzfs/zfs/issues/12439 https://github.com/openzfs/zfs/issues/12439) [more polish](https://github.com/openzfs/zfs/issues/12418 https://github.com/openzfs/zfs/issues/12418) [in the future](https://github.com/openzfs/zfs/issues/11983 https://github.com/openzfs/zfs/issues/11983) ([I'm only so hopeful](https://openzfs.topicbox.com/groups/developer/T86620b1082ed947a/would-anyone-be-able-to-help-with-11679 https://openzfs.topicbox.com/groups/developer/T86620b1082ed9...)). Unfortunately this knowledge isn't wide-spread, and as far as I see the zfs devs seem to NOT recommend using encryption right now without accepting the risks of dataloss

uniqueuid 4y ago
Right, that's the comment I linked to.
rincebrain 4y ago
Most of the devs seem to be of the opinion "we don't use it so we have no opinion", or there wouldn't be horrifying bugs in it over 3 years old. That said, very few if any of them have involved actual irrecoverable data loss versus just requiring lots of handholding to get your data back. edit: I suppose at the point where I'm being quoted on HN about this, I might as well link my list of encryption bugs. https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/edit?usp=sharing https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww... I still need to add a couple of new bugs, but almost all of them are dupes or related to one of two flaws - a narrow race in the ARC (...probably), where you can sometimes NULL dereference for fun and sadness, and zfs change-key and send -i resulting in the receiver incorrectly resetting the metadata for which key an encryptionroot is encrypted by.

65a 4y ago

I filed #11294, it's a non-issue for me now. Seemed like an API change stability issue, which happens. I think what a crypto-user might care about more in ZFS vs LUKS is precisely what gets encrypted: LUKS can be the entire disk, whereas ZFS is going to need some metadata to know what is there, which keys unlock it, etc on at least a per-dataset basis. It's not a tradeoff that affects my particular threat model, but it's worth understanding as a crypto user.