5 ms·

OpenZFS bug reports for native encryption

38 points by transpute 3y ago

nesarkvechnep 3y ago
I guess someone’s mentioned how great ZFS is and this page is here to prove the opposite. Every file system has bugs.
- rincebrain 3y ago
  I mean, the spreadsheet was to illustrate how many bugs this feature had and how unfixed they are. Source: it's my document, though I didn't submit it here, and haven't updated it in a while after I gave up trying to convince other people about how bad an idea using it is.
  - transpute 3y ago
    Thanks for creating the spreadsheet. Discovered while banging head against docs and builds. The feature should be emblazoned as experimental and advertised as seeking developers/funding, to protect ZFS reputation.
- transpute 3y ago
  Were the "stale" / timed-out bugs due to user or developer attrition?
  - nubinetwork 3y ago
    https://github.com/openzfs/zfs/issues/10769 https://github.com/openzfs/zfs/issues/10769 I wasn't a fan of that change, it serves no purpose other than to sweep issues under the carpet (which can be said of any project that uses one). Which is funny because they have just as many open issues now as they did when it was implemented. Had they had better user/developer relations and documentation, it's possible that a lot of open issues would have never been opened. (edit: or remained open otherwise)
    - transpute 3y ago
      That's a troubling policy signal, with impact beyond encryption.
      - nubinetwork 3y ago
        To be fair, I believe the test suite has improved over the years, but adding new features requires meticulous caution to make sure you don't miss any edge cases, at least on the functional side. But then you occasionally get bugs like https://github.com/openzfs/zfs/issues/15453 https://github.com/openzfs/zfs/issues/15453 which can't easily be caught by test suites.
- stephen_g 3y ago
  ZFS is great and generally stable and mature, but it seems like the support for encryption, a buggy optional feature, was merged in without enough scrutiny and the problems are still unfixed. All the advice you'll get from anybody experienced (which I am not very with ZFS, but have been setting up some NAS projects with it so am researching) is to not even touch encryption with a 10 foot pole. As long as you don't enable it (and I expect a lot of GUI driven NAS products probably don't even let you through the GUI or warn you if they do), you're good.
  - didntcheck 3y ago
    Which is a shame, since zero-trust zfs send is a pretty nice feature, if it could be relied upon
    - stephen_g 3y ago
      Yeah, true, I want to use it for backups but unfortunately I can't given the instability!
nubinetwork 3y ago
I was pretty active in ZOL when encryption was added. It was handshakes and applause all around... but shortly afterwards the developer started contributing less and less, while several bugs cropped up... some necessitating sending entire hard drive images over ssh to other developers. Don't get me wrong, I like zfs and use it every day, but that code should have never been merged, and given more scrutiny considering it completely changes how your data is written to disk. Thankfully not many people use encryption.
- rincebrain 3y ago
  As much as I lament the status of native encryption in OpenZFS, and think it would have been better to merge it with more testing, I think what I find to be really more problematic is the lack of enough fixing afterward. Software is buggy, and it's going to have bugs you don't find no matter how much you tested it. The problem was that (AFAIR) one developer at the company that developed this contributed fixes, and then he left, and then almost nobody did.
  - nubinetwork 3y ago
    I think people were more interested in giving Oracle the finger than anything, considering missing feature parity and compatibility were constantly waved in our faces. (Not that ZOL encryption is compatible at all with Oracle's code) I often wonder if encryption would have been merged if Matt had usurped ZOL sooner.
    - rincebrain 3y ago
      "usurp" is an interesting word for "decide ZoL was the project of record and replace the illumos reference platform with it", and since Matt was the one who said no because there's no mitigation other than not using it when I wanted to put a warning in the docs about it, I wouldn't anticipate him doing anything positive.
      - nubinetwork 3y ago
Mic92 3y ago
Most bugs seem to be around `zfs send`. I don't use this feature and my 2 encrypted zfs pools seem work without any issues...
- rincebrain 3y ago
  I've said this before, but while a lot of the bugs are most easily triggered with `zfs send`, you can also trigger them other ways depending on which bug on which platform - illumos, for example, used to be able to reliably trigger a crash with the same trace as one of the "send/recv" bugs on ZoL/OpenZFS with just "zfs clone" (it might still, I haven't tried in a while). Another one of them got what looks like a duplicate posted that triggered on pool creation in the test suite sometimes. So I don't think it's accurate to say "it's safe if you don't use zfs send" - while some of the bugs might actually be exclusive to send/recv, it seems like a number of them are just most easily triggered that way. It clearly works "fine" for some people - e.g. I believe the original authors of the feature still use it in production, last I knew, because their workflow doesn't hit any of these issues. But "using this feature adds a 1% chance of a lion eating your face" seems like something worth warning people about, if there was not a face-eating risk prior. (Especially because a number of them seem to be races, so how much the problem comes up is really going to be hard to predict a priori...)
  - jclulow 3y ago
    FWIW, I believe we've now fixed the clone-related crash issue, as of https://www.illumos.org/issues/14003 https://www.illumos.org/issues/14003
transpute 3y ago
12 Feb 2024, https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt and https://github.com/openzfs/openzfs-docs/issues/494 https://github.com/openzfs/openzfs-docs/issues/494 > A new ticket has been opened for OpenZFS as well in proposing to add warnings against using ZFS native encryption and the send/receive support in production environments. "Among experienced zfs users and developers, it's conventional wisdom that zfs native encryption is not suitable for production usage, particularly when combined with snapshotting and zfs send/recv. There is a long standing data corruption issue with many firsthand user reports...Additionally, if you join #zfs or #zfsonlinux on freenode and mention that you're having an issue with zfs native encryption, you'll be met with advice from developers that zfs native encryption is simply not reliable." From Phoronix comments, a suggestion on implementation: > It should be something like that: ERROR: RAID 5/6 support has know problems and shouldn't be used on production systems. If you really want to use it please type "Enable dangerous RAID5/6 support" or add argument "--enable-dangerous-raid56-support" to this command
- nubinetwork 3y ago
  Why are they still using freenode? Everyone jumped ship like 2 or so years ago?
  - transpute 3y ago
    Hmm, good question.

38 points by transpute 3y ago

nesarkvechnep 3y ago

I guess someone’s mentioned how great ZFS is and this page is here to prove the opposite. Every file system has bugs.

rincebrain 3y ago
I mean, the spreadsheet was to illustrate how many bugs this feature had and how unfixed they are. Source: it's my document, though I didn't submit it here, and haven't updated it in a while after I gave up trying to convince other people about how bad an idea using it is.
- transpute 3y ago
  Thanks for creating the spreadsheet. Discovered while banging head against docs and builds. The feature should be emblazoned as experimental and advertised as seeking developers/funding, to protect ZFS reputation.
transpute 3y ago
Were the "stale" / timed-out bugs due to user or developer attrition?
- nubinetwork 3y ago
  https://github.com/openzfs/zfs/issues/10769 https://github.com/openzfs/zfs/issues/10769 I wasn't a fan of that change, it serves no purpose other than to sweep issues under the carpet (which can be said of any project that uses one). Which is funny because they have just as many open issues now as they did when it was implemented. Had they had better user/developer relations and documentation, it's possible that a lot of open issues would have never been opened. (edit: or remained open otherwise)
  - transpute 3y ago
    That's a troubling policy signal, with impact beyond encryption.
    - nubinetwork 3y ago
      To be fair, I believe the test suite has improved over the years, but adding new features requires meticulous caution to make sure you don't miss any edge cases, at least on the functional side. But then you occasionally get bugs like https://github.com/openzfs/zfs/issues/15453 https://github.com/openzfs/zfs/issues/15453 which can't easily be caught by test suites.
stephen_g 3y ago
ZFS is great and generally stable and mature, but it seems like the support for encryption, a buggy optional feature, was merged in without enough scrutiny and the problems are still unfixed. All the advice you'll get from anybody experienced (which I am not very with ZFS, but have been setting up some NAS projects with it so am researching) is to not even touch encryption with a 10 foot pole. As long as you don't enable it (and I expect a lot of GUI driven NAS products probably don't even let you through the GUI or warn you if they do), you're good.
- didntcheck 3y ago
  Which is a shame, since zero-trust zfs send is a pretty nice feature, if it could be relied upon
  - stephen_g 3y ago
    Yeah, true, I want to use it for backups but unfortunately I can't given the instability!

nubinetwork 3y ago

I was pretty active in ZOL when encryption was added. It was handshakes and applause all around... but shortly afterwards the developer started contributing less and less, while several bugs cropped up... some necessitating sending entire hard drive images over ssh to other developers. Don't get me wrong, I like zfs and use it every day, but that code should have never been merged, and given more scrutiny considering it completely changes how your data is written to disk. Thankfully not many people use encryption.

rincebrain 3y ago
As much as I lament the status of native encryption in OpenZFS, and think it would have been better to merge it with more testing, I think what I find to be really more problematic is the lack of enough fixing afterward. Software is buggy, and it's going to have bugs you don't find no matter how much you tested it. The problem was that (AFAIR) one developer at the company that developed this contributed fixes, and then he left, and then almost nobody did.
- nubinetwork 3y ago
  I think people were more interested in giving Oracle the finger than anything, considering missing feature parity and compatibility were constantly waved in our faces. (Not that ZOL encryption is compatible at all with Oracle's code) I often wonder if encryption would have been merged if Matt had usurped ZOL sooner.
  - rincebrain 3y ago
    "usurp" is an interesting word for "decide ZoL was the project of record and replace the illumos reference platform with it", and since Matt was the one who said no because there's no mitigation other than not using it when I wanted to put a warning in the docs about it, I wouldn't anticipate him doing anything positive.
    - nubinetwork 3y ago

Mic92 3y ago

Most bugs seem to be around `zfs send`. I don't use this feature and my 2 encrypted zfs pools seem work without any issues...

rincebrain 3y ago
I've said this before, but while a lot of the bugs are most easily triggered with `zfs send`, you can also trigger them other ways depending on which bug on which platform - illumos, for example, used to be able to reliably trigger a crash with the same trace as one of the "send/recv" bugs on ZoL/OpenZFS with just "zfs clone" (it might still, I haven't tried in a while). Another one of them got what looks like a duplicate posted that triggered on pool creation in the test suite sometimes. So I don't think it's accurate to say "it's safe if you don't use zfs send" - while some of the bugs might actually be exclusive to send/recv, it seems like a number of them are just most easily triggered that way. It clearly works "fine" for some people - e.g. I believe the original authors of the feature still use it in production, last I knew, because their workflow doesn't hit any of these issues. But "using this feature adds a 1% chance of a lion eating your face" seems like something worth warning people about, if there was not a face-eating risk prior. (Especially because a number of them seem to be races, so how much the problem comes up is really going to be hard to predict a priori...)
- jclulow 3y ago
  FWIW, I believe we've now fixed the clone-related crash issue, as of https://www.illumos.org/issues/14003 https://www.illumos.org/issues/14003

transpute 3y ago

12 Feb 2024, https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt and https://github.com/openzfs/openzfs-docs/issues/494 https://github.com/openzfs/openzfs-docs/issues/494 > A new ticket has been opened for OpenZFS as well in proposing to add warnings against using ZFS native encryption and the send/receive support in production environments. "Among experienced zfs users and developers, it's conventional wisdom that zfs native encryption is not suitable for production usage, particularly when combined with snapshotting and zfs send/recv. There is a long standing data corruption issue with many firsthand user reports...Additionally, if you join #zfs or #zfsonlinux on freenode and mention that you're having an issue with zfs native encryption, you'll be met with advice from developers that zfs native encryption is simply not reliable." From Phoronix comments, a suggestion on implementation: > It should be something like that: ERROR: RAID 5/6 support has know problems and shouldn't be used on production systems. If you really want to use it please type "Enable dangerous RAID5/6 support" or add argument "--enable-dangerous-raid56-support" to this command

nubinetwork 3y ago
Why are they still using freenode? Everyone jumped ship like 2 or so years ago?
- transpute 3y ago
  Hmm, good question.