6 ms·

We are working on it. We know what patch introduced the regression and 0.7.8 is going out soon to revert it. Until then, users should downgrade to 0.7.6 if they

by ryao 9y ago

We are working on it. We know what patch introduced the regression and 0.7.8 is going out soon to revert it. Until then, users should downgrade to 0.7.6 if they have not already. The Gentoo and EPEL maintainers have pulled the affected releases from the repositories (technically masked on Gentoo). Ubuntu was never affected. The regression makes it so that creating a new file could fail with ENOSPC after which files created in that directory could become orphaned. Existing files seem okay, but I have yet to confirm that myself and I cannot speak for what others know. It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. So far, reports have only come from people using coreutils 8.22 or older. The directory size actually gets incremented for each orphaned file, which makes it wrong after orphan files happen. We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. Snapshots of the damaged datasets are problematic though. Until we have a subcommand to fix it (not including the snapshots, which we would have to list), the damage can be removed from a system that has it either by rolling back to a snapshot before it happened or creating a new dataset with 0.7.6 (or another release other than 0.7.7), moving everything to the new dataset and destroying the old. That will restore things to pristine condition. It should also be possible to check for pools that are affected, but I have yet to finish my analysis to be certain that no false negatives occur when checking, so I will avoid saying how for now.

exikyut 9y ago
> We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. How should people behave right now? Will normal usage of production filesystems erase data, or will read/write activity leave the potentially-orphaned files in place? You've also mentioned snapshots being tricky in the thread. Should people stop creating snapshots in case orphaned files are not included in the snapshots? -- > It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. IIUC: - This is specifically due to the fact that `cp` in 8.23 is optimized (8.22 created files in {0..2000} order, 8.23+ randomized the order (I don't quite understand why)) - The script in https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d30 https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d... uses `touch` to create files in random order and some people reported this triggered the bug
- ryao 9y ago
  Existing data is unlikely to be lost, but I strongly suggest that existing systems get off 0.7.7 if they are on it. Having to figure out where the orphaned files belonged would be very inconvenient.
  - luke0016 9y ago
    I have a fairly specific question - I'm in the process of migrating a large array onto another disk using zfs send and receive (6TB). It's been at it for about 2 days now. After the migration, I plan to destroy and recreate a larger array and send it back. If the pool is not being read/written to by anything else, am I safe? Or should I stop the send and receive and downgrade? Thank you for the hard work.
    - ryao 9y ago
      You should be fine. The DMU that handles send/recv is unaffected as long as the snapshots at the source were not taken after 0.7.7 experienced an ENOSPC on file creation.
      - luke0016 9y ago
        As far as I know, it has never experienced an ENOSPC error. It was mentioned elsewhere that a procedure is being developed to detect orphaned files that arise from this bug. Would it be a good idea for me to keep the original array around long enough to run through that procedure just in case?
        ryao 9y ago
        I am the developer who volunteered to develop the code to fix things. Anyway, there is no reason to avoid send/recv with 0.7.7. Send/recv will neither trigger this bug nor make it worse if it were triggered.
        luke0016 9y ago
antongribok 9y ago
This is a good reminder for everyone that snapshots are not backups. Also, thank you for all of the hard work on ZoL!
- ryao 9y ago
  You are welcome. I should clarify the snapshots remark. The problem with how this interacts with snapshots is that the snapshots containing orphaned files cannot be repaired by software without BPR. They can only be listed by software for deletion by the administrator. Also, for the dataset’s tip, the future tool to repair it can only put the orphaned files into a lost+found directory without the original file names.
  - gigatexal 9y ago
    Seriously you and the other maintainers are my heroes. Keep it up! And thanks for the awesome write up.
  - Annatar 9y ago
    lost+found is an incorrect approach for ZFS. If you feel you have to regress to what was 40 years ago and was a bad solution, then the misfeature causing it needs to be engineered instead of hacked together, or better yet not implemented at all. If you have to break from mainstream ZFS and regress that violently, it’s the wrong approach.
    - barrkel 9y ago
      Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?
      - Annatar 9y ago
        I suggest completely backing out the code in question and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. metadata are not files so they have no business going into lost+found.
ryao 9y ago
I put an update here: https://news.ycombinator.com/item?id=16806176 https://news.ycombinator.com/item?id=16806176
BrandonH45 9y ago
Using ZoL extensively in commercial environment - just upgraded to RHEL 7.5 with ZFS 0.7.8.1 - lost a huge 32TB pool with exact same issue as in 0.7.7 - Not an expert but we think that 0.7.8.1 didn't regress the issue properly. Issue still exists. Upgraded from 0.7.3 Love the product and your work.
- BrandonH45 9y ago
  Just wanna give an update to our desperate experience: Rebuilt server on RHEL 7.5 with : 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux And Zol 0.7.7 because we have many other systems using 0.7.7 not showing any issues but on a lower kernel. (3.10.0-693) On the new box, out of the wrapper with a brand new 32TB zpool - directory listings are messed up: >ls /newzpoolmount >ls: reading directory .: Not a directory >total 0 ALthough a new file created can be edited and read - just not listed. No hard link nor any thing else in the zfs. If this is the original issue, then we are bewildered as to why our other boxes with 0.7.7 and a lower kernel dont' show the issue with massive rsyncs running and why upgrading to 0.7.8 makes no difference? (Haven't tried 0.7.6 yet as we came off 0.7.3) Just putting it out there in case this is a useful test scenario.
- BrandonH45 9y ago
  Can you confirm whether 0.7.8 is actually fuctional under RHEL (centos) 7.5 (released last week) as this may be a complicating factor?
  - zcooper 9y ago
    We are experiencing exactly the same issues with 0.7.8 under RHEL 7.5. So far no known remedy.

by ryao 9y ago

exikyut 9y ago

> We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. How should people behave right now? Will normal usage of production filesystems erase data, or will read/write activity leave the potentially-orphaned files in place? You've also mentioned snapshots being tricky in the thread. Should people stop creating snapshots in case orphaned files are not included in the snapshots? -- > It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. IIUC: - This is specifically due to the fact that `cp` in 8.23 is optimized (8.22 created files in {0..2000} order, 8.23+ randomized the order (I don't quite understand why)) - The script in https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d30 https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d... uses `touch` to create files in random order and some people reported this triggered the bug

ryao 9y ago
Existing data is unlikely to be lost, but I strongly suggest that existing systems get off 0.7.7 if they are on it. Having to figure out where the orphaned files belonged would be very inconvenient.
- luke0016 9y ago
  I have a fairly specific question - I'm in the process of migrating a large array onto another disk using zfs send and receive (6TB). It's been at it for about 2 days now. After the migration, I plan to destroy and recreate a larger array and send it back. If the pool is not being read/written to by anything else, am I safe? Or should I stop the send and receive and downgrade? Thank you for the hard work.
  - ryao 9y ago
    You should be fine. The DMU that handles send/recv is unaffected as long as the snapshots at the source were not taken after 0.7.7 experienced an ENOSPC on file creation.
    - luke0016 9y ago
      As far as I know, it has never experienced an ENOSPC error. It was mentioned elsewhere that a procedure is being developed to detect orphaned files that arise from this bug. Would it be a good idea for me to keep the original array around long enough to run through that procedure just in case?
      - ryao 9y ago
        I am the developer who volunteered to develop the code to fix things. Anyway, there is no reason to avoid send/recv with 0.7.7. Send/recv will neither trigger this bug nor make it worse if it were triggered.
        luke0016 9y ago

antongribok 9y ago

This is a good reminder for everyone that snapshots are not backups. Also, thank you for all of the hard work on ZoL!

ryao 9y ago
You are welcome. I should clarify the snapshots remark. The problem with how this interacts with snapshots is that the snapshots containing orphaned files cannot be repaired by software without BPR. They can only be listed by software for deletion by the administrator. Also, for the dataset’s tip, the future tool to repair it can only put the orphaned files into a lost+found directory without the original file names.
- gigatexal 9y ago
  Seriously you and the other maintainers are my heroes. Keep it up! And thanks for the awesome write up.
- Annatar 9y ago
  lost+found is an incorrect approach for ZFS. If you feel you have to regress to what was 40 years ago and was a bad solution, then the misfeature causing it needs to be engineered instead of hacked together, or better yet not implemented at all. If you have to break from mainstream ZFS and regress that violently, it’s the wrong approach.
  - barrkel 9y ago
    Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?
    - Annatar 9y ago
      I suggest completely backing out the code in question and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. metadata are not files so they have no business going into lost+found.

ryao 9y ago

I put an update here: https://news.ycombinator.com/item?id=16806176 https://news.ycombinator.com/item?id=16806176

BrandonH45 9y ago

Using ZoL extensively in commercial environment - just upgraded to RHEL 7.5 with ZFS 0.7.8.1 - lost a huge 32TB pool with exact same issue as in 0.7.7 - Not an expert but we think that 0.7.8.1 didn't regress the issue properly. Issue still exists. Upgraded from 0.7.3 Love the product and your work.

BrandonH45 9y ago
Just wanna give an update to our desperate experience: Rebuilt server on RHEL 7.5 with : 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux And Zol 0.7.7 because we have many other systems using 0.7.7 not showing any issues but on a lower kernel. (3.10.0-693) On the new box, out of the wrapper with a brand new 32TB zpool - directory listings are messed up: >ls /newzpoolmount >ls: reading directory .: Not a directory >total 0 ALthough a new file created can be edited and read - just not listed. No hard link nor any thing else in the zfs. If this is the original issue, then we are bewildered as to why our other boxes with 0.7.7 and a lower kernel dont' show the issue with massive rsyncs running and why upgrading to 0.7.8 makes no difference? (Haven't tried 0.7.6 yet as we came off 0.7.3) Just putting it out there in case this is a useful test scenario.
BrandonH45 9y ago
Can you confirm whether 0.7.8 is actually fuctional under RHEL (centos) 7.5 (released last week) as this may be a complicating factor?
- zcooper 9y ago
  We are experiencing exactly the same issues with 0.7.8 under RHEL 7.5. So far no known remedy.