4 ms·

> We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. How should people be

> We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. How should people behave right now? Will normal usage of production filesystems erase data, or will read/write activity leave the potentially-orphaned files in place? You've also mentioned snapshots being tricky in the thread. Should people stop creating snapshots in case orphaned files are not included in the snapshots? -- > It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. IIUC: - This is specifically due to the fact that `cp` in 8.23 is optimized (8.22 created files in {0..2000} order, 8.23+ randomized the order (I don't quite understand why)) - The script in https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d30 https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d... uses `touch` to create files in random order and some people reported this triggered the bug

ryao 9y ago
Existing data is unlikely to be lost, but I strongly suggest that existing systems get off 0.7.7 if they are on it. Having to figure out where the orphaned files belonged would be very inconvenient.
- luke0016 9y ago
  I have a fairly specific question - I'm in the process of migrating a large array onto another disk using zfs send and receive (6TB). It's been at it for about 2 days now. After the migration, I plan to destroy and recreate a larger array and send it back. If the pool is not being read/written to by anything else, am I safe? Or should I stop the send and receive and downgrade? Thank you for the hard work.
  - ryao 9y ago
    You should be fine. The DMU that handles send/recv is unaffected as long as the snapshots at the source were not taken after 0.7.7 experienced an ENOSPC on file creation.
    - luke0016 9y ago
      As far as I know, it has never experienced an ENOSPC error. It was mentioned elsewhere that a procedure is being developed to detect orphaned files that arise from this bug. Would it be a good idea for me to keep the original array around long enough to run through that procedure just in case?
      - ryao 9y ago
        I am the developer who volunteered to develop the code to fix things. Anyway, there is no reason to avoid send/recv with 0.7.7. Send/recv will neither trigger this bug nor make it worse if it were triggered.
        luke0016 9y ago
        I understand - what I meant to say is that I cannot be sure that the array never ran into an ENOSPC issue prior to the snapshot. Given that, should I keep the old array around until I can do whatever is recommended to detect potential orphans resulting from the bug? Edit: And thank you, again.
        ryao 9y ago
        There is no reason to do that here. Send/recv makes a perfect copy as far as this is concerned. You are welcome. Also, the proper term is pool.
        luke0016 9y ago
        I come from many years of mdadm use, sorry :-) Just so I understand, I should be able to run through whatever the final recommendation is for detecting orphans on the new pool with identical results as if it were run on the original pool, then?
        ryao 9y ago
        Yes.
jwilk 9y ago
> 8.23+ randomized the order (I don't quite understand why) It's using inode order, which speeds up things significantly on some filesystems: https://lists.gnu.org/archive/html/bug-gnulib/2014-02/msg00007.html https://lists.gnu.org/archive/html/bug-gnulib/2014-02/msg000...
- exikyut 9y ago
  Ah, thanks! I knew there was a rationale but hadn't quite gotten it. Nice.
ryao 9y ago
I should have been more comprehensive in my previous reply. Normal usage is ill defined, but reads and writes of existing files have no chance of triggering the issue. Any snapshots that contain the orphaned files will need to be destroyed in order to fully purge the damage from affected pools, but it is fine to make them in the interim. Orphaned files should not be overwritten by further usage because they are still using space allocated from the space maps. You are right about the touch script. Albert Lee designed it after studying syscall traces from a CentOS cp binary. Other things can definitely trigger it. I read on reddit that rclone triggered it. Extracting tar archives has been suggested to also be able to trigger it. I do not expect many systems running 0.7.7 to have actually triggered the bug though. We had a hard time trying to reproduce this on systems without an old enough version of coreutils’ cp. In any case, instructions on what to do to detect and repair the damage will be made available after we finish our analysis and make the tool to fix this. That tool will likely be a subcommand in 0.7.9, which I expect Brian to push out fairly quickly once we have finalized the complete solution. Reverting the patch is a stopgap measure both to stop the population of affected systems from growing and stop the orphaned file counts on affected systems from growing.

ryao 9y ago

Existing data is unlikely to be lost, but I strongly suggest that existing systems get off 0.7.7 if they are on it. Having to figure out where the orphaned files belonged would be very inconvenient.

luke0016 9y ago
I have a fairly specific question - I'm in the process of migrating a large array onto another disk using zfs send and receive (6TB). It's been at it for about 2 days now. After the migration, I plan to destroy and recreate a larger array and send it back. If the pool is not being read/written to by anything else, am I safe? Or should I stop the send and receive and downgrade? Thank you for the hard work.
- ryao 9y ago
  You should be fine. The DMU that handles send/recv is unaffected as long as the snapshots at the source were not taken after 0.7.7 experienced an ENOSPC on file creation.
  - luke0016 9y ago
    As far as I know, it has never experienced an ENOSPC error. It was mentioned elsewhere that a procedure is being developed to detect orphaned files that arise from this bug. Would it be a good idea for me to keep the original array around long enough to run through that procedure just in case?
    - ryao 9y ago
      I am the developer who volunteered to develop the code to fix things. Anyway, there is no reason to avoid send/recv with 0.7.7. Send/recv will neither trigger this bug nor make it worse if it were triggered.
      - luke0016 9y ago
        I understand - what I meant to say is that I cannot be sure that the array never ran into an ENOSPC issue prior to the snapshot. Given that, should I keep the old array around until I can do whatever is recommended to detect potential orphans resulting from the bug? Edit: And thank you, again.
        ryao 9y ago
        There is no reason to do that here. Send/recv makes a perfect copy as far as this is concerned. You are welcome. Also, the proper term is pool.
        luke0016 9y ago
        I come from many years of mdadm use, sorry :-) Just so I understand, I should be able to run through whatever the final recommendation is for detecting orphans on the new pool with identical results as if it were run on the original pool, then?
        ryao 9y ago
        Yes.

jwilk 9y ago

> 8.23+ randomized the order (I don't quite understand why) It's using inode order, which speeds up things significantly on some filesystems: https://lists.gnu.org/archive/html/bug-gnulib/2014-02/msg00007.html https://lists.gnu.org/archive/html/bug-gnulib/2014-02/msg000...

exikyut 9y ago
Ah, thanks! I knew there was a rationale but hadn't quite gotten it. Nice.

ryao 9y ago

I should have been more comprehensive in my previous reply. Normal usage is ill defined, but reads and writes of existing files have no chance of triggering the issue. Any snapshots that contain the orphaned files will need to be destroyed in order to fully purge the damage from affected pools, but it is fine to make them in the interim. Orphaned files should not be overwritten by further usage because they are still using space allocated from the space maps. You are right about the touch script. Albert Lee designed it after studying syscall traces from a CentOS cp binary. Other things can definitely trigger it. I read on reddit that rclone triggered it. Extracting tar archives has been suggested to also be able to trigger it. I do not expect many systems running 0.7.7 to have actually triggered the bug though. We had a hard time trying to reproduce this on systems without an old enough version of coreutils’ cp. In any case, instructions on what to do to detect and repair the damage will be made available after we finish our analysis and make the tool to fix this. That tool will likely be a subcommand in 0.7.9, which I expect Brian to push out fairly quickly once we have finalized the complete solution. Reverting the patch is a stopgap measure both to stop the population of affected systems from growing and stop the orphaned file counts on affected systems from growing.