27 ms·
ZFS on Linux: Unlistable and disappearing files
- cmurf 9y agoLooks like they're on top of the causes and solution. But there's a caveat: possible orphans. https://github.com/zfsonlinux/zfs/issues/7401#issuecomment-379895691 https://github.com/zfsonlinux/zfs/issues/7401#issuecomment-3...
- ryao 9y agoWe will work out a solution for people affected so that they can get those orphaned files back.
- JetSpiegel 9y agoRegression introduced by https://github.com/zfsonlinux/zfs/commit/cc63068e95ee725cce03b1b7ce50179825a6cda5 https://github.com/zfsonlinux/zfs/commit/cc63068e95ee725cce0..., will be reverted and a new version tagged.
- herogreen 9y agoWow that's an old Kernel! https://linux.slashdot.org/story/16/01/31/0424221/linux-kernel-2632-lts-reaches-end-of-life-in-february-2016 https://linux.slashdot.org/story/16/01/31/0424221/linux-kern...
- mrmondo 9y agoMy thoughts exactly, while the RHEL kernel has lots of backported patches - it's by no means complete or near current.
- jlgaddis 9y agoOf course, it also isn't intended to be.
- deleted 9y ago[deleted]
- wila 9y agoWhile the 2.6.32-696.23.1.el6.x86_64 is an old kernel, it is actually the current kernel for CentOS/RHEL 6. Red Hat has backported a lot of fixes to it and CentOS/RHEL 6 is a distribution that is still under maintenance. It is still out there in a lot of places and has been in production for a while. That it is old, is a feature, it also means that it is incredibly stable. Also note that the problem is also reported on a CentOS 7.4 kernel a few posts down.
- ryao 9y agoWe used to support Linux 2.6.16, but support for that was dropped a long time ago. Then we had support for 2.6.26, but support was dropped for that too. 2.6.32 support will be maintained until at least 2020 if I recall correctly. I could see us maintaining it well past that too.
- BrandonH45 9y agoIndeed, but we are just end consumers of RHEL's release cycles. In large commercial environment (banking in this case) we are bound to use the supported versions from commercial vendors by law. My scenario is simple - lots of very stable zfs 0.7.7 running on RHEL 7.4. We saw the 0.7.7 media reports of the bug and upgraded to 0.7.8 in fear of a catastrophe - but all hell broke loose after upgrading. Downgraded to 0.7.7 but got the new 7.5 RHEL kernel and still everything is a mess. Rebuilt some test systems with RHEL release 7.5 and 0.7.7 and still cannot even list a mount point of a brand new zpool without even creating any files on it. Now we are seriously worried. Can't go back to 0.7.6 easily and any way zfs 0.7.6 modules don't load into RHEL 7.5 release - not compatible. We may have to go back to RHEL 7.4 and 0.7.7 which is still stable on many systems. You guys are all heroes for your phenomenal work. Please keep to going...
- drewg123 9y agoIs this ZOL tip or Linux specific? FWIW, the bug does not seem to reproduce on FreeBSD-current (r332158). However, it also does not reproduce on Ubuntu 4.4.0-116-generic running the ZFS stuff from Ubuntu.
- aidenn0 9y agoSee the test-case furthest down in the bug report. They were able to get it to reproduce on current linux distributions. The differences between older and newer distributions had to do with coreutils changes to the "cp" implementation between 8.22 and 8.23 Reverting this commit causes the bug to stop reproducing: https://github.com/zfsonlinux/zfs/commit/cc63068e95ee725cce03b1b7ce50179825a6cda5 https://github.com/zfsonlinux/zfs/commit/cc63068e95ee725cce0... [edit] An inspection of the following makes me think this is ZoL specific. https://github.com/freebsd/freebsd/blob/58941b0245fd8d3d5861206a07477e18635eafec/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c#L1146 https://github.com/freebsd/freebsd/blob/58941b0245fd8d3d5861...
- ryao 9y agoUbuntu was never affected. The regression started in 0.7.7 and Ubuntu is on 0.7.5. HEAD was affected until earlier today when the patch was reverted. I am not sure if the bad patch was ported to the other OpenZFS platforms.
- dmm 9y agoDebian Stable(jessie) is on 0.6.5.9 and Debian unstable(sid) is on 0.7.6 so they shouldn't be affected.
- simcop2387 9y agoDebian stable is stretch, released in june 2017. It's on 0.7.5 currently.
- simcop2387 9y agocorrection, I hadn't updated my server in a while, stretch itself is on 0.6.5.9 also, but stretch-backports is on 0.7.6 (not sure when that happened exactly)
- newnewpdro 9y agoWhy is the thread title hyphenated? It made me expect there's a utility called "zfs-bug" causing data loss.
- xxpor 9y agoThe OP appears to speak German natively. In German they are much more hyphen happy than English. It's probably the most common error I see among German speakers typing English.
- Conan_Kudo 9y agoThis. German merges words together as their modifier mechanism, whereas English just uses word order to indicate modification. Since it's technically _not_ illegal in English to hyphenate to do the same thing, German speakers tend to do this for English. It's a very hard habit for them to break. :)
- xxpor 9y agoOne could argue that technically nothing's illegal in English, since there's no governing body, just what's in common use. Kind of like civil vs common law too, now that I think about it :)
- codetrotter 9y agoSame thing in Norway wrt hyphenation; we use hyphens a lot. And I am well aware that native English speakers use them much less frequently but I intentionally use hyphens in places where an English speaker would not when there is ambiguity that can be resolved by using them. I also use hyphens for aesthetic reasons sometimes just based on what looks better to me. Also like a sibling commenter said, it’s not necessarily an error to hyphenate even though people that grew up in the US or the UK wouldn’t. In the case of OP title I would not have hyphenated though.
- rhabarba 9y agoToo many Germans use "idiot spaces" ("Deppenleerzeichen") though, e.g. "Adler Straße" instead of the correct "Adlerstraße" or "Adler-Straße" ("Eagle Street"). Anglophony hasn't done us good.
- zorkw4rg 9y agobasically the reason I like my filesystems a few decades old and mature, I would not trust zfs or btrfs with anything critical.
- aidenn0 9y agoAre you still using ext2 then? Ext3 was 2001, and ZFS was 2005. JFS2 is only 19 years old so doesn't quite meet the "decades" requirement.
- loeg 9y agoAnd just to round it out, UFS2 was first in a release in 2003 in FreeBSD 5.0. So on the BSDs you're stuck with UFS1 at "decades."
- deleted 9y ago[deleted]
- snvzz 9y agoHe's likely referring to XFS, which meets the requirement.
- foepys 9y agoConsidering that XFS is currently getting new features en masse, this doesn't sound logical. Except when they mean they are using a still maintained kernel before Linux 4.x.
- cat199 9y agonot parent, but - ext3 is improved ext2 which is improved minix fs which is improved fat, and all use the 'traditional' static-map-of-blocks approach, with gradual incremental improvements on top of this.. so the general technology 'family' is decades old even if any one implementation is not.. zfs/btrfs are log-structured, which wasn't really feasible from a hardware/performance standpoint until much later (see also LFS), and so represent implementations from a later 'generation' of technologies
- ryao 9y agoWe are working on it. We know what patch introduced the regression and 0.7.8 is going out soon to revert it. Until then, users should downgrade to 0.7.6 if they have not already. The Gentoo and EPEL maintainers have pulled the affected releases from the repositories (technically masked on Gentoo). Ubuntu was never affected. The regression makes it so that creating a new file could fail with ENOSPC after which files created in that directory could become orphaned. Existing files seem okay, but I have yet to confirm that myself and I cannot speak for what others know. It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. So far, reports have only come from people using coreutils 8.22 or older. The directory size actually gets incremented for each orphaned file, which makes it wrong after orphan files happen. We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. Snapshots of the damaged datasets are problematic though. Until we have a subcommand to fix it (not including the snapshots, which we would have to list), the damage can be removed from a system that has it either by rolling back to a snapshot before it happened or creating a new dataset with 0.7.6 (or another release other than 0.7.7), moving everything to the new dataset and destroying the old. That will restore things to pristine condition. It should also be possible to check for pools that are affected, but I have yet to finish my analysis to be certain that no false negatives occur when checking, so I will avoid saying how for now.
- exikyut 9y ago> We will likely have some way to recover the orphaned files (like ext4’s lost+found) and fix the directory sizes in the very near future. How should people behave right now? Will normal usage of production filesystems erase data, or will read/write activity leave the potentially-orphaned files in place? You've also mentioned snapshots being tricky in the thread. Should people stop creating snapshots in case orphaned files are not included in the snapshots? -- > It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. IIUC: - This is specifically due to the fact that `cp` in 8.23 is optimized (8.22 created files in {0..2000} order, 8.23+ randomized the order (I don't quite understand why)) - The script in https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d30 https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d... uses `touch` to create files in random order and some people reported this triggered the bug
- mafro 9y agoGreat to hear the ZoL guys are right on this. Bravo. It also reminds me why my NAS runs Debian..
- sureaboutthis 9y agoIt also reminds me why my NAS runs FreeBSD.
- ryao 9y agoThe bad patch passed review by developer(s) from other platforms. Matthew Ahrens was a reviewer. This was not merged based on unilateral review by the ZFSOnLinux developers. There was also nothing Linux specific about it. That said, bugs happen. We should be putting new test cases in place to help catch such regressions in the future. If we find more ways to harden the code add against regressions of this nature as we continue our analysis, we will certainly do them too.
- jclulow 9y agoReviewers will help catch bugs, but the engineer who writes the code and seeks to integrate it is ultimately responsible for sufficient testing to avoid issues like this one.
- oarsinsync 9y agoWhen multiple people have reviewed and approved, all of those people are jointly and equally responsible for any fallout. A reviewer needs to take their responsibility as seriously as the coder. If they don’t, it diminishes the value of having the code review. The coder, the reviewers, the approvers, they’re all in this together. It’s unfortunate that this happened, but no single person should be held accountable when there’s a process in place designed to protect against individual mistakes. It’s a shame that this skipped through still anyway, but that’s part of the nature of the resource limitation, especially with F/LOSS. There’s always a risk of this happening. The only way to reduce the risk is to contribute more resources. Blaming the coder is more likely to result in a reduction of resources, as less code gets done.
- beedogs 9y agoAt least it's no btrfs. What a disaster that filesystem's been.
- clircle 9y agoIs there a stable (loaded word, I know) versioning file system available for Linux?
- dmm 9y agoLVM can do snapshots and Redhat is working on its own hybrid storage management solution based on xfs called stratis[0]. Redhat deprecated btrfs recently. I don't think that means anything more than Redhat has lots of xfs devs and no btrfs devs. [0] https://www.phoronix.com/scan.php?page=news_item&px=Stratis-Red-Hat-Project https://www.phoronix.com/scan.php?page=news_item&px=Stratis-...
- jnsaff2 9y agoThere is hope though: https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs
- c-van-z 9y agonilfs?
- Spivak 9y agoThe most stable solution right now is taking your favorite stable fs like ext4 or xfs and using LVM thin pools for snapshots.
- topspin 9y agoThe simple, unqualified answer is no. BTFS is official Linux and apparently "ok" for simple use cases, but there are no end of reports of failures when non-trivial RAID modes are used, demanding work loads are applied, device replacement is attempted, etc. and there are performance problems under a variety of conditions. There are enough qualifications on the BTFS status page[1] that I, for one, do not consider it 'stable.' ZFS is a thing on Linux, but it's not in the kernel and _when_ it breaks the kernel developers don't officially care, except when they happen to have a foot in both camps. This situation naturally limits the size of the ZFS on Linux user base; you're one of the few if you're doing it and that's not where most production users want to be. LVM can snapshot logical volumes and produce `crash consistent' volumes independent of the type of file system. That's been my go-to solution given no other alternative. The fact is Linux has trailed far behind its contemporaries in advanced file systems for... 10+ years now? Not terribly flattering. I suspect the reason is that most production use of Linux occurs in environments that provide many enterprise storage functions independent of the operating system, so there isn't much pressure to, for instance, harden BTRFS until there aren't major deficiencies. I can snapshot/clone/restore/whatever my EBS volumes any time I wish and I can do similar with my private cloud powervault volumes as well. I trust either of these mechanisms far more than _anything_ Linux has ever provided, including LVM. [1] https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status
- spindle 9y agoProbably obvious, but hooray for open source software! What a fantastic response to the bug.
- tinus_hn 9y agoImagine if this had happened to Apple, everyone and his dog would be rolling over each other villifying them.
- spindle 9y agoApple's file systems HAVE had some serious bugs, as you probably know. I haven't noticed that they've been getting all that much press. What press they've been getting is very negative, though, and quite right too.
- Annatar 9y agoI paid good money to Apple so something like that would not happen, so yeah I’d be villifying them. You bet!
- tinus_hn 9y agoAs opposed to the free RedHat Enterprise Linux operating system
- Annatar 9y agoNo, I love to hate redhat because they are the most incompetent company in the computer industry’s history; just take a look at their priority 1 bugs at bugzilla.redat.com and the picture becomes crystal clear. (I’m forced to perform system engineering on RHEL every day and a lot, so my hate for it grows daily.)
- pbarnes_1 9y agoLike storing the passphrase in the hint field...? Shit happens. Software is hard.
- 9y ago
- rhabarba 9y agoSo this is not a "ZFS bug", but a "ZFS on Linux" bug. The actual ZFS on systems which have had ZFS for decades is not affected at all.
- ryao 9y agoIt is a recent regression from 2 months ago that originated in the Linux port. I do not know the status of the other platforms. I and the others involved with the Linux port are still busy handling the issue. I suggest asking the developers of the other platforms whether they had adopted the bad patch or not.
- rhabarba 9y agoI noticed someone has fixed the title, so my comment has lost its point now. Awesome.
- Fnoord 9y ago> The actual ZFS on systems which have had ZFS for decades is not affected at all. ZFS hasn't existed "for decades".
- rhabarba 9y agoOne and a half decades can probably be seen as "decades", but you are right, this was some kind of an exaggeration.
- Annatar 9y agoIt has existed since 2002, and was officially put back into onnv on 31.10.2005, and made available in Solaris 10 update 2 at the end of June 2006. So, it has existed for at least one decade and more.
- Fnoord 9y agoIt hasn't existed since 10 april 1998 therefore "decades" is inaccurate. Furthermore, since it wasn't publicly available from 10 april 2003 (who cares how long it was in development) if you round fairly, then its accurate to say "more than a decade" but "decades" is inaccurate. Because "decades" means "more than 1 decade". It denotes "at least 20 years". That's regardless of the subject we discuss or how good it is etc. Anyway, the original parent admitted their mistake (thank you) so I'm done with that discussion.
- deleted 9y ago[deleted]
- akshayjullia 9y agoPlease make an update fast for it.
- hippich 9y agoFrom title first thought was - "Wow, this is awesome security/privacy feature of the file system!" and then from reading comments I realise it is a bug
- DimitarIbra9 9y agoi stick with old good ext4