13 ms·

In OpenZFS and Btrfs, everyone was just guessing

195 points by whereistimbo 3y ago

lmz 3y ago
Reading the comment re: what the FS dev teams should do by someone who I bet didn't pay a dime for the software just upsets me.
- rini17 3y ago
  Mr. Behlendorf seems to work at an nonprofit that gets grants from DARPA. So if they are US citizen, they did in fact pay for this.
  - deleted 3y ago
    [deleted]
- kalleboo 3y ago
  Also going on about how everything was done right "back in my day". Which doesn't sound true to me, every old graybeard story I read was full of complaining about how buggy and unusable the systems by the old tech giants were.
- throwaway2037 3y ago
  [flagged]
Anon_C0ward 3y ago
Bug has been fix (apparently), btw.. https://github.com/openzfs/zfs/releases/tag/zfs-2.2.2 https://github.com/openzfs/zfs/releases/tag/zfs-2.2.2
radiowave 3y ago
Race condition is as race condition does.
- tibbydudeza 3y ago
  We need RustFS :).
  - anticensor 3y ago
    We need NVMe-KV-LFS, which would be a key-value addressed large object store based on NVMe key-value namespaces.
nabla9 3y ago
> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once a month or so. Oracle had to send on-site team to monitor the database operations to catch the bug in action. They basically had their own datacenter to record all the logs from database, networks and operating system. It took weeks for them to catch the bug. It was related to certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.
- onetimeuse92304 3y ago
  The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board. The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week. It took half a year of debugging and tens of millions of dollars in losses. We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.
  - Fire-Dragon-DoL 3y ago
    Oh god, that's insane
nubinetwork 3y ago
So many hate mongers in that thread, rubbing salt in the wound rather than trying to fix a problem that could happen for any filesystem.
- favorited 3y ago
  People are unreasonably tribal about file systems.
  - cesarb 3y ago
    > People are unreasonably tribal about file systems. I think it's probably worse than usual with ZFS, mostly for two reasons: a lot of hype (justified or not) early on its life, leading to hype aversion and/or backlash, and a "sour grapes" effect because its license ensures it will never (or at least until its copyright expires) be merged into mainline Linux, so those who can't or won't use it (due to a need or desire to run only mainline Linux kernel code) feel left out and/or get annoyed at seeing it pushed as the best thing since sliced bread all the time. Adding to that, its main competitor (btrfs) is known for having had (real or perceived) reliability issues early on its life, making its use more controversial (and its proponents more defensive at seeing the same reliability issues being raised all the time). And file systems are important, since people are reasonably concerned about the integrity of their data. They are also not easy to migrate (other than a rarely used tool to migrate in place from the ext2/3/4 family to btrfs, migrating filesystems usually requires dumping and reloading the whole data), making the choice of filesystems an important decision.
  - BSDobelix 3y ago
    Well, it's typical of the phoronix forum, they even start a fight over zstd compression levels. Like a bunch of 12 year olds who "mastered" installing system x, they now have to defend their choice of distribution, file system, desktop/wm, wayland or x11, and at the same time attack everything else because their choice was superior.
szpght 3y ago
The bug was finally fixed with the following commit: https://github.com/openzfs/zfs/pull/15579/commits/679738cc408d575289af2e31cdb1db9e311f0adf https://github.com/openzfs/zfs/pull/15579/commits/679738cc40... It looks like they managed to exactly pin down what was happening.
- lvncelot 3y ago
  Sidenote: What a great commit message. The expanded comment explains why both checks are necessary, but the commit message gives so much more context for anyone wondering and `git blame`ing that line.
  - solarkraft 3y ago
    Totally. Also: I've never seen Sponsored-by: > Signed-off-by: Rob Norris <rob.norris@klarasystems.com> > Sponsored-by: Klara, Inc. > Sponsored-by: Wasabi Technology, Inc.
    - emaste 3y ago
      This is a longstanding tradition from FreeBSD. A list of our commit message trailers: https://docs.freebsd.org/en/articles/committers-guide/#_include_appropriate_metadata_in_a_footer https://docs.freebsd.org/en/articles/committers-guide/#_incl.... "Sponsored by:" search in FreeBSD commit messages: https://freshbsd.org/?q=%22Sponsored+by%3A%22 https://freshbsd.org/?q=%22Sponsored+by%3A%22
      - solarkraft 3y ago
        I like it! Seems like a great way to incentivise companies to contribute.
        throw0101b 3y ago
        > I like it! Seems like a great way to incentivise companies to contribute. It also helps with lawsuits, which was a major thing right at the beginning of history of BSD. * https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc._v._Berkeley_Software_Design,_Inc https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc..... Linux later had the lawsuit issue with SCO and IBM and tracking where certain things came from (which was not helped by the fact that Linus Torvalds refused to use source code tracking for the longest time (later taking up Bitkeeper, and later developing git)). * https://en.wikipedia.org/wiki/SCO–Linux_disputes https://en.wikipedia.org/wiki/SCO–Linux_disputes
xoa 3y ago
First, there is a FreeBSD Errata Notice for this that offers an nice quick collection of the various bugs and subsequent repairs, with links to summaries, for anyone who is catching up on this issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 Second, I don't like the editorialization of this title ("In OpenZFS and Btrfs, everyone was just guessing") at all. No, nobody was "just guessing", but as far as I know there is no featureful FS that has undergone formal verification. It's a large codebase started by a now long defunct company that solved critical problems and delivered a lot of value, but there absolutely can be issues lurking over long time periods. It's a testament if anything that work and usage did uncover the issue, it wasn't brushed away at all but instead drilled down on and solved in short order, and now the already extensive test suites are expanded again in an organized way. And spekaing of "critical problems": Third, a lot of the commentary around these sorts of things seems to indulge in noticing the rare misses while ignoring many hits. Amongst other reasons part of the core motivation for me to switch to ZFS 100% in 2010/2011 or so and doggedly stick with it ever since was precisely because I experienced data rot (permanent corruption) with my data under previous filesystems where that WAS NOT A BUG. HFS/UFS/NTFS/XFS/whatever, none of them offer any guarantees of data integrity in the first place! Bit flips somewhere, hardware has issues, copying has noise, whatever? RAID-5 write hole? Those or lots of other things are not a bugs at all in old filesystems, because that's just how those primitive things were. I've been carrying forward data since my first Apple IIe, and went back to find some of my old work, early digital photos and drawings I cared about, and somewhere along the line it had gotten mucked up. I know not where or when, because there was no chain of checksumming and trust that would have a chance to alert me. It's impossible for a human to keep up with terabytes of data manually, it has to be fully automated, baked in. At least in ZFS there being any corruption is a drop-everything big deal problem that is incredibly rare and niche and serious people will care very much about. Pretending the old stuff was good or even acceptable is pure bullshit. It doesn't have to be mathematically perfect to deliver value, and importantly to be far superior to everything that came before. And it's inarguable that it's been battle tested very very hard for a very long time at this point. It's certainly saved my bacon a few times.
- XorNot 3y ago
  Yep, this is how I ended up running ZFS. Because with XFS the answer to "oh no a power cut" was "lol, a bunch of files might be zeros. Figure out which ones." The whole weird implication that there's something else which is perfect is always bizarre: there isn't, everyone knows that, it's extremely non-trivial to do that.
zogomoox 3y ago
Wouldn't a file system in a slightly higher level language solve some of this pain? (edit: in terms of being able to understand the system)
- XorNot 3y ago
  For the bug in the commit linked up? I don't see how: the issue is a logic error causing a data race which depends on a whole bunch of factors a higher level language can't model directly. The next step up for that is formal verification which is staggeringly time consuming (read: expensive) and hard to understand in the first place.
- p_l 3y ago
  Maybe something like Ada with SPARK, or at least partially transactional memory to make some types "atomic" in behaviour. But that would make it way harder to add to systems it's already running on.
- rollcat 3y ago
  You're at odds with the desire (necessity, actually) to have precise low-level control, not just of the in-memory layout of the data structures, but also of the performance characteristics of your code (e.g. no unnecessary pointer-chasing). Higher-level languages tend to make it easier to pile up abstractions; we want the orthogonal property of making it harder to shoot yourself in the foot. You can look at ZFS for inspiration. It is more complex than say ext4, but it's less complex than the full stack of mdadm+LVM+ext4 - the latter piles up the abstractions, where ZFS is able to "reach around" and e.g. directly track free physical blocks across multiple devices. So when you need to resilver the pool, you don't need to copy the unused blocks, and reduce the load on the pool (and thus the chance of double failure). Are there programming languages that have such properties? At the risk of perpetuating the meme, I'd say Rust (with no_std) fits that description. I can't tell how would it help in this specific instance (I'm very far from an expert in FS implementation), but it does tend to prevent data races in general.
  - bsder 3y ago
    I like Rust, but you are going to be writing a lot of unsafe Rust for a filesystem implementation. Multiple processes are writing to the filesystem simultaneously. "Ownership" is fuzzy and is moving around. At that point, is Rust buying you anything for how much it's going to get in your way? I really don't see an advantage to Rust when operating at these kinds of low levels.
mcpackieh 3y ago
I have been more than satisfied with single-drive Btrfs with OpenSUSE so far, but nevertheless I can't wait for bcachefs to prove itself. On paper at least, bcachefs seems like the most attractive of the three.
- seanw444 3y ago
  ZFS will still be preferable for production for a long while until bcachefs has proven itself, and bug reports slow down. That said, I'm also stoked for bcachefs. ZFS has always appeared to me like a bloated mess. Btrfs is more minimal, but too much so, in that it doesn't do parity (at least very reliably), which sucks because I don't always want mirroring. And it has a reputation for gobbling your data up. If bcachefs can provide all the most useful features of ZFS, while being minimal and baked into Linux by default, man that will be amazing.
  - mcpackieh 3y ago
    bcachefs's caching (foreground_target, promote_target, background_target) seems more straight forward than ZFS's ZIL/L2ARC stuff since with bcachefs you can use the same device for accelerating writes and reads of frequently accessed files, while (from what I understand) with ZFS you need two partitions, one for each. I'm not a ZFS expert by any means so maybe I've got the wrong ideas, but bcachefs seems a lot more elegant and straight forward in many regards. Still, I'm not jumping to adopt a brand new filesystem.
  - endigma 3y ago
    What does bcachefs need that it doesn't have?
    - seanw444 3y ago
      I worded it weirdly. It has what it needs already I think, but I'm saying: if it can reliably and efficiently do all the most useful functions of ZFS, while being built-in to Linux, and not starting off terribly bloated, that will be fantastic.
    - 3y ago
dmpk2k 3y ago
While I am firmly in the ZFS camp, my feeling is that there has indeed been a gradual slide in the disciplined development of ZFS. After some initial bumps in brand new software, the Sun kernel group did a good job of avoiding corruption. Once Sun was absorbed by Oracle, OSS ZFS moved into illumos, who were overall quite good at doing the same (although they had less resources to play with). OpenZFS brought ZFS to the rest of the world (good), but I can't help but notice that the illumos devs are increasingly worried about pulling changes back from OpenZFS into illumos. The increased popularity is for the best, but the change rate has increased, with all that entails.
- deleted 3y ago
  [deleted]
- ceeam 3y ago
  Too much source, eventually a human cannot follow it. A common software (and not only) problem. Maybe sometimes once in 2-3 decades it's worth taking the lessons learned and re-doing the thing? Simplifying and streamlining it greatly in the process? Like, this bug, the file holes - do we even need them now, with very good compression for once?
- emaste 3y ago
  Note that the bug that is the topic of discussion here predates OpenZFS. Whether or not there has been a slide in disciplined development in OpenZFS, this bug does not support that assertion.
  - aidenn0 3y ago
    Since the topic is whether or not the current OZFS developers understand what is going on well enough to reliably fix the bug, I think it still applies.
    - BSDobelix 3y ago
      No, the topic of that thread is to prevent such bug's in the first place. And i think that OP is right, development speed is really a bit too fast atm for a filestystem. Maybe a codereview ala openbsd could do the job?
runiq 3y ago
So are we linking from forums to other forums now just to talk smack about the forumgoers on said other forum?
- em-bee 3y ago
  yeah, why is this not linking to the actual article? https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle
donmcronald 3y ago
I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing something crazy. With the exception of a couple people like Jim Salters, I don't take any advice I see about ZFS unless I know the person is a dev or has commits on the project. And most of the time I take the advice I've seen 'ryao' post which is, paraphrased, "use the defaults unless you know what you're tuning". That should go without saying, but there are a lot of armchair "geniuses" posting online about ZFS and they are just guessing. And, how to you get any new developers if the expectation is an immediate, complete understanding of the system? That just isn't realistic and, in most cases, the dev that knows a little bit probably knows exponentially more than an average user. Interacting on the bug tracker and debugging things they don't completely understand is going to improve their understanding. There's value there. Things have gotten more complicated too. We aren't using spinning disks without caches any more, right? I can understand a disk from 20 years ago. They're relatively simple. I couldn't tell you the first thing about SSDs or NVMe stuff. There are going to be bugs. Just be glad the devs acknowledge them and fix them. Personally I still rank ZFS as the least likely to lose my data and I'll keep using it. I put roughly zero value in a forum full of people complaining.
- solarkraft 3y ago
  A lot of end users run ZFS through TrueNAS/FreeNAS and I indeed found that community to spread a lot of misinfo and un-actionable advice. I liken them to gamers lapping up any snake oil solution to get 0.1 fps more. I would suggest anyone trying to learn about ZFS to avoid it, it definitely was counter-productive to me. Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.
  - favorited 3y ago
    I think it comes down to the fact that the TrueNAS developers treat it as an appliance (which is a good thing, IMO). However, because so many people's only experience with ZFS is TrueNAS, when iXsystems says "Doing X on TrueNAS is a bad idea, that's not a configuration we support," it gets cargo culted as "doing X on ZFS is bad."
reacharavindh 3y ago
I’m not going to corroborate the entire article, but my experience from an outage 2 days ago on our new ZFS on Linux file server has left a bad taste despite years of great use with ZFS in the FreeBSD world. Ok. So, We hit the deadlock that was fixed here:-> https://github.com/rohan-puri/zfs/commit/8e4d086c13c16bc461bad9f56b3d91c44c2c149e https://github.com/rohan-puri/zfs/commit/8e4d086c13c16bc461b... but it never got merged to the master or release as far as I see it Current Master: https://github.com/openzfs/zfs/blob/acb33ee1c169bf1c1f687db18fa1815ffa68f246/module/os/linux/zfs/zfs_ctldir.c#L365 https://github.com/openzfs/zfs/blob/acb33ee1c169bf1c1f687db1... When I look up the problem, I could only see the issue being discussed, and probably leading to that commit, and then it was not in my current release (2.1) despite several years later. I’m wondering if ZFS still holds that high standard for reliability.
- donmcronald 3y ago
  Isn't it there? https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af5014bbb83e809e124a https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af... git branch --contains fd7265c * master Since 0.8. git branch -r --contains fd7265c | head origin/HEAD -> origin/master origin/compat-5.15-META origin/issue-14573-backport origin/master origin/revert-14721-remove-quota-zap origin/zfs-0.8-release origin/zfs-0.8.7-staging origin/zfs-2.0-release origin/zfs-2.0.3-staging origin/zfs-2.0.4-staging
Wowfunhappy 3y ago
OpenZFS is on an exceedingly short list of software in my life that I actually trust to do what it's supposed to. Where most software just up and falls over every so often, ZFS chugs along, day in and day out. Then ZFS has one bug and everyone starts acting like the sky is falling. And sure, it was a bad bug, but you had to be pretty unlucky to trigger it, and it was present in ZFS for all of a month. Meanwhile, people's Google Drives are apparently forgetting the last the eight months of changes. OpenZFS is fine.
- jmbwell 3y ago
  I’m sure for those who encountered it, it’s not easy to dismiss. But it does seem like this might be getting a little extra attention perhaps because it’s relatively uncommon to hear about a flaw in ZFS at all. Drawbacks and tradeoffs abound, but outright “bugs” doesn’t seem like something ZFS is generally associated with.
- chungy 3y ago
  Block cloning happened to expose this bug easier, but it seems like this bug might actually date back to the very beginning of ZFS with Sun, with the opportunities to trigger it being so rare, that nobody had noticed until now. I suppose it's still a bad bug, but it goes to show that file systems are complicated beasts, practically impossible to test all the ways they can be put through the wringer, and bugs like this can hide for 17 years before being unearthed.
  - evanjrowley 3y ago
    > it seems like this bug might actually date back to the very beginning of ZFS with Sun Looks like you might be right about that. The oldest commit referenced in the fix [0] was from 2006[1], which was ~5 months after Sun released ZFS. Glad this is fixed! [0] https://github.com/openzfs/zfs/pull/15571 https://github.com/openzfs/zfs/pull/15571 [1] https://github.com/illumos/illumos-gate/commit/c543ec060d https://github.com/illumos/illumos-gate/commit/c543ec060d
linsomniac 3y ago
One thing to remember about OpenZFS is that it is in many ways a port of a file system more than the file system itself. I could see developers there being less steeped in the file system, and more familiar with the porting layer. I'm going to spend a moment praising scripts though... A dozen or more years ago I was running a bunch of backup servers. The backup system was rsync to a zfs, then snapshot the zfs. Then do a rolling delete of snapshots to achieve X daily, Y weekly, and Z monthly copies. This is because the hardlink trick is amazingly inefficient. I started off using Nevada, but had issues and no real expertise there, so I switched to Linuz with ZFS fuse. That worked about as well as Nevada, but about once a month or two zfs-fuse would die. I built a script that would simulate the access patterns of the backup systems. I then went through a few iterations with the developers of running the script (which logged its actions as shell commands, so the log could be re-executed), reporting the results to the developers, and them producing a fix. After a few back-and-forths with the developers, zfs-fuse became rock solid for my use case. And the script was the key.
j16sdiz 3y ago
I am not sure why this is linked into the comment. That comment is not interesting. The parent article is interesting.
HankB99 3y ago
I'm a little sad to see this. I've been using ZFS on my PCs for years with minimal issue and significant benefits. [1] I've even contributed [2]. And I remain a huge fan. I don't believe that "everyone is just guessing". There are some pretty knowledgeable folk that work on this. [1] I've triggered some corruption in snapshots on an encrypted pool. No permanent problems resulted and no data was lost. [2] I provided a very minor documentation fix that was encouraged and promptly merged. https://github.com/openzfs/openzfs-docs/pull/472 https://github.com/openzfs/openzfs-docs/pull/472
tibbydudeza 3y ago
The crappiest bug I had was at one store - instead of using the recommended kit they wanted a cheaper solution so opted for a different barcode scanner but still reputable from a reputable supplier. Store started complaining about sometimes scanning muffins - this was where the price was embedded inside the barcode - the prices were ridiculous for muffins but the barcode has a luhn check digit. Must be our code management said - so we looked at our serial driver until we eventually found evidence through logging all serial I/O traffic that it was the barcode scanner that was sending back a corrupted barcode. I would more worried what you can't see - SSD controllers - yikes don't want to think about the bugs in those things.
Rapzid 3y ago
I'd wager money Brian Behlendorf has an idea how ZFS works..
wtcactus 3y ago
I see a lot of comments here using the terms ZFS and Open ZFS interchangeably. But to my understanding, these are 2 different things. - ZFS is the very stable, non open source project that only works on BSD. - Open ZFS is an effort to rewrite ZFS in a totally open and free format that now runs on Linux. This was the idea I had, but I might be incorrect.
- noigai 3y ago
  And to complement the other comments, the ZFS developers quit Oracle when it was made proprietary and moved on to work on OpenZFS, which has continued evolving. Talk by Bryan Cantrill https://www.youtube.com/watch?v=-zRN7XLCRhc https://www.youtube.com/watch?v=-zRN7XLCRhc 44:16 "[paraphrasing Bryan C.] I told Oracle management if they changed the license, the core ZFS would quit. And they lost all the ZFS team."
- gorbypark 3y ago
  From my recollection, the basic story is Sun open sourced Solaris in the early 2000's (OpenSolaris). Today's OpenZFS codebase is an incarnation of the code released in OpenSolaris. It has since been ported to Linux and continues on as a separate thing from ZFS. Oracle has since "un-open sourced" Solaris, and is why ZFS and OpenZFS are different things today, although they both share a heritage.
- keep_reading 3y ago
  Everything is OpenZFS now except Oracle's fork.
- rincebrain 3y ago
  You're completely wrong. Open source ZFS wound up in FreeBSD (and from there, other BSDs) before Sun stopped releasing Solaris source. ZoL was a project based on said open Solaris source to run it on Linux, and while the Linux glue layer was not the FreeBSD glue layer, it was still based on the same ZFS source. OpenZFS switched from being based on the illumos codebase to being based on ZoL a couple years ago, merged in FreeBSD support, and FreeBSD 13+ ships based on OpenZFS and not the original ZFS port to FreeBSD. None of them are a new implementation.
tanelpoder 3y ago
Somewhat related, RedHat withdrew from the attempt to productionize BRTFS, apparently it was fully removed in RHEL 8: - https://access.redhat.com/solutions/197643 https://access.redhat.com/solutions/197643 Also, there's a PDF doc (from a conference presentation), covering many layers of potential alternatives for ZFS features: - https://news.ycombinator.com/item?id=38484598 https://news.ycombinator.com/item?id=38484598 But "just" slapping all these layers on each other (configuring each individually and then hoping for the best) doesn't seem like the best idea for production systems either. I guess back in the Sun/Solaris days, people still had the luxury of releasing (major) features in major versions, every few years instead of weeks or months... and as far as storage, backup, security solutions go - can't really do this right with the move fast & break things culture...
- pnutjam 3y ago
  Suse has been using production btrfs for years now. https://en.opensuse.org/SDB:BTRFS https://en.opensuse.org/SDB:BTRFS
- dralley 3y ago
  Supposedly (and I only know this because Kent has talked about it in various public conferences) Kent has a regular conference call with Red Hat engineers and there is interest from Red Hat in helping it get polished up. So perhaps in a few years bcachefs will be a supported option.
AshamedCaptain 3y ago
So par for the course for about every piece of complex enough software on this planet.
tw04 3y ago
> Good god people, this is not the way something as complex as a file system should be developed. There need to be lead architects that define the operation of the system, document it clearly and concisely, and create verification systems using both coverage and fuzz testing. And as new changes and/or features are introduced the verification systems must be updated to accommodate them. ZFS has literally all of these things. I’m not sure if he’s just new to the community or making really poor assumptions based on his experience with btrfs. Heck just look at GitHub. Anytime a new feature is being developed part of the merge process includes results from the test suites…
- BSDobelix 3y ago
  It's the phoronix-forum...90% troll's and wannabe OS-engineers.
beeboobaa 3y ago
It's easy to forget the difference between an open source project and a corporate project. Feel free to contribute with your vast wealth of knowledge.
jcalvinowens 3y ago
Most of these screeds come down to a fundamental error: trusting a single filesystem with your important data. If you trust a single filesystem with your data, you're wrong. It doesn't matter how fancy it is, how many parity blocks it has, or how many disks it's backed by: a single logical filesystem definitionally cannot be redundant. If you pretend it is, it will blow up in your face one day, and it will be nobody's fault but yours. My backup is a drawer full of individual hard drives. Unlike any filesystem that ever has been or ever will be invented, that drawer cannot corrupt my data.
- admax88qqq 3y ago
  So user's are at fault for trusting a filesystem that advertises itself as being reliable? Even if user's should use multiple redundant filesystems (lol ok), that doesn't invalidate the criticisms in the linked forum post.
  - jcalvinowens 3y ago
    > So user's are at fault for trusting a filesystem that advertises itself as being reliable? Yes, absolutely 100% at fault. There are many failure modes here beyond corruption: human error, electrical surges, flooding, fire. Hell, somebody could break into your house and steal the NAS! Filesystem bugs aren't a big deal. If they are a big deal to you, it's because you're trusting a system with a single point of failure, and the consequences of that inevitable failure are not acceptable to you. Stop doing that. If I could choose between introducing a random filesystem bug and a random wifi driver bug into the kernel on my laptop, I'd take the filesystem bug every day of the week and twice on Sunday. Reinstalling my laptop is at worst a minor inconvenience that takes me an hour, and I can always use a different filesystem until the bug is fixed.
    - therealmocker 3y ago
      What you are describing sounds like the usual backup strategies. Filesystem bugs that silently corrupt your data will also get synced and backed up.
veidr 3y ago
Welcome to fucking software. I updated Mac OS X one time (or maybe it was just iTunes) and it deleted my home directory (remember that one)? I didn't open the mail promptly and missed the recall notice that my car's antilock brakes had a failure mode that was catastrophic in snow, and then I drove in the snow and then couldn't stop the car. I flew on this new model of Boeing jet a couple years ago, and it had a bug and crashed and I died. OK, one of those is not true but this is how ALL software of any complexity is. I'm also old and ornery like the OP and I don't like it and if I get drunk I will talk hella shit about it but it's just a fact. Software sucks. There was indeed a time when we (collectively) did a lot more to try to prove it worked before shipping it. But then: a.) the money people realized it made money if it mostly worked, and made less money if we went all formal-proofy on it b.) the software people realized that all the specs and architecture documents and fuzz testing and automated semantic analysis and literal actual voodoo dolls all did totally help, BUT... (you know, they help ... BUT probably wouldn't have helped in this case, or lots of other cases...) So this is just the way software is in almost every walk of life. ZFS is amazing, one of the most life-improving software technologies for me; up there with antibiotics, and UTF-8. Yes, they had a pretty amazing bug, too. So yeah, "... at least for now, there's simply no way to reliably detect bit rot and other data integrity issues and be assured they can be remedied." Which is just like it ALWAYS HAS BEEN and ALWAYS WILL BE. But in the fullness of time, so far, ZFS has gotten us closer than anything else to that (impossible, unacheivable) ideal.
grumpyprole 3y ago
A terrifying accusation. I've had both OpenZFS and Btrfs eat my data, luckily I had backups but did lose a few files. XFS is probably the best bet on Linux, it was beautifully designed and implemented by SGI back in the day. Hopefully the ChatGPT generation don't start modifying that code too much.
- noigai 3y ago
  > XFS is probably the best bet on Linux Yes, agree. However, I also lost files on it (had files overwritten with 0s, as other people already commented here). Also, it's your best bet of you don't need the extra features provided by ZFS. Thanks to its checksum validation, I detected bad SATA cables three times already (on different systems), instead of getting corrupted data. >it was beautifully designed and implemented by SGI back in the day. Hopefully the ChatGPT generation don't start modifying that code too much. Except it's not what happened. Please check this talk from Dave Chinner, where he explains how XFS was developed and ported to Linux and then made robust and stable by him and other contributors. The disk format has changed many times, now it's robust and resilient against metadata corruption. XFS Development talk by Dave Chinner https://m.youtube.com/watch?v=FegjLbCnoBw&t=683s&pp=ygUQeGZzIGRhdmUgY2hpbm5lcg%3D%3D https://m.youtube.com/watch?v=FegjLbCnoBw&t=683s&pp=ygUQeGZz... History of Linux FS, also by Dave Chinner: https://m.youtube.com/watch?v=DxZzSifuV4Q&pp=ygUQeGZzIGRhdmUgY2hpbm5lcg%3D%3D https://m.youtube.com/watch?v=DxZzSifuV4Q&pp=ygUQeGZzIGRhdmU...
  - grumpyprole 3y ago
    It sounds like it's in safe hands, good to know!
db48x 3y ago
In many ways, ZFS is an ideal case. It was started at Sun, where robust and thoughtful engineering was the norm. That gives us an extraordinarily well–built foundation to build on. We lost a lot when Sun was killed off by Oracle.
Too 3y ago
Lack of architectural direction is a common problem for many open source projects. They may have fancy user-facing documentation and a big test suite, but architecture schemas and rationale for design decisions are nowhere to be found. Usually you have to dig through the git blame until you discover the commit message is nothing but “Merge fix-something into main”, from there you have to crawl the commits to find a PR number. Inside the PR, the only useful data is “Fixes: #1234”. From there, if you are lucky, you can find some rationale hidden among the 100 comments from stale-bots. If you are not lucky, the info you need is described in another issue of another library that is cross-linking to this issue. For something sensitive as a file system, hearing they are no better is extra scary.

195 points by whereistimbo 3y ago

lmz 3y ago

Reading the comment re: what the FS dev teams should do by someone who I bet didn't pay a dime for the software just upsets me.

rini17 3y ago
Mr. Behlendorf seems to work at an nonprofit that gets grants from DARPA. So if they are US citizen, they did in fact pay for this.
- deleted 3y ago
  [deleted]
kalleboo 3y ago
Also going on about how everything was done right "back in my day". Which doesn't sound true to me, every old graybeard story I read was full of complaining about how buggy and unusable the systems by the old tech giants were.
throwaway2037 3y ago
[flagged]

Anon_C0ward 3y ago

Bug has been fix (apparently), btw.. https://github.com/openzfs/zfs/releases/tag/zfs-2.2.2 https://github.com/openzfs/zfs/releases/tag/zfs-2.2.2

radiowave 3y ago

Race condition is as race condition does.

tibbydudeza 3y ago
We need RustFS :).
- anticensor 3y ago
  We need NVMe-KV-LFS, which would be a key-value addressed large object store based on NVMe key-value namespaces.

nabla9 3y ago

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once a month or so. Oracle had to send on-site team to monitor the database operations to catch the bug in action. They basically had their own datacenter to record all the logs from database, networks and operating system. It took weeks for them to catch the bug. It was related to certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.

onetimeuse92304 3y ago
The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board. The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week. It took half a year of debugging and tens of millions of dollars in losses. We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.
- Fire-Dragon-DoL 3y ago
  Oh god, that's insane

nubinetwork 3y ago

So many hate mongers in that thread, rubbing salt in the wound rather than trying to fix a problem that could happen for any filesystem.

favorited 3y ago
People are unreasonably tribal about file systems.
- cesarb 3y ago
  > People are unreasonably tribal about file systems. I think it's probably worse than usual with ZFS, mostly for two reasons: a lot of hype (justified or not) early on its life, leading to hype aversion and/or backlash, and a "sour grapes" effect because its license ensures it will never (or at least until its copyright expires) be merged into mainline Linux, so those who can't or won't use it (due to a need or desire to run only mainline Linux kernel code) feel left out and/or get annoyed at seeing it pushed as the best thing since sliced bread all the time. Adding to that, its main competitor (btrfs) is known for having had (real or perceived) reliability issues early on its life, making its use more controversial (and its proponents more defensive at seeing the same reliability issues being raised all the time). And file systems are important, since people are reasonably concerned about the integrity of their data. They are also not easy to migrate (other than a rarely used tool to migrate in place from the ext2/3/4 family to btrfs, migrating filesystems usually requires dumping and reloading the whole data), making the choice of filesystems an important decision.
- BSDobelix 3y ago
  Well, it's typical of the phoronix forum, they even start a fight over zstd compression levels. Like a bunch of 12 year olds who "mastered" installing system x, they now have to defend their choice of distribution, file system, desktop/wm, wayland or x11, and at the same time attack everything else because their choice was superior.

szpght 3y ago

The bug was finally fixed with the following commit: https://github.com/openzfs/zfs/pull/15579/commits/679738cc408d575289af2e31cdb1db9e311f0adf https://github.com/openzfs/zfs/pull/15579/commits/679738cc40... It looks like they managed to exactly pin down what was happening.

lvncelot 3y ago
Sidenote: What a great commit message. The expanded comment explains why both checks are necessary, but the commit message gives so much more context for anyone wondering and `git blame`ing that line.
- solarkraft 3y ago
  Totally. Also: I've never seen Sponsored-by: > Signed-off-by: Rob Norris <rob.norris@klarasystems.com> > Sponsored-by: Klara, Inc. > Sponsored-by: Wasabi Technology, Inc.
  - emaste 3y ago
    This is a longstanding tradition from FreeBSD. A list of our commit message trailers: https://docs.freebsd.org/en/articles/committers-guide/#_include_appropriate_metadata_in_a_footer https://docs.freebsd.org/en/articles/committers-guide/#_incl.... "Sponsored by:" search in FreeBSD commit messages: https://freshbsd.org/?q=%22Sponsored+by%3A%22 https://freshbsd.org/?q=%22Sponsored+by%3A%22
    - solarkraft 3y ago
      I like it! Seems like a great way to incentivise companies to contribute.
      - throw0101b 3y ago
        > I like it! Seems like a great way to incentivise companies to contribute. It also helps with lawsuits, which was a major thing right at the beginning of history of BSD. * https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc._v._Berkeley_Software_Design,_Inc https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc..... Linux later had the lawsuit issue with SCO and IBM and tracking where certain things came from (which was not helped by the fact that Linus Torvalds refused to use source code tracking for the longest time (later taking up Bitkeeper, and later developing git)). * https://en.wikipedia.org/wiki/SCO–Linux_disputes https://en.wikipedia.org/wiki/SCO–Linux_disputes

xoa 3y ago

First, there is a FreeBSD Errata Notice for this that offers an nice quick collection of the various bugs and subsequent repairs, with links to summaries, for anyone who is catching up on this issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 Second, I don't like the editorialization of this title ("In OpenZFS and Btrfs, everyone was just guessing") at all. No, nobody was "just guessing", but as far as I know there is no featureful FS that has undergone formal verification. It's a large codebase started by a now long defunct company that solved critical problems and delivered a lot of value, but there absolutely can be issues lurking over long time periods. It's a testament if anything that work and usage did uncover the issue, it wasn't brushed away at all but instead drilled down on and solved in short order, and now the already extensive test suites are expanded again in an organized way. And spekaing of "critical problems": Third, a lot of the commentary around these sorts of things seems to indulge in noticing the rare misses while ignoring many hits. Amongst other reasons part of the core motivation for me to switch to ZFS 100% in 2010/2011 or so and doggedly stick with it ever since was precisely because I experienced data rot (permanent corruption) with my data under previous filesystems where that WAS NOT A BUG. HFS/UFS/NTFS/XFS/whatever, none of them offer any guarantees of data integrity in the first place! Bit flips somewhere, hardware has issues, copying has noise, whatever? RAID-5 write hole? Those or lots of other things are not a bugs at all in old filesystems, because that's just how those primitive things were. I've been carrying forward data since my first Apple IIe, and went back to find some of my old work, early digital photos and drawings I cared about, and somewhere along the line it had gotten mucked up. I know not where or when, because there was no chain of checksumming and trust that would have a chance to alert me. It's impossible for a human to keep up with terabytes of data manually, it has to be fully automated, baked in. At least in ZFS there being any corruption is a drop-everything big deal problem that is incredibly rare and niche and serious people will care very much about. Pretending the old stuff was good or even acceptable is pure bullshit. It doesn't have to be mathematically perfect to deliver value, and importantly to be far superior to everything that came before. And it's inarguable that it's been battle tested very very hard for a very long time at this point. It's certainly saved my bacon a few times.

XorNot 3y ago
Yep, this is how I ended up running ZFS. Because with XFS the answer to "oh no a power cut" was "lol, a bunch of files might be zeros. Figure out which ones." The whole weird implication that there's something else which is perfect is always bizarre: there isn't, everyone knows that, it's extremely non-trivial to do that.

zogomoox 3y ago

Wouldn't a file system in a slightly higher level language solve some of this pain? (edit: in terms of being able to understand the system)

XorNot 3y ago
For the bug in the commit linked up? I don't see how: the issue is a logic error causing a data race which depends on a whole bunch of factors a higher level language can't model directly. The next step up for that is formal verification which is staggeringly time consuming (read: expensive) and hard to understand in the first place.
p_l 3y ago
Maybe something like Ada with SPARK, or at least partially transactional memory to make some types "atomic" in behaviour. But that would make it way harder to add to systems it's already running on.
rollcat 3y ago
You're at odds with the desire (necessity, actually) to have precise low-level control, not just of the in-memory layout of the data structures, but also of the performance characteristics of your code (e.g. no unnecessary pointer-chasing). Higher-level languages tend to make it easier to pile up abstractions; we want the orthogonal property of making it harder to shoot yourself in the foot. You can look at ZFS for inspiration. It is more complex than say ext4, but it's less complex than the full stack of mdadm+LVM+ext4 - the latter piles up the abstractions, where ZFS is able to "reach around" and e.g. directly track free physical blocks across multiple devices. So when you need to resilver the pool, you don't need to copy the unused blocks, and reduce the load on the pool (and thus the chance of double failure). Are there programming languages that have such properties? At the risk of perpetuating the meme, I'd say Rust (with no_std) fits that description. I can't tell how would it help in this specific instance (I'm very far from an expert in FS implementation), but it does tend to prevent data races in general.
- bsder 3y ago
  I like Rust, but you are going to be writing a lot of unsafe Rust for a filesystem implementation. Multiple processes are writing to the filesystem simultaneously. "Ownership" is fuzzy and is moving around. At that point, is Rust buying you anything for how much it's going to get in your way? I really don't see an advantage to Rust when operating at these kinds of low levels.

mcpackieh 3y ago

I have been more than satisfied with single-drive Btrfs with OpenSUSE so far, but nevertheless I can't wait for bcachefs to prove itself. On paper at least, bcachefs seems like the most attractive of the three.

seanw444 3y ago
ZFS will still be preferable for production for a long while until bcachefs has proven itself, and bug reports slow down. That said, I'm also stoked for bcachefs. ZFS has always appeared to me like a bloated mess. Btrfs is more minimal, but too much so, in that it doesn't do parity (at least very reliably), which sucks because I don't always want mirroring. And it has a reputation for gobbling your data up. If bcachefs can provide all the most useful features of ZFS, while being minimal and baked into Linux by default, man that will be amazing.
- mcpackieh 3y ago
  bcachefs's caching (foreground_target, promote_target, background_target) seems more straight forward than ZFS's ZIL/L2ARC stuff since with bcachefs you can use the same device for accelerating writes and reads of frequently accessed files, while (from what I understand) with ZFS you need two partitions, one for each. I'm not a ZFS expert by any means so maybe I've got the wrong ideas, but bcachefs seems a lot more elegant and straight forward in many regards. Still, I'm not jumping to adopt a brand new filesystem.
- endigma 3y ago
  What does bcachefs need that it doesn't have?
  - seanw444 3y ago
    I worded it weirdly. It has what it needs already I think, but I'm saying: if it can reliably and efficiently do all the most useful functions of ZFS, while being built-in to Linux, and not starting off terribly bloated, that will be fantastic.
  - 3y ago

dmpk2k 3y ago

While I am firmly in the ZFS camp, my feeling is that there has indeed been a gradual slide in the disciplined development of ZFS. After some initial bumps in brand new software, the Sun kernel group did a good job of avoiding corruption. Once Sun was absorbed by Oracle, OSS ZFS moved into illumos, who were overall quite good at doing the same (although they had less resources to play with). OpenZFS brought ZFS to the rest of the world (good), but I can't help but notice that the illumos devs are increasingly worried about pulling changes back from OpenZFS into illumos. The increased popularity is for the best, but the change rate has increased, with all that entails.

deleted 3y ago
[deleted]
ceeam 3y ago
Too much source, eventually a human cannot follow it. A common software (and not only) problem. Maybe sometimes once in 2-3 decades it's worth taking the lessons learned and re-doing the thing? Simplifying and streamlining it greatly in the process? Like, this bug, the file holes - do we even need them now, with very good compression for once?
emaste 3y ago
Note that the bug that is the topic of discussion here predates OpenZFS. Whether or not there has been a slide in disciplined development in OpenZFS, this bug does not support that assertion.
- aidenn0 3y ago
  Since the topic is whether or not the current OZFS developers understand what is going on well enough to reliably fix the bug, I think it still applies.
  - BSDobelix 3y ago
    No, the topic of that thread is to prevent such bug's in the first place. And i think that OP is right, development speed is really a bit too fast atm for a filestystem. Maybe a codereview ala openbsd could do the job?

runiq 3y ago

So are we linking from forums to other forums now just to talk smack about the forumgoers on said other forum?

em-bee 3y ago
yeah, why is this not linking to the actual article? https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle

donmcronald 3y ago

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing something crazy. With the exception of a couple people like Jim Salters, I don't take any advice I see about ZFS unless I know the person is a dev or has commits on the project. And most of the time I take the advice I've seen 'ryao' post which is, paraphrased, "use the defaults unless you know what you're tuning". That should go without saying, but there are a lot of armchair "geniuses" posting online about ZFS and they are just guessing. And, how to you get any new developers if the expectation is an immediate, complete understanding of the system? That just isn't realistic and, in most cases, the dev that knows a little bit probably knows exponentially more than an average user. Interacting on the bug tracker and debugging things they don't completely understand is going to improve their understanding. There's value there. Things have gotten more complicated too. We aren't using spinning disks without caches any more, right? I can understand a disk from 20 years ago. They're relatively simple. I couldn't tell you the first thing about SSDs or NVMe stuff. There are going to be bugs. Just be glad the devs acknowledge them and fix them. Personally I still rank ZFS as the least likely to lose my data and I'll keep using it. I put roughly zero value in a forum full of people complaining.

solarkraft 3y ago
A lot of end users run ZFS through TrueNAS/FreeNAS and I indeed found that community to spread a lot of misinfo and un-actionable advice. I liken them to gamers lapping up any snake oil solution to get 0.1 fps more. I would suggest anyone trying to learn about ZFS to avoid it, it definitely was counter-productive to me. Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.
- favorited 3y ago
  I think it comes down to the fact that the TrueNAS developers treat it as an appliance (which is a good thing, IMO). However, because so many people's only experience with ZFS is TrueNAS, when iXsystems says "Doing X on TrueNAS is a bad idea, that's not a configuration we support," it gets cargo culted as "doing X on ZFS is bad."

reacharavindh 3y ago

I’m not going to corroborate the entire article, but my experience from an outage 2 days ago on our new ZFS on Linux file server has left a bad taste despite years of great use with ZFS in the FreeBSD world. Ok. So, We hit the deadlock that was fixed here:-> https://github.com/rohan-puri/zfs/commit/8e4d086c13c16bc461bad9f56b3d91c44c2c149e https://github.com/rohan-puri/zfs/commit/8e4d086c13c16bc461b... but it never got merged to the master or release as far as I see it Current Master: https://github.com/openzfs/zfs/blob/acb33ee1c169bf1c1f687db18fa1815ffa68f246/module/os/linux/zfs/zfs_ctldir.c#L365 https://github.com/openzfs/zfs/blob/acb33ee1c169bf1c1f687db1... When I look up the problem, I could only see the issue being discussed, and probably leading to that commit, and then it was not in my current release (2.1) despite several years later. I’m wondering if ZFS still holds that high standard for reliability.

donmcronald 3y ago
Isn't it there? https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af5014bbb83e809e124a https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af... git branch --contains fd7265c * master Since 0.8. git branch -r --contains fd7265c | head origin/HEAD -> origin/master origin/compat-5.15-META origin/issue-14573-backport origin/master origin/revert-14721-remove-quota-zap origin/zfs-0.8-release origin/zfs-0.8.7-staging origin/zfs-2.0-release origin/zfs-2.0.3-staging origin/zfs-2.0.4-staging

Wowfunhappy 3y ago

OpenZFS is on an exceedingly short list of software in my life that I actually trust to do what it's supposed to. Where most software just up and falls over every so often, ZFS chugs along, day in and day out. Then ZFS has one bug and everyone starts acting like the sky is falling. And sure, it was a bad bug, but you had to be pretty unlucky to trigger it, and it was present in ZFS for all of a month. Meanwhile, people's Google Drives are apparently forgetting the last the eight months of changes. OpenZFS is fine.

jmbwell 3y ago
I’m sure for those who encountered it, it’s not easy to dismiss. But it does seem like this might be getting a little extra attention perhaps because it’s relatively uncommon to hear about a flaw in ZFS at all. Drawbacks and tradeoffs abound, but outright “bugs” doesn’t seem like something ZFS is generally associated with.
chungy 3y ago
Block cloning happened to expose this bug easier, but it seems like this bug might actually date back to the very beginning of ZFS with Sun, with the opportunities to trigger it being so rare, that nobody had noticed until now. I suppose it's still a bad bug, but it goes to show that file systems are complicated beasts, practically impossible to test all the ways they can be put through the wringer, and bugs like this can hide for 17 years before being unearthed.
- evanjrowley 3y ago
  > it seems like this bug might actually date back to the very beginning of ZFS with Sun Looks like you might be right about that. The oldest commit referenced in the fix [0] was from 2006[1], which was ~5 months after Sun released ZFS. Glad this is fixed! [0] https://github.com/openzfs/zfs/pull/15571 https://github.com/openzfs/zfs/pull/15571 [1] https://github.com/illumos/illumos-gate/commit/c543ec060d https://github.com/illumos/illumos-gate/commit/c543ec060d

linsomniac 3y ago

One thing to remember about OpenZFS is that it is in many ways a port of a file system more than the file system itself. I could see developers there being less steeped in the file system, and more familiar with the porting layer. I'm going to spend a moment praising scripts though... A dozen or more years ago I was running a bunch of backup servers. The backup system was rsync to a zfs, then snapshot the zfs. Then do a rolling delete of snapshots to achieve X daily, Y weekly, and Z monthly copies. This is because the hardlink trick is amazingly inefficient. I started off using Nevada, but had issues and no real expertise there, so I switched to Linuz with ZFS fuse. That worked about as well as Nevada, but about once a month or two zfs-fuse would die. I built a script that would simulate the access patterns of the backup systems. I then went through a few iterations with the developers of running the script (which logged its actions as shell commands, so the log could be re-executed), reporting the results to the developers, and them producing a fix. After a few back-and-forths with the developers, zfs-fuse became rock solid for my use case. And the script was the key.

j16sdiz 3y ago

I am not sure why this is linked into the comment. That comment is not interesting. The parent article is interesting.

HankB99 3y ago

I'm a little sad to see this. I've been using ZFS on my PCs for years with minimal issue and significant benefits. [1] I've even contributed [2]. And I remain a huge fan. I don't believe that "everyone is just guessing". There are some pretty knowledgeable folk that work on this. [1] I've triggered some corruption in snapshots on an encrypted pool. No permanent problems resulted and no data was lost. [2] I provided a very minor documentation fix that was encouraged and promptly merged. https://github.com/openzfs/openzfs-docs/pull/472 https://github.com/openzfs/openzfs-docs/pull/472

tibbydudeza 3y ago

The crappiest bug I had was at one store - instead of using the recommended kit they wanted a cheaper solution so opted for a different barcode scanner but still reputable from a reputable supplier. Store started complaining about sometimes scanning muffins - this was where the price was embedded inside the barcode - the prices were ridiculous for muffins but the barcode has a luhn check digit. Must be our code management said - so we looked at our serial driver until we eventually found evidence through logging all serial I/O traffic that it was the barcode scanner that was sending back a corrupted barcode. I would more worried what you can't see - SSD controllers - yikes don't want to think about the bugs in those things.

Rapzid 3y ago

I'd wager money Brian Behlendorf has an idea how ZFS works..

wtcactus 3y ago

I see a lot of comments here using the terms ZFS and Open ZFS interchangeably. But to my understanding, these are 2 different things. - ZFS is the very stable, non open source project that only works on BSD. - Open ZFS is an effort to rewrite ZFS in a totally open and free format that now runs on Linux. This was the idea I had, but I might be incorrect.

noigai 3y ago
And to complement the other comments, the ZFS developers quit Oracle when it was made proprietary and moved on to work on OpenZFS, which has continued evolving. Talk by Bryan Cantrill https://www.youtube.com/watch?v=-zRN7XLCRhc https://www.youtube.com/watch?v=-zRN7XLCRhc 44:16 "[paraphrasing Bryan C.] I told Oracle management if they changed the license, the core ZFS would quit. And they lost all the ZFS team."
gorbypark 3y ago
From my recollection, the basic story is Sun open sourced Solaris in the early 2000's (OpenSolaris). Today's OpenZFS codebase is an incarnation of the code released in OpenSolaris. It has since been ported to Linux and continues on as a separate thing from ZFS. Oracle has since "un-open sourced" Solaris, and is why ZFS and OpenZFS are different things today, although they both share a heritage.
keep_reading 3y ago
Everything is OpenZFS now except Oracle's fork.
rincebrain 3y ago
You're completely wrong. Open source ZFS wound up in FreeBSD (and from there, other BSDs) before Sun stopped releasing Solaris source. ZoL was a project based on said open Solaris source to run it on Linux, and while the Linux glue layer was not the FreeBSD glue layer, it was still based on the same ZFS source. OpenZFS switched from being based on the illumos codebase to being based on ZoL a couple years ago, merged in FreeBSD support, and FreeBSD 13+ ships based on OpenZFS and not the original ZFS port to FreeBSD. None of them are a new implementation.

tanelpoder 3y ago

Somewhat related, RedHat withdrew from the attempt to productionize BRTFS, apparently it was fully removed in RHEL 8: - https://access.redhat.com/solutions/197643 https://access.redhat.com/solutions/197643 Also, there's a PDF doc (from a conference presentation), covering many layers of potential alternatives for ZFS features: - https://news.ycombinator.com/item?id=38484598 https://news.ycombinator.com/item?id=38484598 But "just" slapping all these layers on each other (configuring each individually and then hoping for the best) doesn't seem like the best idea for production systems either. I guess back in the Sun/Solaris days, people still had the luxury of releasing (major) features in major versions, every few years instead of weeks or months... and as far as storage, backup, security solutions go - can't really do this right with the move fast & break things culture...

pnutjam 3y ago
Suse has been using production btrfs for years now. https://en.opensuse.org/SDB:BTRFS https://en.opensuse.org/SDB:BTRFS
dralley 3y ago
Supposedly (and I only know this because Kent has talked about it in various public conferences) Kent has a regular conference call with Red Hat engineers and there is interest from Red Hat in helping it get polished up. So perhaps in a few years bcachefs will be a supported option.

AshamedCaptain 3y ago

So par for the course for about every piece of complex enough software on this planet.

tw04 3y ago

> Good god people, this is not the way something as complex as a file system should be developed. There need to be lead architects that define the operation of the system, document it clearly and concisely, and create verification systems using both coverage and fuzz testing. And as new changes and/or features are introduced the verification systems must be updated to accommodate them. ZFS has literally all of these things. I’m not sure if he’s just new to the community or making really poor assumptions based on his experience with btrfs. Heck just look at GitHub. Anytime a new feature is being developed part of the merge process includes results from the test suites…

BSDobelix 3y ago
It's the phoronix-forum...90% troll's and wannabe OS-engineers.

beeboobaa 3y ago

It's easy to forget the difference between an open source project and a corporate project. Feel free to contribute with your vast wealth of knowledge.

jcalvinowens 3y ago

Most of these screeds come down to a fundamental error: trusting a single filesystem with your important data. If you trust a single filesystem with your data, you're wrong. It doesn't matter how fancy it is, how many parity blocks it has, or how many disks it's backed by: a single logical filesystem definitionally cannot be redundant. If you pretend it is, it will blow up in your face one day, and it will be nobody's fault but yours. My backup is a drawer full of individual hard drives. Unlike any filesystem that ever has been or ever will be invented, that drawer cannot corrupt my data.

admax88qqq 3y ago
So user's are at fault for trusting a filesystem that advertises itself as being reliable? Even if user's should use multiple redundant filesystems (lol ok), that doesn't invalidate the criticisms in the linked forum post.
- jcalvinowens 3y ago
  > So user's are at fault for trusting a filesystem that advertises itself as being reliable? Yes, absolutely 100% at fault. There are many failure modes here beyond corruption: human error, electrical surges, flooding, fire. Hell, somebody could break into your house and steal the NAS! Filesystem bugs aren't a big deal. If they are a big deal to you, it's because you're trusting a system with a single point of failure, and the consequences of that inevitable failure are not acceptable to you. Stop doing that. If I could choose between introducing a random filesystem bug and a random wifi driver bug into the kernel on my laptop, I'd take the filesystem bug every day of the week and twice on Sunday. Reinstalling my laptop is at worst a minor inconvenience that takes me an hour, and I can always use a different filesystem until the bug is fixed.
  - therealmocker 3y ago
    What you are describing sounds like the usual backup strategies. Filesystem bugs that silently corrupt your data will also get synced and backed up.

veidr 3y ago

Welcome to fucking software. I updated Mac OS X one time (or maybe it was just iTunes) and it deleted my home directory (remember that one)? I didn't open the mail promptly and missed the recall notice that my car's antilock brakes had a failure mode that was catastrophic in snow, and then I drove in the snow and then couldn't stop the car. I flew on this new model of Boeing jet a couple years ago, and it had a bug and crashed and I died. OK, one of those is not true but this is how ALL software of any complexity is. I'm also old and ornery like the OP and I don't like it and if I get drunk I will talk hella shit about it but it's just a fact. Software sucks. There was indeed a time when we (collectively) did a lot more to try to prove it worked before shipping it. But then: a.) the money people realized it made money if it mostly worked, and made less money if we went all formal-proofy on it b.) the software people realized that all the specs and architecture documents and fuzz testing and automated semantic analysis and literal actual voodoo dolls all did totally help, BUT... (you know, they help ... BUT probably wouldn't have helped in this case, or lots of other cases...) So this is just the way software is in almost every walk of life. ZFS is amazing, one of the most life-improving software technologies for me; up there with antibiotics, and UTF-8. Yes, they had a pretty amazing bug, too. So yeah, "... at least for now, there's simply no way to reliably detect bit rot and other data integrity issues and be assured they can be remedied." Which is just like it ALWAYS HAS BEEN and ALWAYS WILL BE. But in the fullness of time, so far, ZFS has gotten us closer than anything else to that (impossible, unacheivable) ideal.

grumpyprole 3y ago

A terrifying accusation. I've had both OpenZFS and Btrfs eat my data, luckily I had backups but did lose a few files. XFS is probably the best bet on Linux, it was beautifully designed and implemented by SGI back in the day. Hopefully the ChatGPT generation don't start modifying that code too much.

noigai 3y ago
> XFS is probably the best bet on Linux Yes, agree. However, I also lost files on it (had files overwritten with 0s, as other people already commented here). Also, it's your best bet of you don't need the extra features provided by ZFS. Thanks to its checksum validation, I detected bad SATA cables three times already (on different systems), instead of getting corrupted data. >it was beautifully designed and implemented by SGI back in the day. Hopefully the ChatGPT generation don't start modifying that code too much. Except it's not what happened. Please check this talk from Dave Chinner, where he explains how XFS was developed and ported to Linux and then made robust and stable by him and other contributors. The disk format has changed many times, now it's robust and resilient against metadata corruption. XFS Development talk by Dave Chinner https://m.youtube.com/watch?v=FegjLbCnoBw&t=683s&pp=ygUQeGZzIGRhdmUgY2hpbm5lcg%3D%3D https://m.youtube.com/watch?v=FegjLbCnoBw&t=683s&pp=ygUQeGZz... History of Linux FS, also by Dave Chinner: https://m.youtube.com/watch?v=DxZzSifuV4Q&pp=ygUQeGZzIGRhdmUgY2hpbm5lcg%3D%3D https://m.youtube.com/watch?v=DxZzSifuV4Q&pp=ygUQeGZzIGRhdmU...
- grumpyprole 3y ago
  It sounds like it's in safe hands, good to know!

db48x 3y ago

In many ways, ZFS is an ideal case. It was started at Sun, where robust and thoughtful engineering was the norm. That gives us an extraordinarily well–built foundation to build on. We lost a lot when Sun was killed off by Oracle.

Too 3y ago

Lack of architectural direction is a common problem for many open source projects. They may have fancy user-facing documentation and a big test suite, but architecture schemas and rationale for design decisions are nowhere to be found. Usually you have to dig through the git blame until you discover the commit message is nothing but “Merge fix-something into main”, from there you have to crawl the commits to find a PR number. Inside the PR, the only useful data is “Fixes: #1234”. From there, if you are lucky, you can find some rationale hidden among the 100 comments from stale-bots. If you are not lucky, the info you need is described in another issue of another library that is cross-linking to this issue. For something sensitive as a file system, hearing they are no better is extra scary.